Accuracy Assessments in the Era of Machine Learning

Type: Paper
Theme:
Sponsor Groups: Remote Sensing Specialty Group
Organizers: Eliza Bradley, Kevin Magee
Chairs: Kevin Magee

Call for Submissions

In an era where computer vision classification accuracy is said to be approaching ‘human performance,’ it is important to view these claims with scientific skepticism, taking care to evaluate the methods they employ and the metrics against which they were assessed. Classification accuracy has long been a focus, and considerable principles and practices for measuring and assessing error and bias have solidified in the field. However, with modern, deep convolutional neural networks (CNN), we see a new challenge for measuring error and bias – deep CNNs excel at pattern detection, and bias is a powerful pattern. While the concept of optimistic bias, attributable in part to homogenous sampling, is well-known in remote sensing, modern classifiers can exhibit this bias in ways that traditional accuracy assessments may not accurately measure. The consequence of this homogenous sample induced bias is a that homogenous areas tend to have very high classification accuracy, while the transition zones between classes exhibit lower accuracy rates, or worse, may be no more accurate than a random assignment of classes.
Specifically, this session will:
1) examine how modern classifiers exhibit bias;
2) quantify how this error manifests itself spatially in unique ways;
3) and discuss practical solutions to better measure and mitigate bias.

To present a paper in this session, please send your abstract and the Participation Identification Number (PIN) to Kevin Magee (kevin.s.magee@nga.mil) or Eliza Bradley (esb165@psu.edu).

Approve for public release, 20-069.


Description

Classification accuracy assessments are arguably the most important act of the classification process – without it, the results are an uncertain collection of opinions. It is the measurement of error, both thematic and spatial, that provides confidence in the quality of the data, its relative strengths and weaknesses across classes, its locational reliability, and allows scientists a means to communicate its value and limitations for particular applications not only to one another, but to customers and policy makers. Working towards a greater understanding of how bias can differentially impact modern classification methods such as deep CNNs is key to advancing the state of this field, and ensuring that we are properly assessing and mitigating bias across the wider computer vision and remote sensing community.

Approve for public release, 20-069.


Agenda

Type Details Minutes
Discussant Kevin Magee NGA 15
Presenter Nathan Trombley*, Oak Ridge Institute for Science and Education, Leveraging Multiple Methodologies For High-Resolution Mapping of Data-Limited Subpopulations: A Case Study Using ORNL’s UrbanPop to Strengthen LandScan USA 15
Presenter LASYA VENIGALLA*, University of Texas, Dallas, Fang Qiu, Professor and Head of the Department, Geospatial Information Sciences, University of Texas, Dallas, Extracting Urban Features using Neuro-Fuzzy Classifier 15
Presenter Haile K Tadesse, US Environmental Protection Agency (EPA), John J Qu, George Mason University, Alonso A Aguirre, George Mason University, Maction Komwa*, George Mason University, Viviana Maggioni, George Mason University, Land Use Classification and Analysis Using Radar Data Mining in Ethiopia 15

To access contact information login