Chapter 9 Accuracy assessment in remotely sensed categorical information 遥感类别信息精度评估 Jingxiong ZHANG 张景雄 Chapter 9 Accuracy assessment in remotely sensed categorical information 遥感类别信息精度评估 Jingxiong ZHANG 张景雄

Thematic mapping based on mage classification 1) use spectral (radiometric) differences to distinguish spatial classes 2) supervised / unsupervised mode Change detection 1) binary maps of change or no-change 2) categorized change A quick review of Chapters 7 & 8

Figure 1. Simulation: studies: (a) and (b) mean areal- class maps, for time 1 and time 2; (c) and (d) distorted versions for (a) and (b), respectively.

We will discuss Why ? What ? How ? for accuracy assessment

Key points: to construct confusion matrix ( 混淆矩阵 ) to compute accuracy measures Difficult points: Statistics for sampling design and spatial analysis

Why do we bother accuracy assessment? Thematic maps of land cover, forest types, and others can be derived from classification of remotely sensed imagery in combination with ancillary data sets You need to tell map users how well it actually represents what’s out there “Without an accuracy assessment, a classified map is just a pretty picture.”

What is accuracy assessment? Assess how well a classifier works Interpret the usefulness of someone else’s classification

How do we do accuracy assessment? Collect reference data, i.e., “ground truth” determining class types at specific locations Compare a map with the reference to compute accuracy measures Interpretation of the results

Reference Data - possible sources Aerial photo interpretation Ground truthing with GPS GIS layers Make sure we can actually extract from the reference source the information needed For discriminating four species of grass, we may need ground surveys not aerial photographs

Determining size of reference plots Match spatial scale of reference plots and remotely-sensed data Ground plots (5 meters on a side) may not be useful for remotely-sensed imagery (1km) may need aerial photos or even other satellite imagery.

Take into account spatial frequencies of image consider photo reference plots that cover an area 3 pixels on a side Example 1: Low spatial frequency Homogeneous Example 2: High spatial frequency Heterogeneous

Implication for positional accuracy consider the situation where accuracy of position of the image is +/- one pixel Example 1: Low spatial frequency Example 2: High spatial frequency

Determining number and position of samples Make sure to adequately sample the landscape Variety of sampling schemes: Random, stratified random, systematic, etc. The more reference plots, the better You can estimate how many you need statistically In reality, you can never get enough Lillesand and Kiefer: suggest 50 per class as rule of thumb

Sample size (please see the references) n << N p = 0.5

Sampling Methods : Stratified Random Sampling a minimum number of observations are randomly placed in each stratum. observations are randomly placed. Simple Random Sampling

Sampling Methods

Cluster Sampling Randomly placed “centroids” used as a base of several nearby Observations, which can be Selected randomly or systematically

Accuracy assessment Collect reference data, i.e., “ground truth” determining class types at specific locations Compare a map with the reference to compute accuracy measures Interpretation of the results

Compare (through sample locations): class type on classified map = class type determined from reference ? Summarize (cross-tabulation) into an confusion matrix Compute accuracy measures: overall classification accuracy (percent correctly classified pixels, PCC) producer’s / user’s accuracy kappa coefficient of agreement (KHAT)

An example confusion matrix ( 混淆矩阵 ) Class types determined fromreference User’s Accuracy Class types determined from classified map # PlotsConiferdeciduousgrass Totals Conifer % deciduous % grass % Totals Producer’s Accuracy 75%57%80%Total ( PCC ): 71%

Kappa coefficient of agreement Kappa of means there is 46.3% better agreement than by chance alone ( ) / ( ) = Chance agreement =  [Product of row and column marginals for each class] 0.46 for the example

Accuracy assessment Collect reference data, i.e., “ground truth” determining class types at specific locations Compare reference to classified map class type on classified map = class type determined from reference ? Interpretation of the results

Interpreting results of accuracy assessment Misclassification in remotely-sensed data: Classes are land use, not land cover Classes not spectrally separable Spatial scale of remote sensing instrument does not match classification scheme Error in reference data: Interpreter error Subjectivity

Improving Classification Land use/land cover: incorporate other data Elevation, temperature, ownership, etc. Context Spectral inseparability Hyperspectral Multiple dates Scale Different sensor Aggregate pixels Classifiers Use HIERARCHICAL CLASSIFICATION scheme In Maximum Likelihood classification, use Prior Probabilities to weigh minority classes more

Summary Accuracy assessment to add value to remote sensing information products and to ensure their proper use Ground truth is itself difficult to acquire due to the non-trivial task of class definition Sampling design is important for cost- effectiveness in accuracy assessment Accuracy assessment as an integral component in the information process

References Cochran, W.G Sampling techniques. Wiley, New York. Congalton, R A review of assessing the accuracy of classifications of remotely sensed data. Remote Sensing of Environment 37: Nusser, S.M., and E.E. Klaas Final performance report to EPA Region 7, Part II: GAP accuracy assessment pilot study. Environmental Protection Agency Contract X Final Report. Iowa Cooperative Fish and Wildlife Research Unit, Iowa State University, Ames, Iowa. 77 pp. Stehman, S.V Selecting and interpreting measures of thematic classification accuracy. Remote Sensing of Environment 62:77-89.

Questions: 1. The challenges of accuracy assessment in change detection include obtaining reference for: images taken in the past sampling sufficiently the areas that will change in the future Why is this? 2. Compute accuracy measures from the hypothetical example error matrix.