1. Global Land Cover: Approaches to Validation Alan Strahler GLC2000 Meeting JRC Ispra 3/02

2. Three Methods/Approaches Statistical Approaches (After S. Stehman) –Sampling a map using design-based inference to make accuracy statements about the map with known precision –Characterizing a classification process using model- based inference to estimate the accuracy of individual pixel labels Confidence-Building Measures –Carrying out confidence-building measures includes making studies or comparisons without a firm statistical basis that provide confidence in the map

3. Design-Based Inference Definition –Sampling to infer characteristics of a finite population, such as the pixels in a land cover map Probability Sampling –Sampled units are drawn with known probabilities –Example: Random or stratified random sampling Consistency Criterion in Estimation –An estimator of a population parameter must equal the population parameter if the sample size includes the entire population

4. Design-Based Inference, Cont. Consistent estimators include: –Proportion of pixels correctly classified –User’s Accuracy: Given that a pixel is mapped as A, what is the probability that it actually is A on the ground? –Producer’s Accuracy: Given that a pixel is actually of class A, what is the probability that it is mapped correctly? Confusion Matrix –The confusion matrix is the primary tool used to find consistent estimators –Diagonals count matches and marginal totals count number of pixels sampled

5. Problems in Design-Based Inference “Ground Truth” –Determination of the “correct” class for a sampled pixel is not without error Photointerpretation errors occur when fine-scale imagery is used instead of ground visits Misregistration errors—the wrong location is visited or viewed at higher resolution Equivocal Classification Schemes –Classes may not be mutually exclusive or be difficult to resolve Example: Permanent wetland may also be forest (IGBP). Are both labels correct? –Classes may not be well defined Example: What is a golf course? Is it agriculture? Grassland? Urban?

6. Problems in Design-Based Inference, Cont. “Correctness” of Match and Mismatch –Some errors are worse than others—e.g., open shrubland vs. closed shrubland may be minor, while forest classified as water may be major –Leads to fuzzy agreement measures as better indicators of map utility Mixed Pixels –Ground truth pixels may contain multiple classes –Which label is correct? –Leads to fuzzy confusion matrixes Map Comparisons –Given their error structures, how do we conclude that two maps are different? –If they are different, which one is more accurate?

7. Model-Based Inference Focuses on the classification process, not the map –E.g., Which classifier works better? –Maps as realizations of a classification process that makes random errors Reliability Measures –Parameters that are inferred from the classification process –E.g., maximum likelihood classification gives the probability that a pixel belongs to a particular class –Can be mapped and summarized to provide information about the “quality” of a map

8. Confidence-Building Measures “Looks good!” Reconnaissance Measures –Map conforms well to regional landscape attributes— mountains, valleys, agricultural regions, etc. –Spatial structure is sensible, not salt-and-pepper noise or excessively smooth –Land-water boundaries are clear, indicating good registration of input data –Free of major glitches, such as cities in the Sahara Ancillary Comparisons –Does the classifier’s output conform to the general patterns of land cover documented in other datasets or maps? Systematic Assessment –Qualitative assessment of map accuracy in a systematic (wall-to-wall) fashion

9. Summary Design-based inference provides statements of accuracy with known precision at highest cost Model-based inference characterizes the accuracy of the map-making process at lesser cost Confidence-building measures assess map quality at low cost Validation can, and should, rely on all three approaches.