Accuracy of Land Cover Products Why is it important and what does it all mean Note: The figures and tables in this presentation were derived from work originally prepared by Ross Nelson – Laboratory of Terrestrial Physics, NASA Goddard Space Flight Center

Overview Classified maps are not a perfect representation of reality Classified maps are not a perfect representation of reality Accuracy statistics provide objective information about the quality of the land cover classification Accuracy statistics provide objective information about the quality of the land cover classification Accuracy statistics address overall quality and per-class quality Accuracy statistics address overall quality and per-class quality Accuracy assessment can be costly Accuracy assessment can be costly Accuracy assessment is often dropped from a project because of cost or time limitations Accuracy assessment is often dropped from a project because of cost or time limitations

Types of errors Position error Position error Often due to misregistration between the base information and the area being mapped Often due to misregistration between the base information and the area being mapped Can be difficult to notice unless compared to accurate base data Can be difficult to notice unless compared to accurate base data Thematic error Thematic error Due to misidentification of individual features Due to misidentification of individual features Usually the focus of accuracy statistics Usually the focus of accuracy statistics Accuracy assessment sampling often skewed toward detecting thematic error Accuracy assessment sampling often skewed toward detecting thematic error Accuracy assessment sampling design usually allows for “reasonable” position errors by assuring sample points are not close to a class edge Accuracy assessment sampling design usually allows for “reasonable” position errors by assuring sample points are not close to a class edge

Steps for assessing accuracy Sample design Sample design Collection of validation data Collection of validation data Compiling validation data Compiling validation data Analysis Analysis

Sample design Sampling design attempts to minimize bias Sampling design attempts to minimize bias The following decisions must be made: The following decisions must be made: Number of samples Number of samples How will the samples be distributed How will the samples be distributed How will a sample area be defined (point, area) How will a sample area be defined (point, area) How will sampling take place (field, aerial photos) How will sampling take place (field, aerial photos) In many cases adjustments to the “pure” sampling design must be made to accommodate practical realities such as access to sampling points. In many cases adjustments to the “pure” sampling design must be made to accommodate practical realities such as access to sampling points. Sample data must be independent of training data Sample data must be independent of training data Many sampling designs are influenced by the amount of money available and not “pure” statistical theory Many sampling designs are influenced by the amount of money available and not “pure” statistical theory

Collection of validation data Accurate locality (i.e., latitude/longitude, UTM coordinates) data should be associated with each sampling point Accurate locality (i.e., latitude/longitude, UTM coordinates) data should be associated with each sampling point Often useful if geo-coded photographs are acquired in the field Often useful if geo-coded photographs are acquired in the field Information should be collected that is relevant to the map being assessed Information should be collected that is relevant to the map being assessed Ancillary data such as aerial photography can be used in place of data collected in the field if the mapped classes can be accurately identified Ancillary data such as aerial photography can be used in place of data collected in the field if the mapped classes can be accurately identified

Compiling validation data If photos are used to record land cover type they must be acquired in the same time frame as the image used to create a map If photos are used to record land cover type they must be acquired in the same time frame as the image used to create a map For each sampling point the classification value (i.e., land cover type) as determined in the field must be recorded For each sampling point the classification value (i.e., land cover type) as determined in the field must be recorded The classification values from the validation data (reference data) and the classified map should be tallied in a contingency table to facilitate analysis The classification values from the validation data (reference data) and the classified map should be tallied in a contingency table to facilitate analysis

Analysis Analysis is used to determine the accuracy of the map Analysis is used to determine the accuracy of the map Using simple formulas the data in a contingency table can be analyzed to determine a range of accuracy figures Using simple formulas the data in a contingency table can be analyzed to determine a range of accuracy figures Accuracy figures are often presented from the users perspective Accuracy figures are often presented from the users perspective If I select any water pixel on the classified map, what is the probability that I'll be standing in water when I visit that pixel location in the field? and from the producers perspective If I know that a particular area is water, what is the probability that the digital map will correctly identify that pixel as water?

Example of an error matrix showing pixel counts From Canada Centre for Remote Sensing, Natural Resources Canada, 11/21/2005 User's Accuracy: A map-based accuracy. User's accuracy (water) = (# of pixels correctly classified as water) / (total # of pixels classified as water) = 367/400 = 91.75% Producers accuracy: A reference-based accuracy. Producer accuracy (water) = (# of pixels correctly classified as water) / (# ground reference pixels in water) = 367/382 = 96%

Overall accuracy = (# pixels correctly classified) / (total # of pixels) = (90) / 100 = 90% Overall accuracy = (# pixels correctly classified) / (total # of pixels) = (90) / 100 = 90% Omission Error: Excluding a pixel that should have been included in the class (i.e., omission error = 1 - producers accuracy) Omission Error: Excluding a pixel that should have been included in the class (i.e., omission error = 1 - producers accuracy) Commission Error: Including a pixel in a class when it should have been excluded (i.e., commission error = 1 - user's accuracy. Commission Error: Including a pixel in a class when it should have been excluded (i.e., commission error = 1 - user's accuracy. Kappa coefficient: Measures the improvement of the classified map over a random class assignment Kappa coefficient: Measures the improvement of the classified map over a random class assignment Other accuracy terms

Another example

Typical land cover accuracy figures Forest/nonforest, water/no water, soil/vegetated: accuracies in the high 90% Forest/nonforest, water/no water, soil/vegetated: accuracies in the high 90% Conifer/hardwood: 80-90% Conifer/hardwood: 80-90% Genus: 60-70% Genus: 60-70% Species: 40-60% Species: 40-60% Bottom line: The greater the detail (precision) the lower the per class accuracy Bottom line: The greater the detail (precision) the lower the per class accuracy Note: If including a Digital Elevation Model (DEM) in the classification, add 10% Note: If including a Digital Elevation Model (DEM) in the classification, add 10%