TDWG- Lisbon Oct 2003 Data Cleaning Tools and Methodologies Arthur D. Chapman Australia / Brazil Centro de Referência em Informação Ambiental

TDWG- Lisbon Oct 2003 Background ERIN/CRIA speciesLink FAPESP/Biota

TDWG- Lisbon Oct 2003 Species Data Museum/Herbarium Observation Survey

TDWG- Lisbon Oct 2003 Data Error Names Geocode Altitude Collectors Dates

TDWG- Lisbon Oct 2003 Adding Data to the Database Software –Biota –BRAHMS –Specify –BioLink –EGaz –Etc.

TDWG- Lisbon Oct 2003 On-line Tools BioGeomancer ( CRIA-localidade ( Guidelines –MANIS –HISPID –Data Cleaning and Validation

TDWG- Lisbon Oct 2003 Data quality - fitness for use

TDWG- Lisbon Oct 2003 Recording Accuracy and Error Additional Accuracy Fields –Preferably in meters (Point-Radius) Documenting Validation tests –Who –What –How

TDWG- Lisbon Oct 2003 Methods for geocode validation Internal Database Checks Outliers in Geographic Space - GIS Outliers in Environmental Space - Models Statistical outliers

TDWG- Lisbon Oct 2003 Internal Database Checks Internal inconsistencies Checking one field against another –Text location vs geocode Checking one database against another –Gazetteers –DEM –Collectors

TDWG- Lisbon Oct 2003 Geographic outliers - GIS Country, State, named district, etc.

TDWG- Lisbon Oct 2003 Geographic outliers - GIS

TDWG- Lisbon Oct 2003 Geographic Outliers - GIS Collectors – location vs date

TDWG- Lisbon Oct 2003 Environmental Outliers Cumulative Frequency Curves

TDWG- Lisbon Oct 2003 Acacia orites - 19 records - 9 Temperature parameters Reverse Jack-knife

TDWG- Lisbon Oct 2003 Outliers in climate space (T=0.95(√n)+0.2) where ‘n’ is the number of records

TDWG- Lisbon Oct 2003 FloraMap CIAT (Columbia) PCA Cluster Analysis $US100 Modelling 10-minute grids

TDWG- Lisbon Oct 2003 Principal Components Analysis - FloraMap Image from FloraMap (Jones and Gladkov 2001) showing use of Principal Components Analysis to identify an outlier in Rauvolfia littoralis specimen data. A. Principal Components Analysis B. Specimen record. C. Mapped specimen. D. Climate profile

TDWG- Lisbon Oct 2003 Cluster Analysis - FloraMap Image from FloraMap (Jones and Gladkov 2001) showing use of Cluster Analysis to identify an outlier in Rauvolfia littoralis specimen data. A.Cluster Analysis B. Principal Components Analysis. C. Mapped specimen. D. Climate profile. E. Specimen record

TDWG- Lisbon Oct 2003 Diva-GIS Free Simple GIS Modelling (BIOCLIM/Domain) Data Cleaning Tools

TDWG- Lisbon Oct 2003 Diva-GIS – Coordinate Check Using Diva-GIS to check coordinates by comparing a file of point specimen records (red) against a polygon of Bolivian provinces. Input dialogue box is shown at A, where it can be seen that “STATE” in the point file has been set to the equivalent “DEPARTMENT” in the polygon file (Hijmans et al. 2003).

TDWG- Lisbon Oct 2003 Points outside Polygon – Diva GIS Results from Diva-GIS (Hijmans et al. 2003) showing point records that fall outside all polygons in the Bolivian provinces polygon file. The highlighted record shows the linking between the results dialogue box and the mapped record

TDWG- Lisbon Oct 2003 Mismatched Provinces – Diva GIS Results from Diva-GIS (Hijmans et al. 2003) showing point records that do not match set relationships between the specimen point file and the polygon of Bolivian provinces. The highlighted record where the geocoding on the specimen record causes it to fall in the wrong province

TDWG- Lisbon Oct 2003 Assign Coordinates – Diva GIS Results from Diva-GIS (Hijmans et al. 2003) showing point records with geocodes automatically assigned. A. Unambiguous geocodes found by the program and assigned. B. Ambiguous geocodes identified. C. Appropriate geocodes not found.

TDWG- Lisbon Oct 2003 Multiple possibilities – Diva GIS Results from Diva-GIS (Hijmans et al. 2003) showing alternate geocodes for a record where use of the Gazetteer has produced a number of credible alternatives.

TDWG- Lisbon Oct 2003 Cumulative Frequency Curves - DivaGiS Results from Diva-GIS (Hijmans et al. 2003) showing the use of the Cumulative Frequency curve from BIOCLIM to identify possible geocoding errors in Rauvolfia littoralis. A1 and A2 show possible outliers in climate space, B1 and B2 the corresponding mapped records. The Blue lines represent the 97.5 percentile

TDWG- Lisbon Oct 2003 Bioclimatic Envelop – Diva GIS Results from Diva-GIS (Hijmans et al. 2003) showing the use of the Bioclimatic Envelope from BIOCLIM to identify outliers in climate space. In this case the percentile cut off is set at 95. Red points on the envelope correspond with red points on the map, green points in the envelope correspond with yellow points on the map

TDWG- Lisbon Oct 2003 ANUCLIM $AUD1000 (with data files) Modelling (BIOCLIM / ESOCLIM) Cumulative Frequency Curves Parameter Extremes

TDWG- Lisbon Oct 2003 Cumulative Frequency - ANUCLIM Log file of Eucalyptus fastigata from ANUCLIM Version 5.1 (Houlder et al. 2002) showing the species accumulation curve with an identified outlier (labelled “bad”). Information from the “bad” record is displayed at the top of the log file (from Houlder et al. 2000).

TDWG- Lisbon Oct 2003 Parameter extremes - ANUCLIM Log file of Eucalyptus fastigata from ANUCLIM Version 5.1 (Houlder et al. 2002) showing the parameter extremes (top) and associated species accumulation curve (bottom) (from Houlder et al. 2000

TDWG- Lisbon Oct 2003 Statistical Tests Outliers in Latitude Outliers in Altitude Outliers in collectors range/day or week –Especially 17 th, 18 th and 19 th Century collections

TDWG- Lisbon Oct 2003 Thank You… Questions?