1. More Input Methods and Data Quality and Documentation
Lecture 9:
More Input Methods and Data Quality and Documentation
By Austin Troy & Brian Voigt
University of Vermont
Materials by Austin Troy & Brian Voigt © 2011

2. Part 1: Data input methods: Geocoding
------Using GIS--
Part 1:
Data input methods: Geocoding
Materials by Austin Troy & Brian Voigt © 2011

3. Materials by Austin Troy & Brian Voigt © 2011
What is Geocoding?
Convert a list / spreadsheet to spatial data / geographical features
Needs a mechanism to calculate the geographic coordinate
Address matching: uses street address database, created from a streets layer
XY coordinates
Materials by Austin Troy & Brian Voigt © 2011

4. Address Matching Geocoding
Two required inputs:
1) a table with the address records, and
2) a geographic reference layer (e.g. streets, E911)
Output: a point file, where each point represents an address record
Materials by Austin Troy & Brian Voigt © 2011

5. How are addresses matched?
Common method: matching address to street ranges.
Urban areas: usually each street segment (arc) corresponds to a block.
Each segment has attributes for the left from and to and right from and to addresses.
Computer knows topological left and right for each street segment
Step 1:Computer looks for segment with correct name and address range
Step 2: Computer interpolates the position of the address point on segment
Materials by Austin Troy & Brian Voigt © 2011

6. Materials by Austin Troy & Brian Voigt © 2011
Geocoding Example:
1060 Main Street
Look for Main street, then for the block
L-F-ADDR
L-T-ADDR
1000
Main St
1100
1001
1101
direction
R-F-ADDR
R-T-ADDR
Locate point on even (upper) side of street
Position of 1060 is interpolated
1060 Main St
Materials by Austin Troy & Brian Voigt © 2011

7. Address Matching in ArcGIS
Create address locator in ArcCatalog
Defines reference layer
Also where you specify information about your reference layer that ArcGIS might not know, allowing for more efficient geocoding
Many “styles” to choose from in for address locators
Materials by Austin Troy & Brian Voigt © 2011

8. Materials by Austin Troy & Brian Voigt © 2011
Geocoding Service
Geocoding styles are necessary because
Reference layers come in many forms and formats. For instance, a reference layer may have the from right address attribute as fr_rt_add or add_rt_frm
There are other types of geocoding, besides address geocoding, like geocoding points to the center of zip codes, and there are other types of address geocoding besides street address geocoding, like using a property parcel layer as reference.
Materials by Austin Troy & Brian Voigt © 2011

9. Materials by Austin Troy & Brian Voigt © 2011
Geocoding in ArcGIS
In geocoding style interface: choose your reference file and then specify which attributes in the reference layer correspond with the inputs that ArcGIS needs to do geocoding.
It also asks for some information about what to expect in your geocoding table (what the required attribute headings are called) and how sensitive to be to things like spelling differences
Materials by Austin Troy & Brian Voigt © 2011

10. Materials by Austin Troy © 2008
Specify rules for address list
Specify reference file
Specify address range attributes
Specify zone
Materials by Austin Troy © 2008

11. Materials by Austin Troy & Brian Voigt © 2011
Geocoding in ArcGIS
ArcMap >>> Tools >>> Geocode Addresses
Select the geocoding service you want to use
This brings up the geocoding interface where we specify which field holds the address and which holds the zone
Also specify an output shapefile or geodatabase and geocoding sensitivity
Materials by Austin Troy & Brian Voigt © 2011

12. Materials by Austin Troy & Brian Voigt © 2011
Geocoding Results
How many records were successfully matched
How many were unmatchable
How many were potentially matchable
Interactively match the potential ones
Materials by Austin Troy & Brian Voigt © 2011

13. Materials by Austin Troy & Brian Voigt © 2011
Geocoding and Error
Result is only as good as reference data
If the streets layer is only accurate to 200 meters, the geocoded points are assumed to be within 200 meters
If the streets data is consistently 100 meters to the north, then the geocoded points will be the same too
Some roads layers may have better attributes than other too
Materials by Austin Troy & Brian Voigt © 2011

14. Materials by Austin Troy & Brian Voigt © 2011
Geocoding and Error
Here’s an example where the same address list was geocoded with two different street layers.
Note here how the same house is 100 m off between the two geocoding attempts
100 m
Materials by Austin Troy & Brian Voigt © 2011

15. Materials by Austin Troy © 2008
Geocoding and Error
Here we see that many points were coded for Napa1 that were not coded for Napa2 possibly because Napa1’s street reference layer is newer, and has more streets
Materials by Austin Troy © 2008

16. Materials by Austin Troy & Brian Voigt © 2011
Geocoding and Error
This error is due to an attribute error in one of the layers which puts that address in the wrong street segment
300 m
Materials by Austin Troy & Brian Voigt © 2011

17. Materials by Austin Troy & Brian Voigt © 2011
Geocoding and Error
Rural street segments are also more subject to more error because street segments longer, so relies more on interpolation
Urban road segments: smaller, more precise
Rural road segments: long road segments, less precise
Materials by Austin Troy & Brian Voigt © 2011

18. Materials by Austin Troy & Brian Voigt © 2011
Geocoding in Action
Mapping hazard zone properties in L.A. to see effects on property values
Materials by Austin Troy & Brian Voigt © 2011

19. Materials by Austin Troy & Brian Voigt © 2011
XY Geocoding
We can also create points from a table by their latitude and longitude
Do this by clicking:
Then we specify the X,Y coordinate fields and the spatial reference system
CA haz. waste sites
Materials by Austin Troy & Brian Voigt © 2011

20. Part 2: Data input methods: Digitizing
------Using GIS--
Part 2:
Data input methods: Digitizing
Materials by Austin Troy & Brian Voigt © 2011

21. Materials by Austin Troy & Brian Voigt © 2011
Digitizing
This is generally the process of converting data from analog to digital with a device, such as a digitizing tablet or mouse, to create new vector features
User defines features by pointing and clicking.
Table digitizing involves use of a digitizing tablet or table
A digitizing table is a big table with an electronic mesh that can sense the position of a digitizing cursor
Transmits X,Y coordinates of each mouse / cursor click to the computer and usually joins those with lines
Materials by Austin Troy & Brian Voigt © 2011

22. Digitizing Notice how it is attached with tape
If it moves, the map will be inaccurate, because it’s recording position relative to the tablet, not the map
Source:

23. Materials by Austin Troy & Brian Voigt © 2011
Digitizing
Snapping: Arc will also snap closed any unsnapped lines or polygons and will crop dangling lines, based on user-defined tolerances
Snap tolerance: won’t snap together
Snap tolerance: will snap together
Snapped to other arc
Dangling arc
Materials by Austin Troy & Brian Voigt © 2011

24. Materials by Austin Troy & Brian Voigt © 2011
Digitizing
Digitizing on a tablet requires defining “control points” which allow the conversion of the digitized map to real world coordinates.
Usually, a corner point on the map of known geographic location is digitized first and its coordinates are assigned in some sort of header file
“Heads up” digitizing involves scanning a paper map to a digital file, or otherwise obtaining a digital raster map/ image and digitizing “on top” of it on computer
Materials by Austin Troy & Brian Voigt © 2011

25. Part 3: Spatial Data Quality
Materials by Austin Troy & Brian Voigt © 2011

26. Materials by Austin Troy & Brian Voigt © 2011
Data Quality
Accuracy + Precision = Quality
Error = f(accuracy, precision)
Cost vs. quality tradeoff
Materials by Austin Troy & Brian Voigt © 2011

27. Materials by Austin Troy © 2008
Accuracy
“the degree to which information on a map or in a digital database matches true or accepted values.”
From Kenneth E. Foote and Donald J. Huebner
Reflection of how close a measurement represent the actual quantity measured and of the number and severity of errors in a dataset or map.
Image source:
Materials by Austin Troy © 2008

28. Materials by Austin Troy & Brian Voigt © 2011
Precision
Intensity or level of preciseness, or exactitude in measurements. The more precise a measurement is, the smaller the unit which you intend to measure
Hence, a measurement down to a fraction of a cm is more precise than a measurement to a cm
However, data with a high level of precision can still be inaccurate—this is due to errors
Each application requires a different level of precision
Materials by Austin Troy & Brian Voigt © 2011

29. Random and Systematic Error
Error can be systematic or random
Systematic error can be rectified if discovered, because its source is understood
Materials by Austin Troy & Brian Voigt © 2011
Image source:

30. Random and Systematic Error
Systematic errors affect accuracy, but are usually independent of precision; data can use highly precise methods but still be inaccurate due to systematic error
Accurate and precise: no systematic , little random error
Accurate and imprecise: no systematic , but considerable random error
inaccurate and precise: little random error but significant systematic error
inaccurate and imprecise: both types of error
Materials by Austin Troy & Brian Voigt © 2011

31. Measurement of Accuracy
Positional accuracy is often stated as a confidence interval: e.g cm +/- .01 means true value lies between and
One of the key measurements of positional accuracy is root mean squared error (RMSE); equals squared difference between observed and expected value for observation i divided by total number of observations, summed across each observation i
This is just a standardized measure of error—how close the predicted measure is to observed
Materials by Austin Troy & Brian Voigt © 2011

32. Materials by Austin Troy & Brian Voigt © 2011
Positional Accuracy
Positional accuracy standards specify that acceptable positional error varies with scale
Data can have high level of precision but still be positionally inaccurate
Positional error is inversely related to precision and to amount of processing
Materials by Austin Troy & Brian Voigt © 2011

33. Accuracy is tied to scale
Materials by Austin Troy © 2008

34. Positional Error Standards
Different agencies have different standards for positional error
Example: USGS horizontal positional requirements state that 90% of all points must be within 1/30th of an inch for maps at a scale of 1:20,000 or larger, and 1/50th of an inch for maps at scales smaller than 1:20,000
Materials by Austin Troy & Brian Voigt © 2011

35. Positional Error Standards
USGS Accuracy standards on the ground:
1:4,800 ± feet
1:10,000 ± feet
1:12,000 ± feet
1:24,000 ± feet
1:63,360 ± feet
1:100,000 ± feet
See image from U. Colorado showing accuracy standards visually
Hence, a point on a map represents the center of a spatial probability distribution of its possible locations
Thanks to Kenneth E. Foote and Donald J. Huebner, The Geographer's Craft Project, Department of Geography, The University of Colorado at Boulder for links
Materials by Austin Troy & Brian Voigt © 2011

36. Materials by Austin Troy & Brian Voigt © 2011
Attribute Accuracy
Attribute accuracy and precision refer to quality of non-spatial, attribute data
Precision for numeric data means lots of digits
Example: recording income down to cents, rather than just dollars
Quantitative measurement errors: e.g. truncation
A common error is to measure a phenomenon in only one phase of a temporal cycle: bird counts, river flows, average weather metrics, soil moisture
Materials by Austin Troy & Brian Voigt © 2011

37. Categorical Attributes
Accuracy refers to amount of misclassification of categorical data
The chance for misclassification grows as number of possible classes increases; accuracy is a function of precision, or number of classes
If just classifying as “land and water”, that is not very precise, and not likely to result in an error
Materials by Austin Troy & Brian Voigt © 2011

38. Other Measures of Data Quality
Logical consistency
Completeness
Data currency/timeliness
Accessibility
These apply to both attribute and positional data
Image source:
Materials by Austin Troy & Brian Voigt © 2011

39. Examples of Currency and Timeliness
Materials by Austin Troy & Brian Voigt © 2011

40. Shelburne Road: 1937
Shelburne Road: 2003

41. New North End and Colchester: 2003

42. Common sources of error
Numerical processing (math operations, data type, rounding, etc)
Geocoding (e.g. rural address matching and street interpolation)
Topological errors from digitizing (overshoots, dangling nodes, slivers, etc)
Automated classification steps, like unsupervised or supervised land cover classification in remote sensing, can result in processing errors
Materials by Austin Troy & Brian Voigt © 2011

43. Error Propagation and Cascading
Propagation: one error leads to another
Cascading: errors are allowed to propagate unchecked from one layer to the next and on to the final set of products or recommendations
Cascading error can be managed to a certain extent by conducting “sensitivity analysis”
Image source:
Materials by Austin Troy & Brian Voigt © 2011

44. Materials by Austin Troy & Brian Voigt © 2011
Conflation
When one layer is better in one way and another is better in another and you wish to get the best of both
Way of reconciling best geometric and attribute features from two layers into a new one
Very commonly used for case where one layer has better attribute accuracy or completeness and another has better geometric accuracy or resolution
Also used where newer layer is produced for some theme but it has lower resolution than older one
Materials by Austin Troy & Brian Voigt © 2011

45. Materials by Austin Troy & Brian Voigt © 2011
Conflation
Attribute conflation: transferring attributes from an attribute rich layer to features in an attribute poor layer
Feature conflation: improvement of features in one layer based on coordinates and shapes in another, often called rubber sheeting. User either transforms all features or specifies certain features to be kept fixed
Materials by Austin Troy & Brian Voigt © 2011

46. Materials by Austin Troy & Brian Voigt © 2011
Conflation Example
Source: Stanley Dalal, GIS cafe
Materials by Austin Troy & Brian Voigt © 2011

47. Materials by Austin Troy & Brian Voigt © 2011
Metadata
To avoid many of these errors, good documentation of source data is needed
Metadata is data documentation, or “data about data”
Ideally, the metadata describes the data according to federally recognized standards of accuracy
Almost all state, local and federal agencies are required to provide metadata with geodata they make
Materials by Austin Troy & Brian Voigt © 2011

48. Materials by Austin Troy & Brian Voigt © 2011
Metadata
Information retrieval, cataloguing, querying and searching for data electronically.
Describing fitness for use and documenting the usability and quality of data.
Describing how to transfer, access or process data
Documenting all relevant characteristics of data needed to use it
Materials by Austin Troy & Brian Voigt © 2011

49. Materials by Austin Troy & Brian Voigt © 2011
Metadata
The federal geographic data committee (FGDC) is a federal entity that developed a “Content Standard for Digital Geospatial Metadata” in 1998, which is a model for all spatial data users to follow
Purpose is: “to provide a common set of terminology and definitions for the documentation of digital geospatial data.”
All federal agencies are required to use these standards
Materials by Austin Troy & Brian Voigt © 2011

50. Materials by Austin Troy & Brian Voigt © 2011
Metadata
Metadata usually include sections similar to these
Materials by Austin Troy & Brian Voigt © 2011

51. Materials by Austin Troy & Brian Voigt © 2011
Metadata
Critical components usually break down into:
Dataset identification, overview
Data quality
Spatial reference information
Data definition
Administrative information
Meta metadata
Materials by Austin Troy & Brian Voigt © 2011

52. Materials by Austin Troy & Brian Voigt © 2011
Metadata
Data identification, overview and administrative info:
General info: name and brief ID of dataset and owner organization, geographic domain, general description/ summary of content, data model used to represent spatial features, intent of production, language used , reference to more detailed documents, if applicable
Constraints on access and use
This is usually where info on currency is found
Materials by Austin Troy & Brian Voigt © 2011

53. Materials by Austin Troy & Brian Voigt © 2011
Metadata
Data quality should address:
Positional accuracy
Attribute accuracy
Logical consistency
Completeness
Lineage
Processing steps
Materials by Austin Troy & Brian Voigt © 2011

54. Materials by Austin Troy & Brian Voigt © 2011
Metadata
Spatial reference should include:
horizontal coordinate system (e.g. State Plane)
Includes projection used, scale factors, longitude of central meridian, latitude of projection origin, distance units
Geodetic model (e.g. NAD 83), ellipsoid, semi-major axis
Materials by Austin Troy & Brian Voigt © 2011

55. Materials by Austin Troy & Brian Voigt © 2011
Metadata
Data definition, also known as “Entity and Attribute Information,” should include:
Entity types (e.g. polygon, raster)
Information about each attribute, including label, definition, domain of values
Sometimes will include a data dictionary, or description of attribute codes, while sometimes it will reference a documents with those codes if they are too long and complex
Materials by Austin Troy & Brian Voigt © 2011

56. Materials by Austin Troy & Brian Voigt © 2011
Metadata
Data distribution info usually includes:
Name, address, phone, of contact person and organization
Liability information
Ordering information, including online and ordering by other media; usually includes fees
Materials by Austin Troy & Brian Voigt © 2011

57. Materials by Austin Troy & Brian Voigt © 2011
Metadata
Metadata reference, or meta-metadata
This is data about the metadata
Contains information on
When metadata updated
Who made it
What standard was used
What constraints apply to the metadata
Materials by Austin Troy & Brian Voigt © 2011

58. Materials by Austin Troy & Brian Voigt © 2011
Metadata in ArcGIS
ArcGIS allows you to display, import and export metadata in and to a variety of Metadata formats:
It defaults to FGDC ESRI which looks like:
Materials by Austin Troy & Brian Voigt © 2011

59. Materials by Austin Troy & Brian Voigt © 2011
Metadata in ArcGIS
XML is the most flexible form because its tag structure allows it to be used in programming; tags can be called as variables or can be created through form interfaces; allows for compatibility across platforms and programs
Materials by Austin Troy & Brian Voigt © 2011

60. Materials by Austin Troy & Brian Voigt © 2011
Metadata in ArcGIS
In the past, complete metadata was only available as text; you had to create most embedded metadata tags yourself. Today many state and nationwide datasets come with complete embedded metadata including full attribute codes
E.g. NEDs, NLCD, all VCGI data
Materials by Austin Troy & Brian Voigt © 2011

61. Materials by Austin Troy & Brian Voigt © 2011
Metadata in ArcGIS
Can edit, import, edit and export metadata in multiple formats allowing helping with proper sharing of data.
Can also make templates to save time in repeat documentation of big data sets
See NPS metadata extension for cool utilities
Materials by Austin Troy & Brian Voigt © 2011