1. Data acquisition and integration
Lecture notes
Helena Mitasova,
NCSU MEAS
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

2. Geospatial Analysis and Modeling MEA592 – Helena Mitasova
Outline
Brief overview of what you should already know from the GIS Introductory courses
mapping: data acquisition
coordinate systems and transformations
geospatial data models: raster, vector
raster-vector conversions and resampling
geospatial formats and conversions
data repositories, interpreting metadata
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

3. Geospatial Analysis and Modeling MEA592 – Helena Mitasova
Data acquisition
Mapping technologies:
which you have used for your work?
Passive and active aerial and satellite sensors
On-ground surveys : (RTK)GPS, total station, laser scanner
In situ thematic data collection: climate and air quality stations, water sampling stations, species mapping, soil sampling; georeferencing usually through GPS
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

4. Data acquisition: Remote Sensing
Satellite examples:
LANDSAT 1-7 (since 1972), 30m multispec., 15m panchrom.
SPOT 1-5 (20-2.5m image, 30m DEM, France),
AVHRR(Adv. Very High Res. Radiometer 1km),
Terra: MODIS (500m, temp, aerosol), ASTER (30m, temp, DEM)
Iconos, Quickbird ( m resolution)
SRTM Shuttle Radar Topography Mission, lidar (ICESAT I)
Airborne examples
Photogrammetry: ortho, oblique, infrared, multispectral
Lidar
Future: UAV, on-board processing, sensor networks
Intro: are any mapping technologies described? Surveying class and GPS class - CALS, forestry, CCEE?
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

5. Satellite Remote Sensing
Sensors:
Data:
SRTM
Intro: are any mapping technologies described? Surveying class and GPS class - CALS, forestry, CCEE?
LANDSAT
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

6. Airborne Remote Sensing
Sensors:
Data: x,y,z points
1 point per 0.3m
Orthophotography
0.15m resolution
Intro: are any mapping technologies described? Surveying class and GPS class - CALS, forestry, CCEE?
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

7. Data acquisition: ground-based
GPS, RTK-GPS
terrestrial photogrammetry static and mobile
laser scanners static or mobile on cars/robots
discipline specific monitoring and sampling stations (econet station, ISCO sampler)
Products: georeferenced points with attributes or “streetview” imagery
Intro: are any mapping technologies described? Surveying class and GPS class - CALS, forestry, CCEE?
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

8. Data acquisition: ground based
Satellite imagery
Ground based imagery
Google Street view
Intro: are any mapping technologies described? Surveying class and GPS class - CALS, forestry, CCEE?
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

9. Data acquisition: ground based
Equipment: RTK GPS, Sonar, laser scanner, ISCO sampler
Data: airborne lidar + RTK GPS, groud-based laser scanner
Intro: are any mapping technologies described? Surveying class and GPS class - CALS, forestry, CCEE?
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

10. Geospatial Analysis and Modeling MEA592 – Helena Mitasova
From mapping to GIS
georeferencing (real-time during mapping with GPS)
feature or theme extraction
building GIS data model representation (raster or vector with attribute database)
Mapped data (imagery or points) are transformed into georeferenced, discrete representations of landscape features
Intro - this has been already covered
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

11. Geospatial Analysis and Modeling MEA592 – Helena Mitasova
Georeferencing
Georeferenced data: location on Earth is represented in a Coordinate Referenced System
MANY coordinate systems exist, they evolve over time as accuracy of the Earth measurements improves
Intro - how much is covered, how much do we need? Do the students understand prj file or EPSG projection specification>
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

12. Geospatial Analysis and Modeling MEA592 – Helena Mitasova
Coordinate systems
Geographic coordinate system
(learn it if you don't know it!):
geoid -> ellipsoid –> (sphere) -> latitude/longitude
GPS, large regions, data exchange (USGS, Google)
units are ? degree-minutes-seconds
requires complex algorithms for distances, areas
Intro - how much is covered, how much do we need? Do the students understand prj file or EPSG projection specification>
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

13. Geospatial Analysis and Modeling MEA592 – Helena Mitasova
Coordinate systems
Projected Reference Systems - cartesian coordinate systems based on projections:
geoid – ellipsoid - developable surface – plane – x,y
developable surfaces: conic, cylindrical, azimuthal (plane)
type of distortion: conformal, equidistant, equal area
Intro - how much is covered, how much do we need? Do the students understand prj file or EPSG projection specification>
image from Neteler&Mitasova, 2008&
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

14. Cartographic Projections
To learn more about Projected Reference Systems please read:
excellent, easy to understand material about projections and map properties with lots of graphics and mathematical foundations, and fun to read
see also links to references in this document
Intro - how much is covered, how much do we need? Do the students understand prj file or EPSG projection specification>
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

15. National and state systems
National/State Coordinate systems defined by:
Reference spheroid/geoid and datum
Vertical datum
Projection
Goal was to minimize distortions on maps that were used to measure distances and areas – less important now when distances and areas are computed directly from data
Intro - how much is covered, how much do we need? Do the students understand prj file or EPSG projection specification>
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

16. National and state systems
Reference geoid and datum:
North American: Clarke NAD27, Grs80 - NAD83
World geodetic system WGS84
Vertical datums: NGVD29 - National Geodetic Vertical Datum 1929, NAVD88 – North american Vertical Datum 1988
Projections
Lambert Conformal Conic (LCC): states in US
Universal Transverse Mercator (UTM): USGS, military
Albers Equal Area (conic): USGS national map
Intro - how much is covered, how much do we need? Do the students understand prj file or EPSG projection specification>
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

17. On-line mapping systems
Spherical Mercator: cylindrical on sphere, large distortions
Official name: Popular Visualization CRS and sphere
Used by Google, Microsoft and others
EPSG (group that maintains standardized list of parameters for official georeference coordinate systems ) did not like it:
“We have reviewed the coordinate reference system used by Microsoft, Google, etc. and believe that it is technically flawed. We will not devalue the EPSG dataset by including such inappropriate geodesy and cartography.”
In 1989, seven North American professional geographic organizations adopted a resolution that called for a ban on all rectangular coordinate maps (especially Mercator).
Intro - how much is covered, how much do we need? Do the students understand prj file or EPSG projection specification>
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

18. Popular visualization CRS
The reference system was eventually included under the code not recommended for professional work
Intro - how much is covered, how much do we need? Do the students understand prj file or EPSG projection specification>
Winkel tripel projection - hybrid, for world only
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

19. Coordinate systems in GIS
Representation of coordinate systems in GIS
Metadata file
ESRI PRJ file
EPSG codes provided by OGP - Int. Org. of Oil and Gas Producers Surveying and Positioning Committee, formerly EPSG – european petroleum survey group
Vertical datum support often missing in GIS – specialized tools
Intro - how much is covered, how much do we need? Do the students understand prj file or EPSG projection specification>
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

20. Coordinate systems in GIS
Coordinate system definitions for the dataset used for assignments
ESRI PRJ file (readable ASCII)
PROJCS["NAD_1983_StatePlane_North_Carolina_FIPS_3200",
GEOGCS["GCS_North_American_1983",DATUM["D_North_American_1983",
SPHEROID["GRS_1980", , ]],
PRIMEM["Greenwich",0.0],UNIT["Degree", ]],
PROJECTION["Lambert_Conformal_Conic"],
PARAMETER["False_Easting", ], PARAMETR["False_Northing",0.0],
PARAMETER["Central_Meridian",79.0],
PARAMETER["Standard_Parallel_1",
PARAMETER["Standard_Parallel_2", ],
PARAMETER["Latitude_Of_Origin",33.75],UNIT["Meter",1.0]]
EPSG translated to input parameters of the PROJ software
NAD83(High Accuracy Reference Network HARN) / North Carolina
<3358> +proj=lcc +lat_1= lat_2= lat_0= lon_0=-79 +x_0= y_0=0 +ellps=GRS80 +units=m +no_defs
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

21. Coordinate transformations
Data often come in different coordinate systems:
USGS, federal agencies: Geographic coordinates, Albers equal area, UTM
State agencies: State Plane
Older data may have different datums (NAD27, NAD83)
Coordinate transformations
x,y -> longitude, latitude -> x’,y’
on-fly transformation may be time consuming, especially for raster : resampling/reinterpolation to regular grid is required
Intro: is georeferencing covered? Setting scanning resolution, selecting GCPs? georeferencing maps and imagery (requires rectification)
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

22. Geospatial data models
Mapped, georeferenced data are transformed into discrete GIS representations using
raster (regular grid)
vector (feature: point, line, area/polygon)
geospatial data models
Intro - this has been already covered
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

23. Geospatial data models
Two different types of objects/phenomena
continuous fields: w=f(x,y), w=f(x,y,z)
each point in space is assigned a distinct value, change between two neighboring points is relatively small: elevation, precipitation
represented by raster data model, but vector model is also used: meshes, TIN, isolines or points.
discrete objects/features: lines, points or areas with attributes
represented by vector data model as geometry(shape) with attribute table or object based (geodatabase); raster representation is also used : roads, streams, census blocks, land use, schools
Intro - this has been already covered
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

24. Geospatial data models: raster
continuous: elevation, precipitation
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

25. Geospatial data models: raster
continuous: elevation, precipitation discrete: land use, roads
5 developed
1 water
3 herbaceous
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

26. Geospatial Analysis and Modeling MEA592 – Helena Mitasova
2D raster data model
header + matrix of values (INT, FP, DP)
continuous field : value assigned to a grid point
discrete object : cat value assigned to pixel (area)
imagery - several bands
Elevations
Speed limit
north:
south:
east:
west:
rows: 235
cols: 231
north:
south:
east:
west:
rows: 235
cols: 231
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

27. 2D raster data model for volumes
multiple surfaces (set of 2D raster layers) can be used to represent soil horizons or geological layers
combined representation:
continuous (horizontally)
discrete (vertically)
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

28. Geospatial Analysis and Modeling MEA592 – Helena Mitasova
3D raster data model
% org. carbon
header + 3D matrix of values
vertical scale is usually much finer than horizontal
mostly used for 3D continuous representation w=f(x,y,z)
north:
south:
east:
west:
top: 130
bottom: 20
rows: 235
cols: 231
levels:10
soil pH
contribution of real-world 3D data
(point samples, layers, volumes)
from NC to the dataset will be welcome
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

29. Raster data - changing resolution
Continuous data - reinterpolation
30m to 10m: elevation
Nearest
neighbor
Spline,
bicubic
polynomial
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

30. Raster data - changing resolution
Discrete data -resampling
30m to 10m: elevation geology
Felsic Mica
Quartzite
Quartz diorite
Metam granite
Amphibolite
Nearest
neighbor
Spline,
bicubic
polynomial
interpolation creates categories that do not exist
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

31. Raster: increasing resolution
elevation 30m
10m
10m
nearest
neighbor
slope in the
center
cell is zero!
interpolation
10m –
new image
zi z0
zk zj zi zm
zi=z0, i=1,…n
zi=f(zk), i=1,…n; k=1,…m
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

32. Raster: increasing resolution
elevation 30m
10m
10m
nearest
neighbor
slope in the
center
cell is zero!
interpolation
10m –
new image
geology 30m
nearest
neighbor
10m
interpolation
10m
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

33. Raster: increasing resolution
elevation 30m
20m
nearest neighbor 20m, not all “flats” are square
interpolation 20m
no problem
similar to 30m to 10m
20m
geology 30m
nearest neighbor 20m:
area for each class may change
but do not use interpolation !
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

34. Raster: decreasing resolution
elevation 10m
nearest neighbor 30m m
For some applications average, min or max
may be more appropriate, see also nearest neighbor operations
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

35. Raster: decreasing resolution
elevation 10m
nearest neighbor 30m m
soilsID: min or max will work but not average
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

36. Geospatial data models: vector
Discrete: streets, streams, geodetic points, census blocks
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

37. Geospatial data models: vector
Discrete: streets, streams, geodetic points, census blocks
Continuous: isolines, points
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

38. Geospatial data models: vector
vector data model - geometry:
[x,y,(z)] points representing points, lines, areas
topology: nodes, vertices, centroids, line, polyline, boundary, polygon
3D vector data: face, kernel volume
points, lines
areas
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

39. Vector data: geometry + attributes
points, lines and areas are abstract representations of complex features (firestation – point, road – centerline, ...)
attributes are stored in data management systems
geometry
L 9 1 ….
B 10
.....
C 1 1
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

40. Vector data: geometry + attributes
points, lines and areas are abstract representations of complex features (firestation – point, road – centerline, ...)
attributes are stored in data management systems
geometry
attributes
Cat ID LABEL LOCATION CITY MUN_COUNT PUMPER PUMPER_TAN TANKER
21 0 RFD # Trailwood Dr Raleigh M
L 9 1 ….
cat|MAJORRDS_|ROAD_NAME|MULTILANE|PROPYEAR|
OBJECTID|SHAPE_LEN
1|1|NC-50|no|0|1|
B 10
.....
Cat| OBJECTID| BLOCK_| BLOCK_ID|BLOCKNUM|
TOTAL_POP| POP_1RACE| WHITE_ONLY| BLACK_ONLY|AMIND_ONLY|ASIAN_ONLY|HWPAC_ONLY|OTHER_ONLY|
POP_2RACES|HISPANIC|MALE|FEMALE|P_UNDER_5|
1|83117|83118|83117| |44|44|41|0|3|0|0|
0|0|5|25|19|1 ...
C 1 1
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

41. Geospatial data models: 3D vector
3D vector data (x,y,z): points, lines, areas and volumes
volumes: face, kernel volume
extrude from footprint by given elevation
full 3D model (CAD, Sketchup)
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

42. Geospatial data models: 3D vector
Entire city - buildings extruded from
footprints using height from associated
database and stored as 3D vector data
Full 3D model with draped texture created in Sketchup
See 3D NCSU in Google Earth -
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

43. Vector to vector data conversions
polygons to points: centroids or line vertices
Data geometry is not modified:
subset is selected and stored in a different data structure
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

44. Vector to vector data conversions
polygons to lines (boundaries)
Data geometry is not modified:
subset is selected and stored in a different data structure
Topology building is required for conversions
point to line, line to polygon
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

45. Vector to vector data conversions
Generalization (downscaling) - geometry is simplified
roads, streams, contours, building footprints, urban areas,coastlines
line to simplified line
polygon (building footprint, urban area) to point symbol
Both data geometry and type can be modified
Needs to be considered when combining local, state and national scale data
Streams: 1:2000 local, 1:24000 topomap, 1:1mil national
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

46. Conversions between data models
Vector to Raster
Raster to Vector
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

47. Vector -> Raster conversions
continuous: interpolation, covered in separate lecture;
discrete: nearest neighbor
Streets to speed limit 30m resolution
raster map, null replaced by 5
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

48. Vector -> Raster conversions
continuous: interpolation, binning;
discrete: nearest neighbor
areas: attribute value applies to the entire polygon – only complete polygons can be converted to be fully valid
Streets to speed limit raster map,
null replaced by 5
Census blocks
to population
10m and 30m resolution
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

49. Raster->Vector data conversions
Continuous data: sampling points
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

50. Raster->Vector data conversions
Continuous data: sampling points, isolines
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

51. Raster->Vector data conversions
Discrete data: points – center of grid cell
lines, polygon border lines: connected grid cell centers
thinning and smoothing is often performed for lines
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

52. Raster -> Vector conversions
areas – boundary, centroid, requires building topology
connects points on grid cell boundary
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

53. Common geospatial data formats
Raster ?
Vector ?
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

54. Common geospatial data formats
Raster
GIS software: ascii and binary - ArcGRID, GRASS, SURFER, ...
Imagery: MrSID, GeoTIFF, BIN, USGS DOQ, JPEG2000, ERDAS
Graphics: GIF, JPG,PNG, Bitmap, Pixmap
HDF, NetCDF
Vector
KML, Shape, ArcSDE, GML, MapInfo, TIGER, PostGIS, OracleSpatial
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

55. Geospatial data format conversion
properties of the format are now stored with data – automated format recognition and conversion
Geospatial Data Abstraction Library (GDAL/OGR)
gdal.osgeo.org
given format -> single abstract model -> new format
includes commandline utilities for data processing
Related PROJ library provides coordinate system transformations
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

56. Geospatial Analysis and Modeling MEA592 – Helena Mitasova
Data repositories
Major web geospatial data repositories
Explore: CLICK, SRTMV4, LDART, NCFlood
Metadata
* Identification_Information
* Data_Quality_Information
* Spatial_Data_Organization_Information
* Spatial_Reference_Information
* Entity_and_Attribute_Information
* Distribution_Information
* Metadata_Reference_Information
see example:
Intro: know major web sites for GIS data downloading, learn to read metadata file, recognize data formats
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

57. Summary and references
Data acquisition
Bolstad: GIS fundamentals, Ch. 5,
Chang Ch. 5.2, 6
Coordinate systems and transformations, georeferencing
mandatory reading:
Data models: raster / vector, continuous / discrete
Chang Ch. 3,4,5, Neteler Ch. 2.1, and 4.2.1
links on the relevant slides
Geospatial Analysis and Modeling MEA592 – Helena Mitasova

58. Summary and References
Data models: raster / vector, continuous / discrete
Chang Ch. 3,4,5, Neteler Ch. 2.1, and 4.2.1
Raster-vector conversions and resampling
Chang 5.5, Neteler Ch 5.3,6.7
Geospatial data formats, conversions
Chang Ch 3,4,5.2-4, Neteler Ch. 4
Data repositories
Geospatial Analysis and Modeling MEA592 – Helena Mitasova