1. Archiving State and Local Agency Digital Geospatial Data: An Overview of the Problem Area Steven P. Morris Head of Digital Library Initiatives North Carolina State University Libraries GICC Archival and Long Term Access Kickoff MeetingFebruary 29, 2008

2. Note: Percentages based on the actual number of respondents to each question 2 Outline Risks to Digital Geospatial Data Value in Temporal/Historical Data Archiving Challenges Content Identification and Selection Issues Industry Engagement Archives Processes Conclusion

3. Note: Percentages based on the actual number of respondents to each question 3 NC Geospatial Data Archiving Project Partnership between university library (NCSU) and state agency (NCCGIA), with Library of Congress under the National Digital Information Infrastructure and Preservation Program (NDIIPP) One of 8 initial NDIIPP collection building partnerships Focus on state and local geospatial content in North Carolina (state demonstration) Tied to NC OneMap initiative, which provides for seamless access to data, metadata, and inventories Objective: engage existing state/federal geospatial data infrastructures in preservation Serve as catalyst for discussion within industry

4. Note: Percentages based on the actual number of respondents to each question 4 NCGDAP Goals Repository Goal Capture at-risk data Explore technical and organizational challenges Project End Goal Data Producers: Improved temporal data management practices Archives: More efficient means of acquiring and preserving data; Progress towards best practices Temporal data management vs. long-term preservation

5. Note: Percentages based on the actual number of respondents to each question 5 Risks to Geospatial Data

6. 6 How would you describe your current geospatial archive? Last week’s set of nightly tape backups Several boxes of CD’s and DVD’s Bob’s hard drive A collection of files in our “GIS Folder” A stand-alone spatial database The data back-end for our internet mapping application An enterprise GIS

7. 7 Digital Preservation Points of Failure Data is not saved, or … can’t be found, or … media is obsolete, or … media is corrupt, or … format is obsolete, or … file is corrupt, or … meaning is lost Solutions: MigrationEmulation EncapsulationXML

8. 8 Risks to Geospatial Data Producer focus on current data Data overwrite as common practice Future support of data formats in question No open, supported format for vector data Shift to web services-based access Data becoming more ephemeral Inadequate or nonexistent metadata Impedes discovery and use Increasing use of spatial databases for data management The whole is greater than the sum of the parts

9. Note: Percentages based on the actual number of respondents to each question 9 Value in Older Geospatial Data

10. 10 Value in Older Data: Cultural Heritage Future uses of data are difficult to anticipate (as with Sanborn Maps)

11. 11 Value in Older Data: Solving Business Problems Suburban Development 1993/2002 Near Mecklenburg-Cabarrus County border Land use change analysis Real estate trends analysis Site location analysis Disaster response Resolution of legal challengesImpervious surface maps

12. 12 Problem: Flood and Hurricane Preparedness

13. 13 Application: Impervious Surface Change Mapping A.B. C. D. 2002 Impervious 2004 Aerial Photography 2004 Impervious using 2002 Mask 2004 Impervious Update

14. 14 Problem: Beach Erosion and Shoreline Change

15. 15 Application: Shoreline Change Mapping

16. 16 Problem: Tracking Land Use Change

17. 17 Application: Land Use Change Mapping Input Data Output GIS Data Using Mecklenburg County 2002 true color orthorectified aerial photography Developing Areas

18. Note: Percentages based on the actual number of respondents to each question 18 Preservation Challenges

19. 19 Challenge: Vector Data Formats No widely-supported, open vector formats for geospatial data Spatial Data Transfer Standard (SDTS) not widely supported Geography Markup Language (GML) – diversity of application schemas and profiles a challenge for “permanent access” Spatial Databases The whole is more than the sum of the parts, and the whole is very difficult to preserve Can export individual data layers for curation, but relationships and context are lost Some thinking of using the spatial database as the primary archival platform

20. 20 Challenge: Cartographic Representation Counterpart to the map is not just the dataset but also models, symbolization, classification, annotation, etc.

21. 21 Challenge: Geospatial Web Services How to capture records from decision- making processes?

22. 22 Challenge: Preservation Metadata Results from a 2006 survey of all 100 NC counties and 25 largest NC municipalities

23. 23 Challenge: Data Capture Response: yes = 65.3%, no = 34.7%* (out of 57.6% response rate) 2006 Frequency of Capture Survey targeting North Carolina counties and municipalities

24. Note: Percentages based on the actual number of respondents to each question 24 Challenge: Digital Object Complexity

25. Note: Percentages based on the actual number of respondents to each question 25 Where is the Dataset?

26. Note: Percentages based on the actual number of respondents to each question 26 Here’s One! Files Multi-file dataset Georeferencing Metadata file Symbolization file Additional documentation License Disclaimer More Metadata FGDC Acquisition metadata Transfer metadata Ingest metadata Archive rights Archive processes Collection metadata Series metadata

27. Note: Percentages based on the actual number of respondents to each question 27 Other Challenges Rights management Data versioning Semantic issues Large scale content transfer Integrating older analog data More …

28. Note: Percentages based on the actual number of respondents to each question 28 Different Ways to Approach Preservation Technical solutions: How do we preserve acquired content over the long term? Cultural/Organizational solutions: How do we make the data more preservable—and more prone to be preserved— from point of production? Current use and data sharing requirements – not archiving needs – are most likely to drive improved preservability of content and improvement of metadata

29. Note: Percentages based on the actual number of respondents to each question 29 Content Identification and Selection Issues

30. Note: Percentages based on the actual number of respondents to each question 30 What do Inventories (e.g. RAMONA) Offer to Archives? Data Availability Information Detailed information by data layer Contact Information Minimal Metadata Descriptive, technical, administrative Rights Information Document Technical Environment Software used, formats, transfer methods Future Data Development Plans

31. Note: Percentages based on the actual number of respondents to each question 31 Selection Issues Most content is already at some level of risk Early-Middle-Late Stage issues Middle stage is usually the “sweet spot”, e.g. TIFF orthophotos vs. raw images or compressed images Also added-value products: digital maps, cartographic representation Digital maps: “record” or not? Frequency of capture

32. Note: Percentages based on the actual number of respondents to each question 32 Problem: Multiple choice for: format type, coordinate system, tiling scheme

33. 33 Geospatial Data Types – Spatial Databases Vector and raster data Relationships Behaviors Annotation Data Models

34. Note: Percentages based on the actual number of respondents to each question 34 GIS Software Software project file (.mxd,.apr, …) Data layer file (.avl,.lyr, …) PDF map exports Web Services-based representations Geospatial Data Types – Cartographic

35. Note: Percentages based on the actual number of respondents to each question 35 Mobile, LBS, and, social networking applications Long-term cultural heritage value in non-overhead imagery: more descriptive of place and function Oblique Imagery Road Videologs Tax Dept. Photos Street View Images Other Geospatial Data Types – Place-based Data

36. Note: Percentages based on the actual number of respondents to each question 36 Time series – vector data Parcel Boundary Changes 2001-2004, North Raleigh, NC Continuously updated data: Frequency of snapshots? Different for various framework layers?

37. Note: Percentages based on the actual number of respondents to each question 37 Sept. 2006 Frequency of Capture Survey Survey objective: Document current practices for obtaining archival snapshots of county/municipal geospatial vector data layers Seek guidance about frequency of capture Survey topics: General questions about data archiving practice Specific questions about parcels, street centerlines, jurisdictional boundaries, and zoning Survey subjects: All 100 counties and 25 municipalities 58% response rate Survey conducted September 2006

38. Note: Percentages based on the actual number of respondents to each question 38 Data Capture Survey Results: Overview Two-thirds of responding agencies create and retain periodic snapshots Long-term retention more common in counties with larger populations Storage environments vary, with servers and CD- ROMs most common Offsite storage (or both onsite and offsite) is used by nearly half of the respondents Popularity of historic images has resulted in scanning and geo-referencing of hardcopy aerial photos among one-third of the respondents

39. Note: Percentages based on the actual number of respondents to each question 39 Survey Observations Process of survey formulation and implementation helped to socialize the problem of archiving data Local innovation needs to be mined further to inform development of best practices Business drivers for archiving need more study (e.g., stated adherence to retention policy) Exposure to peer practice encourages archiving Pronounced local interest in scanning/rectifying older analog maps and imagery

40. Note: Percentages based on the actual number of respondents to each question 40 Engaging Industry

41. Note: Percentages based on the actual number of respondents to each question 41 Framework data communities Snapshot frequency, naming schemes, classification, GML application schemas, format strategies Metadata standards and outreach Persistent identifiers, versioning, feedback on metadata quality Content exchange networks/content replication For data improvement projects, disaster preparedness, aggregation by regional service providers, … and archives Where does archiving and preservation fit in? Points of Engagement with Spatial Data Infrastructure (e.g. NC OneMap)

42. Note: Percentages based on the actual number of respondents to each question 42 Content Exchange Infrastructure High volume of state/federal requests for local data Solving the present-day problems of data sharing is a pre-requisite to solving the problem of long-term access Leveraging more compelling business reasons to put the data in motion (disaster preparedness, business continuity, highway construction, census, …) Content exchange networks: Minimize need to make contact Add technical, administrative, descriptive metadata Establish rights and provenance

43. Note: Percentages based on the actual number of respondents to each question 43 Archives Processes

44. Note: Percentages based on the actual number of respondents to each question 44 Retention schedules Geospatial data Administrative records Record accessioning Appraisal system System documentation Archival data and metadata standards Rules for disposition of local government records Maine GeoArchives Project Components

45. Note: Percentages based on the actual number of respondents to each question 45 Compliance Responsible Credibility Completeness Authenticity Soundness Maine GeoArchives: Functional Requirements Auditability Availability Exportable Renderable Redactable Adopted set of functional requirements for recordkeeping systems to insure permanent retention of data layers

46. Note: Percentages based on the actual number of respondents to each question 46 Conclusion

47. Note: Percentages based on the actual number of respondents to each question 47 What are the points of intersection between archive needs and business continuity/disaster preparedness and other business needs? How to best stimulate and learn from innovation at the state/regional/local level? How to make data more preservable from point of production and on through data transfer How to most effectively move data in an efficient, well- documented manner with clarified rights How to best make State Archives a part of spatial data infrastructure? Defining the record: data vs. derivative components Key issues

48. Note: Percentages based on the actual number of respondents to each question 48 Cultural: Changing Industry Thinking Is the geospatial industry “temporally-impaired?” Lack of access to older data Lack for tool/model support for temporal analysis Metadata: poor support for changing data Education: building class projects around available data (i.e., not temporal) Increased interest now in temporal applications? Increased demand for temporal data? Improved tool support: ArcGIS 9.2 animation tools; Geodatabase History, etc.

49. Note: Percentages based on the actual number of respondents to each question 49 Questions? Contact: Steve Morris Head, Digital Library Initiatives NCSU Libraries ph: (919) 515-1361 Steven_Morris@ncsu.edu http://www.lib.ncsu.edu/ncgdap