North Carolina Geospatial Data Archiving Project (NCGDAP) JISC/NDIIPP Joint Digital Preservation Workshop – May 2006 Presented by: Rob Farrell, Steve Morris, and Jim Tuttle Issue Local government agency data resources usually lack metadata. Solution Gather available metadata details from GIS inventories and capture technical and administrative details during the data transfer process. Future Scenarios The NC One Map metadata outreach program is expanding state-wide. The new RAMONA survey tool includes the rapid development of basic metadata elements online. Project Phases Overview Review existing inventories of state, regional, and local agency data holdings Target data for acquisition, focusing on GIS vector data, digital orthophotography, digital maps, and georeferenced tabular data Determine data acquisition frequency for continuously updated data layers such as land parcels and municipal boundaries Determine optimal data formats for preservation and develop plans for archiving unique spatial data formats Acquire state, local, and regional agency data as well as federal agency products of local scope Capture, process, and normalize existing FGDC metadata Use the Metadata Encoding and Transfer Standard (METS) to form preservation metadata records that include descriptive, administrative, and technical components Explore Open Geospatial Consortium specifications and technologies for potential use in automated identification and capture of data Work within the NC One Map framework of inventories, data sharing agreements, and technical best practices Explore state, local, and federal partnerships in developing strategies for digital preservation infrastructure development Advisory Committee formed from elements of NC Geographic Information Coordinating Council subcommittees Data AcquisitionDigital Repository Development Data Selection and Identification Partnership Building Ingest data into DSpace open source digital repository software for bit-level preservation Develop redundant archive, with offsite storage Implement format migration strategy for at-risk data resources Explore longer-term archival approaches with state, federal, and local partners Explore options for exposure of time-versioned and archived content through access and discovery tools NCGDAP Context: Long-Term Risks to Geospatial Data Media Failure or Obsolescence - old or unreadable tapes Format Obsolescence - how long will formats be supported? Absence of Metadata - insufficient documentation to support use of the data Naming and Coding Inconsistency - data layer names, attributes, and coding schemes vary across agencies Permanent Access Not Available - data distribution focuses on current copies Shift Towards Web Services-based Distribution - threat to formation of secondary archives (libraries, etc.) The North Carolina State University Libraries and the North Carolina Center for Geographic Information and Analysis have partnered on a three-year project to collect and preserve at-risk digital geospatial data resources from state and local government agencies. This project draws on a cooperative agreement with the Library of Congress in connection with NDIIPP as one of eight Collaborative Collection Development Partnerships. Although this venture will focus solely on North Carolina, it is expected to serve as a demonstration project for other states. The project is closely tied to NC OneMap, a combined state, federal, and local initiative that is focused on facilitating seamless access to geospatial data across all levels of government. One of the stated goals of NC OneMap is that “Historic and temporal data will be maintained and available." Project Introduction Mitigating Community Practice Risks – 3 examples Project Introduction, Context, and Phases Overview Mitigating Technological Risks – 3 examples Issue Most geospatial data sets are multi-file objects involving multiple formats. Solution Group files as single logical item during ingest workflow. Use METS for distribution packaging. Future Scenarios The GeoDRM initiative may stimulate geospatial industry attention to the issue of content packaging. A Single Vector Layer = Layer Components Geometry file Index file Attribute file Metadata file Coordinate System file Spatial Index files Spatial Database Image downloaded from USGS seamless data download Raster Data Vector Data Topological Relationships XML Database Export TIFF Images Feature Class Layers Potential Ingest Objects Issue Spatial databases have become an increasingly common data management environment. These complex systems include components that are difficult to preserve. Solution Archive data layer exports, raster exports, XML representation of the database components, and the original spatial database. Future Scenarios Spatial database technical directions being pursued by vendors may make it more feasible to preserve spatial databases in their entirety. Issue There is no open, widely-supported format for geospatial vector data. Solution Settle for now on the shapefile as an openly documented, easily-archived, and widely supported format. Future Scenarios Initial discussions regarding an archival profile of the Geography Markup Language (GML) are underway, involving the Open Geospatial Consortium, ESRI, and NARA. ISO Local Data Dictionary FGDC Industry Profiles Issue Metadata may occur in html, xml, text, or pdf and may adhere to a broad content standard such as ISO or FGDC, an industry profile, or a locally defined scheme. Solution Normalize the format to an XML representation of a commonly utilized profile of the FGDC content standard and retain original metadata record. Future Scenarios The finalization and adoption of the North American Profile of the ISO metadata encoding standard will promote industry-wide consistency in metadata records. Issue Many geospatial data layers are volatile in nature, and may be updated daily or weekly. Solution Capture snapshots of data layers at defined intervals and encourage maintenance of snapshots by data custodians. Future Scenarios Preservation partners will conduct a survey to determine optimal data capture frequency for each framework data layer. State Archives retention schedule processes may provide a means to formalize the capture process. Parcel delineation in northern Raleigh March 1997 Parcel delineation in northern Raleigh February 2005 Metadata Encodings and Formats Results from 2003 North Carolina local government survey