Reference Model for an Open Archival Information System (OAIS) ESIP Summer Meeting John Garrett – ADNET Systems at NASA/GSFC ESIP Summer Meeting John Garrett – ADNET Systems at NASA/GSFC

Topics (time permitting) OAIS Reference Model Follow-on/Related Standards –Producer-Archive Interface Methodology Abstract Standard (PAIMAS) –Repository Audit and Certification Metrics (draft) Requirements for Bodies Providing Audits (draft) –Producer-Archive Interface Specification (PAIS) (draft) –XML Formatted Data Unit (XFDU) Contributors: Don Sawyer, Daniele Boucon, Lou Reich, David Giaretta and many others involved in CCSDS Archiving and Packaging standards development OAIS Reference Model Follow-on/Related Standards –Producer-Archive Interface Methodology Abstract Standard (PAIMAS) –Repository Audit and Certification Metrics (draft) Requirements for Bodies Providing Audits (draft) –Producer-Archive Interface Specification (PAIS) (draft) –XML Formatted Data Unit (XFDU) Contributors: Don Sawyer, Daniele Boucon, Lou Reich, David Giaretta and many others involved in CCSDS Archiving and Packaging standards development

OAIS Reference Model Home Consultative Committee for Space Data Systems (CCSDS) International group of space agencies Develop variety of science discipline-independent standards Became working body for an ISO TC 20/ SC 13 about 1990 TC20: Aircraft and Space Vehicles SC13: Space Data and Information Transfer Systems – –Ensured broad participation, including traditional archives, libraries, companies (Not restricted to space communities; all participation was welcomed!) Consultative Committee for Space Data Systems (CCSDS) International group of space agencies Develop variety of science discipline-independent standards Became working body for an ISO TC 20/ SC 13 about 1990 TC20: Aircraft and Space Vehicles SC13: Space Data and Information Transfer Systems – –Ensured broad participation, including traditional archives, libraries, companies (Not restricted to space communities; all participation was welcomed!)

OAIS RM 1.Negotiating and accepting information 2.Obtaining sufficient control of the information to ensure long-term preservation 3.Determining the "designated community" 4.Ensuring that information is independently understandable 5.Following documented policies and procedures 6.Making the preserved information available OAIS Information ModelOAIS Mandatory Responsibilities: OAIS Functional Model OAIS Environment and Data Flows In the Beginning: OAIS Reference Model CCSDS B-1 Reference Model for an Open Archival Information System (OAIS) (ISO 14721:2003)

Negotiates and accepts information from information producers Obtains sufficient control to ensure long-term preservation Determines which communities (designated) need to be able to understand the preserved information Ensures the information to be preserved is independently understandable to the Designated Communities Follows documented policies and procedures that ensure the information is preserved against all reasonable contingencies Makes the preserved information available to the Designated Communities in forms understandable to those communities Negotiates and accepts information from information producers Obtains sufficient control to ensure long-term preservation Determines which communities (designated) need to be able to understand the preserved information Ensures the information to be preserved is independently understandable to the Designated Communities Follows documented policies and procedures that ensure the information is preserved against all reasonable contingencies Makes the preserved information available to the Designated Communities in forms understandable to those communities OAIS Responsibilities

Archival Information Package (AIP) Content Information Preservation Description Information (PDI) e.g., Hardcopy document Document as an electronic file together with its format description Scientific data set consisting of image file, text file, and format descriptions file describing the other files e.g., How the Content Information came into being, who has held it, how it relates to other information, and how its integrity is assured OAIS Archival Information Package Packaging Information Package Description further described by delimited by derived from e.g., How to find Content information and PDI on some medium e.g., Information supporting customer searches for AIP

Preservation Description Information (PDI) Reference Information –Provides one or more identifiers, or systems of identifiers, by which the Content Information may be uniquely identified –Bibliographic Description, Persistent IDs Provenance Information –Describes the source of Content Information, who has had custody of it, what is its history –Logs of migrations Context Information –Describes how the Content Information relates to other information outside the Information Package –Pointers to related collections Fixity Information –Protects the Content Information from undocumented alteration –Digital signatures, Checksums Reference Information –Provides one or more identifiers, or systems of identifiers, by which the Content Information may be uniquely identified –Bibliographic Description, Persistent IDs Provenance Information –Describes the source of Content Information, who has had custody of it, what is its history –Logs of migrations Context Information –Describes how the Content Information relates to other information outside the Information Package –Pointers to related collections Fixity Information –Protects the Content Information from undocumented alteration –Digital signatures, Checksums

View of an OAIS Environment OAIS (archive) Management ProducerConsumer Producer provides the information to be preserved Management sets overall OAIS policy Consumer seeks and acquires preserved information of interest

OAIS Functional Entities SIP = Submission Information Package AIP = Archival Information Package DIP = Dissemination Information Package SIP Descriptive Info. AIP DIP Administration PRODUCERPRODUCER CONSUMERCONSUMER queries result sets MANAGEMENT Ingest Access Data Management Archival Storage Descriptive Info. Preservation Planning orders

Producer Consumer queries result sets orders OAIS Archival Information Packages External Data Flow View Submission Information Packages Dissemination Information Packages

Conformance How does an archive conform? –It discharges the set of minimal responsibilities –It supports the basic information concepts that address a definition of information and types of information packages How do other documents conform? –By using OAIS terms and concepts Certification Standard in progress How does an archive conform? –It discharges the set of minimal responsibilities –It supports the basic information concepts that address a definition of information and types of information packages How do other documents conform? –By using OAIS terms and concepts Certification Standard in progress

OAIS Update Many improvements including: Authenticity Information properties Risk management Emulation Federation Many improvements including: Authenticity Information properties Risk management Emulation Federation

SIP = Submission Information Package SIP DIP Administration PRODUCERPRODUCER CONSUMERCONSUMER queries result sets MANAGEMENT Ingest Access Data Management Archival Storage Descriptive Info. Preservation Planning orders AIP AIP = Archival Information Package DIP = Dissemination Information Package PAIMAS Focus Producer-Archive Interface Methodology Abstract Standard

PAIMAS Methodology The Archive Project is broken into 4 main phases: Preliminary Phase, Formal Definition Phase, Transfer Phase, Validation Phase. PAIMAS identifies: the phases in the process of transferring information, the objective of the phases, Extensive action tables of actions that must be carried out, the expected results. PAIMAS is a basis: for further specialization by a particular community for the identification of standards and implementation guides, for identification and development of a set of software tools. CCSDS B-1 Producer-Archive Interface Methodology Abstract Standard. (ISO 20652:2006)

Data ready to archive PAIMAS phases & relationships Preliminary Agreement Submission Agreement including Dictionary and Formal Model Transferred object files Validation agreement Phase objective Preliminary Phase Formal Definition Phase Transfer Phase Validation Phase Anomalies Validate the transferred objects Define the information to be archived resources estimation Develop agreement (data to be delivered, complementary elements, schedule) Actual transfer of the objects

Preliminary phase: sub-phases Action table Description

Repository Audit and Certification - Metrics Closing in on public draft that will be submitted to CCSDS and ISO Builds on previous audit work by TRAC and many others Closing in on public draft that will be submitted to CCSDS and ISO Builds on previous audit work by TRAC and many others INCLUDED TOPICS ORGANISATIONAL INFRASTRUCTURE GOVERNANCE & ORGANIZATIONAL VIABILITY ORGANIZATIONAL STRUCTURE & STAFFING PROCEDURAL ACCOUNTABILITY &.PRESERVATION POLICY FRAMEWORK FINANCIAL SUSTAINABILITY CONTRACTS, LICENSES, & LIABILITIES DIGITAL OBJECT MANAGEMENT INGEST: ACQUISITION OF CONTENT INGEST: CREATION OF THE AIP PRESERVATION PLANNING AIP PRESERVATION INFORMATION MANAGEMENT ACCESS MANAGEMENT INFRASTRUCTURE AND SECURITY RISK MANAGEMENT Technical Infrastructure Risk Management Security risk management The repository shall verify each AIP for completeness and correctness at the point it is created. Supporting Text This is necessary in order to ensure that what is maintained over the long term is as it should be and can be traced to the information provided by the Producers. Examples of Ways the Repository can Demonstrate it is Meeting this Requirement Description of the procedure that verifies completeness and correctness of the AIPs; logs of the procedure. Discussion The repository should be sure that the AIPs it creates are as they are expected to be by …

18 PAIS Objectives Producer-Archive Interface Specification Provide formal modelling of data objects that are to be transferred from Producer to Archives –XML-based interchange of the model and SIPs Implementation standard for Producer – Archive Interface –Conformity with the OAIS Reference Model –Conformity with the PAIMAS –Conformity with the XFDU Aimed mainly at Formal Definition Phase with applicability to Transfer Phase with Validation Closing in on public draft that will be submitted to CCSDS and ISO possibly by end of year Producer-Archive Interface Specification Provide formal modelling of data objects that are to be transferred from Producer to Archives –XML-based interchange of the model and SIPs Implementation standard for Producer – Archive Interface –Conformity with the OAIS Reference Model –Conformity with the PAIMAS –Conformity with the XFDU Aimed mainly at Formal Definition Phase with applicability to Transfer Phase with Validation Closing in on public draft that will be submitted to CCSDS and ISO possibly by end of year

19 PAIS Basic Entities

XFDU Packaging Standard Rationale Technology and Requirements Evolution Physical media -->Electronic Transfer No standard language for metadata--> XML Homogeneous Remote Procedure Call-->CORBA, SOAP Little understanding of long-term preservation-->OAIS RM Record formats-->Self describing data formats New Requirements Describe multiple encodings of a data object Better describe the relationships among a set of data objects. Technology and Requirements Evolution Physical media -->Electronic Transfer No standard language for metadata--> XML Homogeneous Remote Procedure Call-->CORBA, SOAP Little understanding of long-term preservation-->OAIS RM Record formats-->Self describing data formats New Requirements Describe multiple encodings of a data object Better describe the relationships among a set of data objects. Use of XML based technologies Designed to be extensible to include new XML technologies as they emerge Linkage of data and software Direct mapping to OAIS Information Models Support both media and network exchange Support for multiple encoding/compression on individual objects or on entire package Mapping to current SFDU Packaging & Data Description Metadata where possible Maximal use of existing standards and tools from similar efforts Technical Drivers

XFDU Conceptual View CCSDS B-1 XML Formatted Data Unit (XFDU) Structure and Construction Rules. (ISO 13527:2009) Open source XFDU Toolkit Library developed as a reference implementation ( available at: Interoperability testing completed with ESA XFDU Implementation Partnered with JPL/PDS to establish a NASA Testbed XFDU Briefing on Scalability at the Collaborative Expedition Workshop / Toward Scalable Data Management (available at: bin/wiki.pl?ExpeditionWorkshop/TowardScalableDataManagement_2008_06_10) bin/wiki.pl?ExpeditionWorkshop/TowardScalableDataManagement_2008_06_10