1. Reference Model for an Open Archival Information System (OAIS) ESIP Summer Meeting John Garrett – ADNET Systems at NASA/GSFC 2009-07-09 ESIP Summer Meeting John Garrett – ADNET Systems at NASA/GSFC 2009-07-09

2. Topics (time permitting) OAIS Reference Model Follow-on/Related Standards –Producer-Archive Interface Methodology Abstract Standard (PAIMAS) –Repository Audit and Certification Metrics (draft) Requirements for Bodies Providing Audits (draft) –Producer-Archive Interface Specification (PAIS) (draft) –XML Formatted Data Unit (XFDU) Contributors: Don Sawyer, Daniele Boucon, Lou Reich, David Giaretta and many others involved in CCSDS Archiving and Packaging standards development OAIS Reference Model Follow-on/Related Standards –Producer-Archive Interface Methodology Abstract Standard (PAIMAS) –Repository Audit and Certification Metrics (draft) Requirements for Bodies Providing Audits (draft) –Producer-Archive Interface Specification (PAIS) (draft) –XML Formatted Data Unit (XFDU) Contributors: Don Sawyer, Daniele Boucon, Lou Reich, David Giaretta and many others involved in CCSDS Archiving and Packaging standards development

3. OAIS Reference Model Home Consultative Committee for Space Data Systems (CCSDS) International group of space agencies Develop variety of science discipline-independent standards Became working body for an ISO TC 20/ SC 13 about 1990 TC20: Aircraft and Space Vehicles SC13: Space Data and Information Transfer Systems –http://www.ccsds.org/ –Ensured broad participation, including traditional archives, libraries, companies (Not restricted to space communities; all participation was welcomed!) Consultative Committee for Space Data Systems (CCSDS) International group of space agencies Develop variety of science discipline-independent standards Became working body for an ISO TC 20/ SC 13 about 1990 TC20: Aircraft and Space Vehicles SC13: Space Data and Information Transfer Systems –http://www.ccsds.org/ –Ensured broad participation, including traditional archives, libraries, companies (Not restricted to space communities; all participation was welcomed!)

4. OAIS RM 1.Negotiating and accepting information 2.Obtaining sufficient control of the information to ensure long-term preservation 3.Determining the "designated community" 4.Ensuring that information is independently understandable 5.Following documented policies and procedures 6.Making the preserved information available OAIS Information ModelOAIS Mandatory Responsibilities: OAIS Functional Model OAIS Environment and Data Flows In the Beginning: OAIS Reference Model CCSDS 650.0-B-1 Reference Model for an Open Archival Information System (OAIS) (ISO 14721:2003) http://public.ccsds.org/publications/archive/650x0b1.pdf

5. Negotiates and accepts information from information producers Obtains sufficient control to ensure long-term preservation Determines which communities (designated) need to be able to understand the preserved information Ensures the information to be preserved is independently understandable to the Designated Communities Follows documented policies and procedures that ensure the information is preserved against all reasonable contingencies Makes the preserved information available to the Designated Communities in forms understandable to those communities Negotiates and accepts information from information producers Obtains sufficient control to ensure long-term preservation Determines which communities (designated) need to be able to understand the preserved information Ensures the information to be preserved is independently understandable to the Designated Communities Follows documented policies and procedures that ensure the information is preserved against all reasonable contingencies Makes the preserved information available to the Designated Communities in forms understandable to those communities OAIS Responsibilities

6. Archival Information Package (AIP) Content Information Preservation Description Information (PDI) e.g., Hardcopy document Document as an electronic file together with its format description Scientific data set consisting of image file, text file, and format descriptions file describing the other files e.g., How the Content Information came into being, who has held it, how it relates to other information, and how its integrity is assured OAIS Archival Information Package Packaging Information Package Description further described by delimited by derived from e.g., How to find Content information and PDI on some medium e.g., Information supporting customer searches for AIP

7. Preservation Description Information (PDI) Reference Information –Provides one or more identifiers, or systems of identifiers, by which the Content Information may be uniquely identified –Bibliographic Description, Persistent IDs Provenance Information –Describes the source of Content Information, who has had custody of it, what is its history –Logs of migrations Context Information –Describes how the Content Information relates to other information outside the Information Package –Pointers to related collections Fixity Information –Protects the Content Information from undocumented alteration –Digital signatures, Checksums Reference Information –Provides one or more identifiers, or systems of identifiers, by which the Content Information may be uniquely identified –Bibliographic Description, Persistent IDs Provenance Information –Describes the source of Content Information, who has had custody of it, what is its history –Logs of migrations Context Information –Describes how the Content Information relates to other information outside the Information Package –Pointers to related collections Fixity Information –Protects the Content Information from undocumented alteration –Digital signatures, Checksums

8. View of an OAIS Environment OAIS (archive) Management ProducerConsumer Producer provides the information to be preserved Management sets overall OAIS policy Consumer seeks and acquires preserved information of interest

9. OAIS Functional Entities SIP = Submission Information Package AIP = Archival Information Package DIP = Dissemination Information Package SIP Descriptive Info. AIP DIP Administration PRODUCERPRODUCER CONSUMERCONSUMER queries result sets MANAGEMENT Ingest Access Data Management Archival Storage Descriptive Info. Preservation Planning orders

10. Producer Consumer queries result sets orders OAIS Archival Information Packages External Data Flow View Submission Information Packages Dissemination Information Packages

11. Conformance How does an archive conform? –It discharges the set of minimal responsibilities –It supports the basic information concepts that address a definition of information and types of information packages How do other documents conform? –By using OAIS terms and concepts Certification Standard in progress How does an archive conform? –It discharges the set of minimal responsibilities –It supports the basic information concepts that address a definition of information and types of information packages How do other documents conform? –By using OAIS terms and concepts Certification Standard in progress

12. OAIS Update Many improvements including: Authenticity Information properties Risk management Emulation Federation Many improvements including: Authenticity Information properties Risk management Emulation Federation

13. SIP = Submission Information Package SIP DIP Administration PRODUCERPRODUCER CONSUMERCONSUMER queries result sets MANAGEMENT Ingest Access Data Management Archival Storage Descriptive Info. Preservation Planning orders AIP AIP = Archival Information Package DIP = Dissemination Information Package PAIMAS Focus Producer-Archive Interface Methodology Abstract Standard

14. PAIMAS Methodology The Archive Project is broken into 4 main phases: Preliminary Phase, Formal Definition Phase, Transfer Phase, Validation Phase. PAIMAS identifies: the phases in the process of transferring information, the objective of the phases, Extensive action tables of actions that must be carried out, the expected results. PAIMAS is a basis: for further specialization by a particular community for the identification of standards and implementation guides, for identification and development of a set of software tools. CCSDS 651.0-B-1 Producer-Archive Interface Methodology Abstract Standard. (ISO 20652:2006) http://public.ccsds.org/publications/archive/651x0b1.pdf

15. Data ready to archive PAIMAS phases & relationships Preliminary Agreement Submission Agreement including Dictionary and Formal Model Transferred object files Validation agreement Phase objective Preliminary Phase Formal Definition Phase Transfer Phase Validation Phase Anomalies Validate the transferred objects Define the information to be archived resources estimation Develop agreement (data to be delivered, complementary elements, schedule) Actual transfer of the objects

16. Preliminary phase: sub-phases Action table Description

17. Repository Audit and Certification - Metrics http://wiki.digitalrepositoryauditandcertification.org/bin/view Closing in on public draft that will be submitted to CCSDS and ISO Builds on previous audit work by TRAC and many others http://wiki.digitalrepositoryauditandcertification.org/bin/view Closing in on public draft that will be submitted to CCSDS and ISO Builds on previous audit work by TRAC and many others INCLUDED TOPICS ORGANISATIONAL INFRASTRUCTURE GOVERNANCE & ORGANIZATIONAL VIABILITY ORGANIZATIONAL STRUCTURE & STAFFING PROCEDURAL ACCOUNTABILITY &.PRESERVATION POLICY FRAMEWORK FINANCIAL SUSTAINABILITY CONTRACTS, LICENSES, & LIABILITIES DIGITAL OBJECT MANAGEMENT INGEST: ACQUISITION OF CONTENT INGEST: CREATION OF THE AIP PRESERVATION PLANNING AIP PRESERVATION INFORMATION MANAGEMENT ACCESS MANAGEMENT INFRASTRUCTURE AND SECURITY RISK MANAGEMENT Technical Infrastructure Risk Management Security risk management 4.2.8 The repository shall verify each AIP for completeness and correctness at the point it is created. Supporting Text This is necessary in order to ensure that what is maintained over the long term is as it should be and can be traced to the information provided by the Producers. Examples of Ways the Repository can Demonstrate it is Meeting this Requirement Description of the procedure that verifies completeness and correctness of the AIPs; logs of the procedure. Discussion The repository should be sure that the AIPs it creates are as they are expected to be by …

18. 18 PAIS Objectives Producer-Archive Interface Specification Provide formal modelling of data objects that are to be transferred from Producer to Archives –XML-based interchange of the model and SIPs Implementation standard for Producer – Archive Interface –Conformity with the OAIS Reference Model –Conformity with the PAIMAS –Conformity with the XFDU Aimed mainly at Formal Definition Phase with applicability to Transfer Phase with Validation Closing in on public draft that will be submitted to CCSDS and ISO possibly by end of year Producer-Archive Interface Specification Provide formal modelling of data objects that are to be transferred from Producer to Archives –XML-based interchange of the model and SIPs Implementation standard for Producer – Archive Interface –Conformity with the OAIS Reference Model –Conformity with the PAIMAS –Conformity with the XFDU Aimed mainly at Formal Definition Phase with applicability to Transfer Phase with Validation Closing in on public draft that will be submitted to CCSDS and ISO possibly by end of year

19. 19 PAIS Basic Entities

20. XFDU Packaging Standard Rationale Technology and Requirements Evolution Physical media -->Electronic Transfer No standard language for metadata--> XML Homogeneous Remote Procedure Call-->CORBA, SOAP Little understanding of long-term preservation-->OAIS RM Record formats-->Self describing data formats New Requirements Describe multiple encodings of a data object Better describe the relationships among a set of data objects. Technology and Requirements Evolution Physical media -->Electronic Transfer No standard language for metadata--> XML Homogeneous Remote Procedure Call-->CORBA, SOAP Little understanding of long-term preservation-->OAIS RM Record formats-->Self describing data formats New Requirements Describe multiple encodings of a data object Better describe the relationships among a set of data objects. Use of XML based technologies Designed to be extensible to include new XML technologies as they emerge Linkage of data and software Direct mapping to OAIS Information Models Support both media and network exchange Support for multiple encoding/compression on individual objects or on entire package Mapping to current SFDU Packaging & Data Description Metadata where possible Maximal use of existing standards and tools from similar efforts Technical Drivers

21. XFDU Conceptual View CCSDS 661.0-B-1 XML Formatted Data Unit (XFDU) Structure and Construction Rules. (ISO 13527:2009) http://public.ccsds.org/publications/archive/661x0b1.pdf Open source XFDU Toolkit Library developed as a reference implementation ( available at: http://sindbad.gsfc.nasa.gov/xfduhttp://sindbad.gsfc.nasa.gov/xfdu Interoperability testing completed with ESA XFDU Implementation Partnered with JPL/PDS to establish a NASA Testbed XFDU Briefing on Scalability at the Collaborative Expedition Workshop / Toward Scalable Data Management (available at: http://colab.cim3.net/cgi- bin/wiki.pl?ExpeditionWorkshop/TowardScalableDataManagement_2008_06_10)http://colab.cim3.net/cgi- bin/wiki.pl?ExpeditionWorkshop/TowardScalableDataManagement_2008_06_10