Prepared by: Lou Reich (CSC) and Don Sawyer (NASA) ISO “Reference Model For an Open Archival Information System (OAIS)” Tutorial Presentation for DADs Workshop Prepared by: Lou Reich (CSC) and Don Sawyer (NASA) June 22, 1998 1

Overview of Tutorial Background Information OAIS Purpose, Scope and Applicability OAIS Concepts OAIS Responsibilities OAIS Detailed Reference Model – Archival Information Model – Functional Model Analysis of Archive Issues Using OAIS RM – Archive Associations – Migration Summary

Background Information

Genesis of the Effort (1) The international context : Multiplication of data holding sites, Rapidly growing needs for the long term preservation of digital data Incomplete understanding of the digital preservation issues Constant technological changes and lack of standards in the area of digital archiving Initial framework : ISO Technical Committee (TC) 20, Aircraft and Space Vehicles, and its Sub-Committee (SC) 13: Space Data and Information Transfer Systems proposal : Define an Archive reference model and its Services categories Address data used in conjunction with space missions Address intermediate and indefinite long term storage of digital data First step before developing specific standards needed to support archive services In connection with the rapid decrease of electronic storage cost and the rapid increase in netwokf connectivity and bandwith 2

Genesis of the Effort (2) Proposal made to Consultative Committee for Space Data Systems (CCSDS) and ISO TC 20/SC 13 Develop a ‘Reference Model’ to establish common terms and concepts Ensure broad participation, including traditional archives (Not restricted to space communities; all participation is welcome!) Focus on data in electronic forms, but recognize that other forms exist in most archives Follow up with additional archive standards efforts as appropriate Impact of both CCSDS and ISO procedures The CCSDS reference model will become an ISO standard

Status of the Effort (Updated 1998-10-10) Reference Model will be submitted as a draft international standard in December, 1998 Current version is White Book 4.0, available under Reference Materials heading, at: http://ssdoo.gsfc.nasa.gov/nost/isoas/overview.html Widest possible exposure is desirable now Participation is still welcome Full CCSDS and ISO standard by June, 1999

Open Archival Information System (OAIS) Reference Model standard(s) are developed using a public process and are freely available Information Any type of knowledge that can be exchanged Independent of the forms (i.e., physical or digital) used to represent the information Data are the representation forms of information Archival Information System Hardware, software, and people who are responsible for the acquisition, preservation and dissemination of the information Additional OAIS responsibilities are identified later and are more fully defined in the Reference Model document This reference model is called : Reference Model for an OAIS (Open Archival Information System) These tems have to be explained : In the framework of the archiving reference model elaboration, a glossary has been defined. This is a crucial point for the common understandding of archival concepts 4

Document Organization Introduction Purpose and Scope, Applicability, Rationale, Road Map for Future Work, Document Structure, and Definitions of Terms OAIS Concepts High level view of OAIS functionality and information models OAIS external environment Minimum responsibilities to become an “OAIS” Detailed Models Functional model descriptions and information model perspectives Migration perspectives Media migration, compression, and format conversions Archive Cooperation Criteria to distinguish types of cooperation among archives Annexes Scenarios of existing archives, compatibility with other standards The current version ot the Reference model document is organised as follow The parts in blue will no detailed to day 6

Purpose, Scope and Applicability

Purpose Framework for understanding and applying concepts needed for long-term digital information preservation Long-term is long enough to be concerned about changing technologies Starting point for model addressing non-digital information Provides set of minimal responsibilities to distinguish an OAIS from other uses of ‘archive’ Framework for comparing architectures and operations of existing and future archives Basis for comparing data models of digital information preserved by archives, including their changes over time Expands consensus on elements and processes needed for long-term digital information preservation Guides the identification of future OAIS standards Does NOT specify any implementation

Scope Addresses minimal responsibilities of an OAIS Addresses a full range of archival functions including Ingest, Archival Storage, Data Management, Access and Dissemination Internal and external interfaces High level services at the interfaces Addresses the data models used to represent information Addresses migration of digital information to new media and new forms Addresses interoperability among OAIS archives Provides illustrative examples for context

Applicability Organizations with responsibility to preserve information over the long-term Produce information that may need long-term preservation Consumers of information from long-term archives May be applicable to organizations maintaining temporary archives Rapid technology changes may drive same preservation issues Information held may need long-term preservation Functional and information modeling concepts may be useful The access models are applicable to all entities Standards developers as basis for future standards

OAIS Concepts

Model View of an OAIS’s Environment Producer is the role played by those persons, or client systems, who provide the information to be preserved Management is the role played by those who set overall OAIS policy as one component in a broader policy domain Consumer is the role played by those persons, or client systems, who interact with OAIS services to find and acquire preserved information of interest The environment surrounding the OAIS is given by this simple model outside the OAIS are : producers : play the rôle of those who provide the information to preserve management : play the rôle of those who set overall OAIS policy consumers : play the rôle of those who interact with the OAIS services to find information of interest and to access this information OAIS (archive) Producer Consumer Management 7

OAIS Information Definition Information is defined as any type of knowledge that can be exchanged, and this information is always expressed (i.e., represented) by some type of data In general, it can be said that “Data interpreted using its Representation Information yields Information” In order for this Information Object to be successfully preserved, it is critical for an archive to clearly identify and understand the Data Object and its associated Representation Information Interpreted Using its Yields Data Object Representation Information Information Object

Information Package Definition Preservation Description Information Content Information An Information Package is a conceptual container of two types of information called Content Information and Preservation Description Information (PDI)

Information Package Variants Submission Information Package Negotiated between Producer and OAIS Sent to OAIS by a Producer Archival Information Package Information Package used for preservation Includes complete set of Preservation Description Information for the Content Information Dissemination Information Package Includes part or all of one or more Archival Information Packages Sent to a Consumer by the OAIS

External Data Flow Diagram Producer Submission Information Packages OAIS Archival Information Packages Legend = Entity Information Package Data Object = Data Flow = queries This diagram concentrates on the flow of information among producer, Consumers and the OAIS, and does not include flows that involve management query response Dissemination Information Packages orders Consumer 8

OAIS Responsibilities

OAIS Responsibilities Accepts Information Packages from information producers Assumes sufficient control to ensure long-term preservation Determines which communities need to be able to understand the preserved information Ensures the information to be preserved is independently understandable to the designated communities Follows documented policies and procedures which ensure the information is preserved against all reasonable contingencies Makes the preserved information available to the Designated Communities in forms understandable to those communities 11

Detailed Models Overview

Overview of Detailed Models It was decided to do both a functional and an information model of the OAIS Both models were tasked to: Use the models to better communicate OAIS Concepts Use a well established, formal modeling technique Stay as implementation independent as possible Avoid detailed designs

Detailed Models Information Model

General Principles Define classes of “information objects’ that illustrate information necessary to enable Long-term storage and access to Archives The class definition should be implementation Independent Use a variant of Object Modeling Technique (OMT) as a notation

OMT Notation Overview Class: Multiplicity of Associations: Class Name Class Exactly one Class Many (zero or more) Aggregation: Assembly Class Class Optional (zero or one) 1+ Class One or more Part -1 Class Part-2 Class Specialization: Association: Association Name Parent Class Class-1 Class-2 Child -1 Class Child-2 Class

Information Objects Information Object Representation 1+ interpreted using Data Physical Digital Bit Sequence

Representation Information The Representation Information accompanying a physical object like a moon rock may give additional meaning, as a result of some analysis, to the physically observable attributes of the rock The Representation Information accompanying a digital object, or sequence of bits, is used to provide additional meaning. It typically maps the bits into commonly recognized data types such as character, integer, and real and into groups of these data types. It associates these with higher level meanings which can have complex inter-relationships that are also described

Classes of Representation Information Structural Information: applied to turn bit sequences into common computer data types, aggregations of these data types, and mapping rules which map from the underlying data-types to the higher level structures needed to understand the Digital Object Semantic Information: includes meanings associated with all the elements of the structural information, operations that may be performed on each data-type, and their inter-relationships.

Recursive Nature of Representation Information Preexisting standards that define primitive data-types Mapping rules that map those primitive data-type into the more complex data-type concept used by the Data Object Other semantic informa-tion that aids in the under-standing of the Data such as a Data Dictionary

Sample Representation Net

Types of Information Used in OAIS

Content Information The information which is the primary object of preservation An instance of Content Information is the information that an archive is tasked to preserve. Deciding what is the Content Information may not be obvious and may need to be negotiated with the Producer The Data Object in the Content Information may be either a Digital Object or a Physical Object (e.g., a physical sample, microfilm)

Preservation Description Information Provenance Information Describes the source of Content Information, who has had custody of it, what is its history Context Information Describes how the Content Information relates to other information outside the Information Package Reference Information Provides one or more identifiers, or systems of identifiers, by which the Content Information may be uniquely identified Fixity Information Protects the Content Information from undocumented alteration

Example of Preservation Description Information Content Information Type Reference Provenance Context Fixity Space Science Data Bibliographic Information Software Package Object Identifier Journal Reference Mission, instrument, and title attribute set ISBN Title Author Name Version number Serial Number Instrument Description Processing History Sensor Description Instrument Instrument mode Processing history Decommunication map Software Interface Specifications Printing history Copyright Position in series Manuscripts References Revision Histroy License holder Registration Calibration history Related data sets Mission Funding history Related References Dewy Decimal System Publishing Data Publisher Help file User Guide Related Software Language CRC Checksum Reed-Solomon coding Author Digital signature Cover Certificate Encryption

Descriptive Information Contain the data that serves as the input to documents or applications called Access Aids. Access Aids can be used by a consumer to locate, analyze, retrieve, or order information from the OAIS.

Packaging Information Information which, either actually or logically, binds and relates the components of the package into an identifiable entity on specific media Examples of Packaging Information include tape marks, directory structures and filenames

OAIS Archival Information Package Package (AIP) Packaging Information Package Descriptor derived from delimited by e.g., Information supporting customer searches for AIP e.g., How to find Content information and PDI on some medium Preservation Description Information (PDI) Content Information further described by The AIP is a basic concept in the OAIS. The AIP contains a piece of information that the OAIS has to be preserve. The AIP isq defined to be composed of two types of information objects : one is called the Content Information (CI) and the other is called the Preservation Description Information (PDI) (terminology under discussion) The purpose in making this distinction is to propote a clear difference between that information (the CI) that is the primary focus of archival preservation and that information which is needed to support the long term preservation or the primary information. The AIP is a generic term, it will be specialised in AIC (Archiving Information Collection) and AIU (Archiving Information Unit) e.g., • Hardcopy document • Document as an electronic file together with its format description • Scientific data set consisting of images and text in three electronic files together with format descriptions e.g., • How the Content Information came into being, who has held it, how it relates to other information, and how its integrity is assured 9

AIP Types Based on the difference in Content Object complexity AIUs contain a single Data Object as the Content Object AICs contain multiple AIPs in their Content Objects Each member of an AIC is an AIP containing Content Information and PDI The AIC contains unique PDI on the collection process

Package Descriptors and Access Aids Package descriptors are needed by an OAIS to provide visibility and access to the OAIS holdings Package Descriptors contain 1 or more Associated Descriptions which describe the AIP Content Information from the point of view of a single Access Aid Some example of Access Aids Include: Finding Aids - assist the consumer in locating information of interest Ordering Aids - allow the consumer to discover the cost of and order AIUs of interest Retrieval Aids - enable authorized users to retrieve the AIU described by the Unit Descriptor from Archival Storage

Model of All Data Objects Stored in Data Management

Information Model Summary Presented a model of information objects as containing data objects and representation objects Classified information required for Long-term archiving into 4 classes: Content Information, PDI, Packaging Information and Descriptive Information Described how these classes would be aggregated and related in an AIP to fully describe an instance of Content Information Presented information needed for Access, in addition to that needed for Long-term Preservation Put the Access oriented structures in the context of the other data needed to operate an OAIS

Detailed Models Functional View

General Principles Highlight the major functional areas important to digital archiving Use functional decomposition to clarify the range of functionality that might be encountered Don't decompose beyond two levels to avoid becoming too implementation dependent Provide a useful set of terms and concepts Do not imply that all archives need to implement all the sub-functions Identify some common services which are likely to be needed, and are assumed to be available, as underlying support

Common Services Modern, distributed computing applications assume a number of supporting services Examples of Common Services include: inter-process communication name services temporary storage allocation exception handling security file and directory services

OAIS Functional Entities Data Management DI DI P R O D U C E Requests Ingest Acces other information C O N S U M E R SIP AIP AIP Archival Storage DIP Administration We can see, in this view, that the AIP is a specialized object create from a more genenal object called ‘Information package’ MANAGEMENT SIP = Submission Information Package AIP = Archival Information Package DIP = Dissemination Information Package DI = Descriptive Information 10

Functional Entities In An OAIS Ingest: This entity provides the services and functions to accept Submission Information Packages (SIPs) from Producers and prepare the contents for storage and management within the archive Archival Storage: This entity provides the services and functions for the storage, maintenance and retrieval of Archival Information Packages Data Management: This entity provides the services and functions for populating, maintaining, and accessing both descriptive information which identifies and documents archive holdings and internal archive administrative data. Administration: This entity manages the overall operation of the archive system Access: This entity supports consumers in determining the existence, description, location and availability of information stored in the OAIS and allowing consumers to request and receive information products

Ingest Functions Schedule Submission Delivery: negotiates a data submission schedule with the producer Receive Submission: provides the appropriate storage capability or devices to receive a SIP from the producer. The Receive SIP function may represent a legal transfer of custody for the CI in the SIP, and may require that special access controls be placed on the contents Generate Archival Information Package: transforms one or more SIPs into one or more AIPs that conforms to the internal data model of the archive. Generate Descriptive Information: extracts Descriptive Information from the AIPs to populate the data management system. Coordinate Updates: responsible for transferring the AIPs to Archival Storage and the Descriptive Information to Data Management

Archival Storage Functions Receive Data: receives a transfer request and an Archival Information Package from the staging area and moves the data to permanent storage within the archive Manage Storage Hierarchy: positions the contents of the Archival Information Packages (AIPs) on the appropriate media based on directions from ingest (transfer request), administrative policies or usage statistics Refresh Media: provides the capability to reproduce the Archive Holdings over time Error Checking: provides statistically acceptable assurance that no components of the archive information package are corrupted during any internal archive data transfer or transformation Disaster Recovery: provides a mechanism for producing duplicate copies of AIPs (AIUs and AICs) in the archive collection Provide Data: provides copies of stored AIPs to Access

Data Management Functions The Administer Database: responsible for maintaining the integrity of the Data Management database Perform Queries: receives a query request from Access and Dissemination and executes the query to generate a result set} that is transmitted to the requester The Generate Report: receives a report request and executes any queries or other processes necessary to generate the report then supplies the report to the requester Receive Database Update: adds, modifies or deletes information in the Data Management persistent storage Activate Request: maintains a record of subscription requests and periodically compares it to the contents of the archive to determine if all needed data is available. If needed data is available, this function generates a Dissemination Request which is sent to the Access. This function can also generate Dissemination Requests on a periodic basis

Access Functions (1 of 2) Prepare Finding Aids: provides tools and products which provide an overview of information products available in the archive system Receive Requests: provides a single user interface to the information holdings of the archive. This interface will normally be via computer network or dial-up link to an on-line service, but might also be implemented in the form of a walk-in facility, printed catalog ordering service, or fax-back type service Coordinate Request: function determines the resources needed to fulfill archive requests and forwards them to those entities for execution. Query and report requests will be fulfilled by data management while dissemination requests will generally require information from data management and archival storage

Access Functions (2 of 2) Generate DIP: accepts a dissemination request, retrieves the data from Archival Storage and moves a copy of the data to a staging area for further processing. This function also transmits a report request to Data Management to generate descriptive information. If special processing is required this function accesses data objects in staging storage and applies the requested processes. This function places the completed DIP package in the staging area and notifies both Coordinate Request and the Deliver DIP function that the package is ready for delivery. Deliver DIPs: handles both on-line and off-line deliveries of DIPs to consumers Provide Access Controls: provides a hierarchy of security controls depending on the needs of the archive system. These include restricting access to certain information due to security classifications or copyright restrictions.

Administration Functions (1 of 2) Negotiate Submission Agreement: solicits desirable archival information for inclusion into the OAIS and negotiates submission agreements with data producers Manage System Configuration: provides system engineering for the archive system to systematically control changes to the configuration. This function maintains integrity and tractability of the configuration during all phases of the system life cycle. It also audits system operations, system performance, and system usage and plans for system evolution Physical Access Control: provides mechanisms to restrict or allow physical access (doors, locks, guards) to elements of the archive as determined by archive policies

Administration Functions (2 of 2) Develop Standards and Policies: is responsible for developing and maintaining the archive system data standards. These standards include format standards, documentation standards and the procedures to be followed during the ingestion process. It will also develop policies for Archival Storage hierarchy management and migration policies Audit AIPs: is carried out by the archive data engineers and may also involve an outside committee (e.g., science and technical review). The audit process must verify that the quality of the data meets the requirements of the archive and the review committee Interact with Management: receives and carries out Management policies. These policies include such things as the OAIS charter, scope, resource utilization guidelines, and pricing policies. It also provides OAIS performance information to Management

Analysis of Archive Issues Using OAIS RM Archive Associations

Archive Cooperation Users of multiple OAIS archives have reasons to wish for some uniformity or cooperation among the OAISs. Consumers Common finding aids to aid in locating information over several OAIS archives Common Package Descriptor schema for access Common DIP schema for dissemination, or a single global access site. Producers common SIP schema for submission to different archives a single depository for all their products. Managers Cost reduction through sharing of expensive hardware increasing the uniformity and quality of user interactions with the OAIS

Categories of Archive Interactions Independent: no knowledge by one OAIS of Standards implemented at another Cooperating: Potentially common submission standards, and common dissemination standards, but no common access. One archive may make subscription requests for key data at the cooperating archive Federated: Access to all federated OAIS is provided through a common set of access aids that provide visibility into all participating OAISs. Global dissemination and Ingest are options Shared resources: An OAIS in which Management has entered into agreements with other OAISs is to share resources to reduce cost. This requires various standards internal to the archive (such as ingest-storage and access-storage interface standards), but does not alter the community’s view of the archive

Cooperating Archives Method B Method A Ing Acc OAIS Ing Acc Ing Acc Adm Adm Adm OAIS OAIS Producer Consumer Adm Adm Adm Adm Adm Adm Ing Acc Method B OAIS Method A Adm Adm Adm The first set of cooperating OAIS merely have an agreement to share at least on common SIP and DIP format to enable the transfer of holdings The second set of cooperating OAIS have standardized their DIP and SIP formats for use by producers and consumers

Federated Archives

Levels of Autonomy in Associated Archives No interactions and therefore no association Associations that maintain your autonomy. You have to do certain things to participate, but you can leave the association without notice or impact to you. Associations that bind you by contract. To change the nature of this association you will have to re-negotiate the contract. The amount of autonomy retained depends on how difficult it is to negotiate the changes.

Analysis of Archive Issues Using OAIS RM Migration

Digital Migration Digital Migration is defined to be the transfer of digital information, while intending to preserve it, within the OAIS. Focus on the preservation of the full information content Internal OAIS perspective. Three major motivators are seen to drive Digital Migrations of Archival Information Packages within an OAIS: Media Decay Increased Cost Effectiveness New Consumer Service Requirements

Digital Migration Approaches Two basic types of migration in response to motivators: Repackaging Transformation Each of these comes in two basic flavors, ordered by increasing risk of information loss: Physical Repackaging (Replication) Media replacement with no bit changes Digital Repackaging Some bit changes in Packaging Information Reversible Transformations Bit changes in Content Information are reversible by an algorithm Non-reversible Transformations Bit changes in Content Information are not reversible by an algorithm

Some Migration Strategies Use media types with long-lived support Enhances chances that Replication will be useful Minimize use of media format attributes for holding Content Information Allows digital repackaging without having to also do transformations when migrating to new media types Maintain originals if non-reversible transformations are required

AIP Versions and Editions An AIP that undergoes a transformation during migration becomes a new version Edition An AIP that is revised to improve its information content is termed a new edition. This is not a migration. Derivation An AIP that is the result of being derived from one or more other AIPs is termed a derived AIP. This is not a migration.

Summary and Request

Summary Reference model is to be applicable to all digital archives, and their Producers and Consumers Identifies a minimum set of responsibilities for an archive to claim it is an OAIS Establishes common terms and concepts for comparing implementations, but does not specify an implementation Provides detailed models of both archival functions and archival information Discusses OAIS information migration and interoperability among OAISs

Request for Participation (Updated 1998-10-10) Ultimate success of this effort depends on obtaining adequate review and comment Reference Model Red Book/ISO Draft International Standard (DIS) expected December 1998 Current version is White Book 4.0, available under Reference Materials heading, at: http://ssdoo.gsfc.nasa.gov/nost/isoas/overview.html Comments are being actively solicited Send to donald.sawyer@gsfc.nasa.gov