Presentation is loading. Please wait.

Presentation is loading. Please wait.

PAWN: A Novel Ingestion Workflow Technology for Digital Preservation Mike Smorul, Joseph JaJa, Yang Wang, and Fritz McCall.

Published by Modified over 8 years ago

Similar presentations

Presentation on theme: "PAWN: A Novel Ingestion Workflow Technology for Digital Preservation Mike Smorul, Joseph JaJa, Yang Wang, and Fritz McCall."— Presentation transcript:

1 PAWN: A Novel Ingestion Workflow Technology for Digital Preservation Mike Smorul, Joseph JaJa, Yang Wang, and Fritz McCall

2 Overall Principles Consistent with the Open Archival Information System (OAIS) model Distributed, secure ingestion Use of web/grid technologies – platform independent Minimal client-side requirements Ease of integration with archival storage or data grid systems.

3 Producer

4 Provides data to an Archive based on a prior agreement. Consists of a management/metadata server and an ingestion client. Provides initial arrangement, context, and metadata.

5 Archive - receiving

6 Archive – receiving Receives data from a Producer Validates bitstreams and metadata, and sends acknowledgement to Producer. Arranges into collections and specifies preservation policy. Publishes bitstreams into a digital archive.

7 Archive – Long term preservation Implemented using grid technologies. Use the existing prototype NARA/UMD/SDSC site. Automated replication and integrity checking. Enforces access control and preservation policy

8 Ingestion Workflow 1. Negotiate Submission Agreement. 2. Workflow Initialization and Submission Information Packet (SIP) creation. 3. Transfer of SIPs to archive. 4. Validation of SIP transfer 5. Organization of data into collections and transfer into persistent archive.

9 Submission Agreement Based on data appraisal and record schedule, including format and metadata. Create machine actionable set of rules describing items. Final Submission Agreement is composed of:  METS document for application defaults  METS Constraint document to limit METS form to submission parameters

10 METS Overview Provides a framework for linking structural organization of objects with metadata. Using XML namespace, metadata from various XML schema can be attached to objects  Ie, dublin core, FGDC, etc Extensible for more complex metadata http://www.loc.gov/standards/mets/

11 Sample METS Document

12 Why METS Constraints? METS doesn’t provide a way to create machine interpretable rules describing a collection  Ie: allow only JPEG files in certain structural areas METS profiles allow for developer interpretable rules, not machine interpretable

13 METS Constraints Allows structural, metadata, and file constraints. Structural Constraints:  Restrict child div’s and restrict pointers to div, file, and other mets documents File Constraints:  Restrict files by mime-type or validation tests Metadata Constraints:  Restrict allowed metadata schema.

14 METS Constraints Example... ID="DIV2" ORDER="1" LABEL="Reports for 1997“ DMDID="tree97"> <div ID="DIV3" ORDER="2" LABEL="Meeting Notes for 1997" DMDID="tree98">...... …

15 Ingestion Workflow 1. Negotiate Submission Agreement. 2. Workflow Initialization and Submission Information Packet creation. 3. Transfer of SIPs to archive. 4. Validation of SIP transfer 5. Organization of data into collections and transfer into persistent archive.

16 Initialize Ingestion workflow Instantiate Producer management server to track registered objects Establish a working trust relationship with the Archive Issue clients.

17 Create SIP Each client registers objects stored locally with producer management server  Register file types, validation tests, etc  Client follows rules in Submission Agreement Producer-wide agents can arrange registered object to give a broader context

18 SIP Example METS Handles all areas of a SIP except Physical Object and Descriptive Information Descriptive Information can be embedded into METS as 3 rd party XML schema

19 Client Interface

20 Ingestion Workflow 1. Negotiate Submission Agreement. 2. Workflow Initialization and Submission Information Packet creation. 3. Transfer of SIPs to archive. 4. Validation of SIP transfer 5. Organization of data into collections and transfer into persistent archive.

21 Transfer SIP to archive Retrieve previously registered SIP from producer management server Authenticate to archive Update provenance information in METS document with file structure of SIP Transfer METS document describing SIP and container for SIP physical objects Archive acknowledges transfer completion to producer management server

22 Ingestion Workflow 1. Negotiate Submission Agreement. 2. Workflow Initialization and Submission Information Packet creation. 3. Transfer of SIP to archive. 4. Validation of SIP transfer 5. Organization of data into collections and transfer into persistent archive.

23 Validation of SIP transfer Check incoming SIP against constraints documents. Ensure object integrity by verifying checksums/cryptographic digest Validate bitstreams against tests described in METS document Update METS document with validation results and movement of objects on receiving server

24 Ingestion Workflow 1. Negotiate Submission Agreement. 2. Workflow Initialization and Submission Information Packet creation. 3. Transfer of SIP to archive. 4. Validation of SIP transfer 5. Organization of data into collections and transfer into persistent archive.

25 Final transfer to archive Transfer objects to digital archive Update provenance information in METS document with handle to object in archive Transfer METS document into archive Return accept/reject messages to producer metadata server

26 Component Overview

27 Producer Components Database to track registered objects Certificate Authority management  Web service for archive security check Management server supplies web service interfaces to ingestion clients and management operations. Clients are designed to be standalone, with security certificates issued by producer

28 Archive Components Receiving servers validate connecting clients and validate SIPs Validation Services are simple webservice calls. Abstract I/O layer into digital archive. All components are scalable using standard load balancing techniques.

29 Recap Implemented using web technologies Architecture independent OAIS compliant XML based metadata  METS based SIPs  Add-on constraints describing Submission Agreement

30 Questions?? For more information  http://www.umiacs.umd.edu/research/adapt

Download ppt "PAWN: A Novel Ingestion Workflow Technology for Digital Preservation Mike Smorul, Joseph JaJa, Yang Wang, and Fritz McCall."

Similar presentations

An Introduction to Repositories Thornton Staples Director of Community Strategy and Alliances Director of the Fedora Project.

An Introduction to Repositories Thornton Staples Director of Community Strategy and Alliances Director of the Fedora Project.
October 28, 2003Copyright MIT, 2003 METS repositories: DSpace MacKenzie Smith Associate Director for Technology MIT Libraries.

October 28, 2003Copyright MIT, 2003 METS repositories: DSpace MacKenzie Smith Associate Director for Technology MIT Libraries.
Fedora Users’ Conference Rutgers University May 14, 2005 Researching Fedora's Ability to Serve as a Preservation System for Electronic University Records.

Fedora Users’ Conference Rutgers University May 14, 2005 Researching Fedora's Ability to Serve as a Preservation System for Electronic University Records.
ASCR Data Science Centers Infrastructure Demonstration S. Canon, N. Desai, M. Ernst, K. Kleese-Van Dam, G. Shipman, B. Tierney.

ASCR Data Science Centers Infrastructure Demonstration S. Canon, N. Desai, M. Ernst, K. Kleese-Van Dam, G. Shipman, B. Tierney.
1 Introduction to XML. XML eXtensible implies that users define tag content Markup implies it is a coded document Language implies it is a metalanguage.

1 Introduction to XML. XML eXtensible implies that users define tag content Markup implies it is a coded document Language implies it is a metalanguage.
ADAPT An Approach to Digital Archiving and Preservation Technology Principal Investigator: Joseph JaJa Lead Programmers: Mike Smorul and Mike McGann Graduate.

ADAPT An Approach to Digital Archiving and Preservation Technology Principal Investigator: Joseph JaJa Lead Programmers: Mike Smorul and Mike McGann Graduate.
PAWN: Producer-Archive Workflow Network University of Maryland Institute for Advanced Computer Studies Joseph Ja’Ja, Mike Smorul, Mike McGann.

PAWN: Producer-Archive Workflow Network University of Maryland Institute for Advanced Computer Studies Joseph Ja’Ja, Mike Smorul, Mike McGann.
May 23 2007 Archiving 2007 1 PAWN: A Policy-Driven Software Environment for Implementing Producer- Archive Interactions in Support of Long Term Digital.

May Archiving PAWN: A Policy-Driven Software Environment for Implementing Producer- Archive Interactions in Support of Long Term Digital.
Tools and Services for the Long Term Preservation and Access of Digital Archives Joseph JaJa, Mike Smorul, and Sangchul Song Institute for Advanced Computer.

Tools and Services for the Long Term Preservation and Access of Digital Archives Joseph JaJa, Mike Smorul, and Sangchul Song Institute for Advanced Computer.
Producer-Archive Workflow Network (PAWN) Goals Consistent with the Open Archival Information System (OAIS) model Use of web/grid technologies and platform.

Producer-Archive Workflow Network (PAWN) Goals Consistent with the Open Archival Information System (OAIS) model Use of web/grid technologies and platform.
Supporting Customized Archival Practices Using the Producer-Archive Workflow Network (PAWN) Mike Smorul, Mike McGann, Joseph JaJa.

Supporting Customized Archival Practices Using the Producer-Archive Workflow Network (PAWN) Mike Smorul, Mike McGann, Joseph JaJa.
NextGRID & OGSA Data Architectures: Example Scenarios Stephen Davey, NeSC, UK ISSGC06 Summer School, Ischia, Italy 12 th July 2006.

NextGRID & OGSA Data Architectures: Example Scenarios Stephen Davey, NeSC, UK ISSGC06 Summer School, Ischia, Italy 12 th July 2006.
Brief Overview of Major Enhancements to PAWN. Producer – Archive Workflow Network (PAWN) Distributed and secure ingestion of digital objects into the.

Brief Overview of Major Enhancements to PAWN. Producer – Archive Workflow Network (PAWN) Distributed and secure ingestion of digital objects into the.
PREMIS What is PREMIS? o Preservation Metadata Implementation Strategies When is PREMIS use? o PREMIS is used for “repository design, evaluation, and archived.

PREMIS What is PREMIS? o Preservation Metadata Implementation Strategies When is PREMIS use? o PREMIS is used for “repository design, evaluation, and archived.
July 20 2007 NAGARA 1 Producer-Archive Workflow Network Mike Smorul, Mike McGann, Joseph JaJa Institute for Advanced Computer Science Studies University.

July NAGARA 1 Producer-Archive Workflow Network Mike Smorul, Mike McGann, Joseph JaJa Institute for Advanced Computer Science Studies University.
Robust Tools for Archiving and Preserving Digital Data Joseph JaJa, Mike Smorul, and Mike McGann Institute for Advanced Computer Studies Department of.

Robust Tools for Archiving and Preserving Digital Data Joseph JaJa, Mike Smorul, and Mike McGann Institute for Advanced Computer Studies Department of.
PAWN: A Novel Ingestion Workflow Technology for Digital Preservation

PAWN: A Novel Ingestion Workflow Technology for Digital Preservation
WMS: Democratizing Data

WMS: Democratizing Data
PAWN Progress July 06, 2006. Overview of changes New flexible environment for setting up and managing interactions between producers and the archive Domains.

PAWN Progress July 06, Overview of changes New flexible environment for setting up and managing interactions between producers and the archive Domains.
Tools and Services for the Long Term Preservation and Access of Digital Archives Joseph JaJa, Mike Smorul, and Sangchul Song Institute for Advanced Computer.

Tools and Services for the Long Term Preservation and Access of Digital Archives Joseph JaJa, Mike Smorul, and Sangchul Song Institute for Advanced Computer.

Similar presentations

Ads by Google