We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byJacob Kennedy
Modified over 3 years ago
IBM Haifa Research Lab © 2008 IBM Corporation Contacts: Simona Cohen, Michael Factor, Dalit Naor http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml Preservation DataStores (PDS): A demonstration of Preservation Aware Storage
IBM Haifa Research Lab © 2008 IBM Corporation 2http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml PDS Overview OAIS–based preservation-aware storage, media-agnostic, and generic storage to support logical preservation Manage preservation specific metadata –Compute fixity –Update technical provenance –Manage PDI RepInfo –Ensure referential integrity Storlet container –Module container that can execute restricted modules with predefined interfaces for data intensive functions, e.g., transformations, fixity calculations –Optimal scheduling –Update PDS modules, e.g., fixity algorithm, packaging format Managing availability/ data loss –Physically co-locate data and metadata –Cluster related AIPs on the same media unit, based upon their relative importance AIP identifier generation – globally unique identifiers Content Data Object Representation Information Reference ContextFixity Provenance Content Information Preservation Descriptive Information
IBM Haifa Research Lab © 2008 IBM Corporation 3http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml PDS Architecture Layered approach is based on open standards: OAIS, XAM, OSD In CASPAR, all layers are utilized. In other embodiments, only some layers can be used Utilizes XAM to provide logical abstraction of containers (XSets) Offloads preservation functionality to storage ingestAIP, accessAIP, transformAIP, getPreservationPolicy
IBM Haifa Research Lab © 2008 IBM Corporation 4http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml AIP Representation in the Preservation Engine The basic building-block of an AIP is the Information Object –A single instance of data is a byte stream –Zero or more instances of RepInfo to interpret the data RepInfo is also an AIP –Shared resource with unique ID –RepInfo recursive network –Keep the design simple
IBM Haifa Research Lab © 2008 IBM Corporation 5http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml Preservation of an AIP Ingest AIP –Storing an AIP in PDS Bit and logical preservation of an AIP –Bit migrations –Format migrations - data transformation Transformation modules are packed as AIPs and preserved Transformation result is a new version for the original AIP –Migrations are documented as Provenance records –During migrations PDS performs operations on AIP Update PDI (e.g., fixity calculation, additional provenance events) Execute previously loaded storlets Access AIP –Retrieval of an AIP By retrieval time, the original AIP may have several versions and copies
IBM Haifa Research Lab © 2008 IBM Corporation Demo
IBM Haifa Research Lab © 2008 IBM Corporation 7http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml MST Data Natural Environment Research Council (NERC) Mesosphere-Stratosphere-Troposphere (MST) Radar at Aberystwyth Provides measurements of vertical and horizontal components of the wind Highly complex data Preserved for the atmospheric scientific community Needs long-term preservation for research purposes Community is highly dependent on interoperability
IBM Haifa Research Lab © 2008 IBM Corporation 8http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml British Atmospheric Data Centre and Self-Describing Data Formats BADC strongly advocate the use of self-describing files to capture provenance and semantic information Two self-describing file formats compete in the Atmospheric Science Community –NASA AMES (ASCII) –NetCDF (binary) The current user community finds NetCDF more suitable
IBM Haifa Research Lab © 2008 IBM Corporation 9http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml 10 years later 20 years later Ingest AIP Load and apply transformation Access and view transformed data View AIP Demo Flow
IBM Haifa Research Lab © 2008 IBM Corporation 10http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml pds_view Video:
IBM Haifa Research Lab © 2008 IBM Corporation 11http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml Ingest AIP flow in PDS Preservation Engine PDS web service XAM XAM to OSD (VIM) OSD ingest (AIP) XUID AIP ID create XSet write fields (AIP data) commit create OSD objects write (XSet data) Unpack AIP Generate AIP ID Compute fixity Add ingest provenance event store (OID, XUID) mapping persistently store (XUID, AIP ID) mapping persistently
IBM Haifa Research Lab © 2008 IBM Corporation 12http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml 10 years later 20 years later Ingest AIP Load and apply transformation Access and view transformed data View AIP Demo Flow
IBM Haifa Research Lab © 2008 IBM Corporation 13http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml pds_ingest Video:
IBM Haifa Research Lab © 2008 IBM Corporation 14http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml
IBM Haifa Research Lab © 2008 IBM Corporation 15http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml Format Change Converting NetCDF to NASA-AMES –The MST NetCDF file version is no longer supported by the NetCDF foundations libraries, and files are not backwards compatible with this NetCDF version –The skills base of the atmospheric scientist in the future is heavily biased towards the use of ASCII-based file formats –The NASA-AMES format has become the preferred data format of the atmospheric scientists community Converting NetCDF to PNG –Provide additional quick image view of the data for users with limited access permissions
IBM Haifa Research Lab © 2008 IBM Corporation 16http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml Load transformation flow in PDS Preservation Engine PDS web service loadTransformation (AIP) AIP ID Register transformation Ingest transformation AIP
IBM Haifa Research Lab © 2008 IBM Corporation 17http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml Transform AIP flow in PDS PDS web service transformAip (targetAIPID, transformationAIPID) AIP ID Access transformation AIP Access target AIP Execute transformation Ingest result as new AIP, a version of the target AIP Preservation Engine
IBM Haifa Research Lab © 2008 IBM Corporation 18http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml 10 years later 20 years later Ingest AIP Load and apply transformation Access and view transformed data View AIP Demo Flow
IBM Haifa Research Lab © 2008 IBM Corporation 19http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml pds_xform Video:
IBM Haifa Research Lab © 2008 IBM Corporation 20http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml
IBM Haifa Research Lab © 2008 IBM Corporation 21http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml Access AIP flow in PDS Preservation Engine PDS web service XAM XAM to OSD (VIM) OSD access (AIP ID) read OSD objects AIP data AIP XSet data open XSet (XUID) read fields Lookup XUID by AIP ID Lookup OSD OID by XUID Validate AIP ID Validate fixity Add access provenance event Package AIP
IBM Haifa Research Lab © 2008 IBM Corporation 22http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml Demo Flow 10 years later 20 years later Ingest AIP Load and apply transformation Access and view transformed data View AIP
IBM Haifa Research Lab © 2008 IBM Corporation 23http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml pds_access Video:
IBM Haifa Research Lab © 2008 IBM Corporation Thank you! Haifa preservation team: Simona Cohen, Michael Factor, Aner Hamama, Ealan Henis, Dalit Naor, Petra Reshef, Shahar Ronen
25 seconds left….. 24 seconds left….. 23 seconds left…..
WEEK 1 You have 10 seconds to name…
Addition 1’s to
San Diego Supercomputer CenterUniversity of California, San Diego Preservation Research Roadmap Reagan W. Moore San Diego Supercomputer Center
OAIS Rathachai Chawuthai Information Management CSIM / AIT Issued document 1.0.
Credit hours: 4 Contact hours: 50 (30 Theory, 20 Lab) Prerequisite: TB143 Introduction to Personal Computers.
Business Transaction Management Software for Application Coordination 1 Business Processes and Coordination.
Metadata for preservation: the Cedars perspective Michael Day UKOLN: UK Office for Library and Information Networking University of Bath
Addition Facts = = =
Jeopardy Topic 1Topic Q 1Q 6Q 11Q 16Q 21 Q 2Q 7Q 12Q 17Q 22 Q 3Q 8Q 13Q 18Q 23 Q 4Q 9Q 14Q 19Q 24 Q 5Q 10Q 15Q 20Q 25 Final Jeopardy.
ABC Technology Project Mrs. Kiddle. ABCs of Technology Word 1 Word 2 Word 3 Word 4 Word 5 Word 6 Word 7 Word 8 Word 9 Word 19 Word 20 Word 21 Word 22.
Executional Architecture Lecture Conceptual vs execution Conceptual Architecture Execution Architecture Component Connector Domain-level responsibilities.
NATIONAL AERONAUTICS AND SPACE ADMINISTRATION ISO 14721:2003 OAIS - RM 6 July, 2004 Richard Ullman / 18th APAN, eScience Workshop, Cairns Austrailia 1.
BT Wholesale October Creating your own telephone network WHOLESALE CALLS LINE ASSOCIATED.
1 9 Moving to Design Lecture Analysis Objectives to Design Objectives Figure 9-2.
DIVIDING INTEGERS 1. IF THE SIGNS ARE THE SAME THE ANSWER IS POSITIVE 2. IF THE SIGNS ARE DIFFERENT THE ANSWER IS NEGATIVE.
By D. Fisher Geometric Transformations. Reflection, Rotation, or Translation 1.
We will resume in: 25 Minutes We will resume in: 24 Minutes.
1 What is the Internet Archive We are a Digital Library Mission Statement: Universal access to human knowledge Founded in 1996 by Brewster Kahle in San.
Future Access to the Scientific and Cultural Heritage – A shared Responsibility Birte Christensen-Dalsgaard State and University Library.
1 © 2004, Cisco Systems, Inc. All rights reserved. CCNA 1 v3.1 Module 1 Introduction to Networking.
DuraSpace: Digital Information All Ways, Always Pretoria, South Africa May 14 th, 2009.
Describing Complex Products as Configurations using APL Arrays.
OAIS (archive) OAIS (archive) Producer Management Consumer.
Distributed DBMS©M. T. Özsu & P. Valduriez Ch.15/1 Outline Introduction Background Distributed Database Design Database Integration Semantic Data Control.
17 Copyright © 2005, Oracle. All rights reserved. Deploying Applications by Using Java Web Start.
DRIVER Long Term Preservation for Enhanced Publications in the DRIVER Infrastructure 1 WePreserve Workshop, October 2008 Dale Peters, Scientific Technical.
Copyright CompSci Resources LLC Web-Based XBRL Products from CompSci Resources LLC Virginia, USA. Presentation by: Colm Ó hÁonghusa.
ICDL-Contentra Workshop 29 th November /11/2013 Contentra Technologies Confidential (RajuB)1.
DigiTools support of Web Services Repositories and Web Services workshop | 2 June 2009 Alan Oliver, Business Development Director.
PAWN: A Novel Ingestion Workflow Technology for Digital Preservation Mike Smorul, Joseph JaJa, Yang Wang, Mike McGann, and Fritz McCall.
DATABASE SYSTEM CONCEPTS AND ARCHITECTURE CHAPTER 2 1.
© 2011 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. Towards a Model-Based Characterization of Data and Services Integration Paul.
1 Copyright © 2010, Elsevier Inc. All rights Reserved Fig 2.1 Chapter 2.
Putting the Pieces Together Grace Agnew Slide User Description Rights Holder Authentication Rights Video Object Permission Administration.
18 Copyright © 2005, Oracle. All rights reserved. Distributing Modular Applications: Introduction to Web Services.
ERPANET Workshop on Workflow / 13. October 2004 / Stephan Heuscher / p. 1 Workflows in Digital Preservation.
Flex Your APEX Implementing Oracle E-Business Suite Descriptive Flexfields in Application Express Shane Bentz InterVarsity Christian Fellowship/USA.
© 2005 by Prentice Hall 1 Chapter 1: The Database Environment Modern Database Management 7 th Edition Jeffrey A. Hoffer, Mary B. Prescott, Fred R. McFadden.
OGF-23 iRODS Metadata Grid File System Reagan Moore San Diego Supercomputer Center.
SUBTRACTING INTEGERS 1. CHANGE THE SUBTRACTION SIGN TO ADDITION 2. TAKE THE INVERSE OF THE SECOND NUMBER 3. FOLLOW THE RULES FOR ADDITION 4. ADD THE OPPOSITE.
10 Systems Analysis and Design in a Changing World, Fifth Edition.
M-1 INGEST OVERVIEW Don Sawyer National Space Science Data Center NASA/GSFC October 13, 1999.
Info to Enterprise Migration Implementation Case Study: SBC Corporation Presented to the Crystal Decisions Regional Users Group for the Bay Area on October.
® IBM Software Group © 2006 IBM Corporation Writing Good Use Cases Module 2: Finding Actors and Use Cases.
HATHI TRUST A Shared Digital Repository Digital Repositories for Preservation and Access Digital Directions 2013 Jeremy York July 22, 2013 Unless otherwise.
© 2017 SlidePlayer.com Inc. All rights reserved.