We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byJacob Kennedy
Modified over 2 years ago
IBM Haifa Research Lab © 2008 IBM Corporation Contacts: Simona Cohen, Michael Factor, Dalit Naor Preservation DataStores (PDS): A demonstration of Preservation Aware Storage
IBM Haifa Research Lab © 2008 IBM Corporation 2http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml PDS Overview OAIS–based preservation-aware storage, media-agnostic, and generic storage to support logical preservation Manage preservation specific metadata –Compute fixity –Update technical provenance –Manage PDI RepInfo –Ensure referential integrity Storlet container –Module container that can execute restricted modules with predefined interfaces for data intensive functions, e.g., transformations, fixity calculations –Optimal scheduling –Update PDS modules, e.g., fixity algorithm, packaging format Managing availability/ data loss –Physically co-locate data and metadata –Cluster related AIPs on the same media unit, based upon their relative importance AIP identifier generation – globally unique identifiers Content Data Object Representation Information Reference ContextFixity Provenance Content Information Preservation Descriptive Information
IBM Haifa Research Lab © 2008 IBM Corporation 3http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml PDS Architecture Layered approach is based on open standards: OAIS, XAM, OSD In CASPAR, all layers are utilized. In other embodiments, only some layers can be used Utilizes XAM to provide logical abstraction of containers (XSets) Offloads preservation functionality to storage ingestAIP, accessAIP, transformAIP, getPreservationPolicy
IBM Haifa Research Lab © 2008 IBM Corporation 4http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml AIP Representation in the Preservation Engine The basic building-block of an AIP is the Information Object –A single instance of data is a byte stream –Zero or more instances of RepInfo to interpret the data RepInfo is also an AIP –Shared resource with unique ID –RepInfo recursive network –Keep the design simple
IBM Haifa Research Lab © 2008 IBM Corporation 5http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml Preservation of an AIP Ingest AIP –Storing an AIP in PDS Bit and logical preservation of an AIP –Bit migrations –Format migrations - data transformation Transformation modules are packed as AIPs and preserved Transformation result is a new version for the original AIP –Migrations are documented as Provenance records –During migrations PDS performs operations on AIP Update PDI (e.g., fixity calculation, additional provenance events) Execute previously loaded storlets Access AIP –Retrieval of an AIP By retrieval time, the original AIP may have several versions and copies
IBM Haifa Research Lab © 2008 IBM Corporation Demo
IBM Haifa Research Lab © 2008 IBM Corporation 7http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml MST Data Natural Environment Research Council (NERC) Mesosphere-Stratosphere-Troposphere (MST) Radar at Aberystwyth Provides measurements of vertical and horizontal components of the wind Highly complex data Preserved for the atmospheric scientific community Needs long-term preservation for research purposes Community is highly dependent on interoperability
IBM Haifa Research Lab © 2008 IBM Corporation 8http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml British Atmospheric Data Centre and Self-Describing Data Formats BADC strongly advocate the use of self-describing files to capture provenance and semantic information Two self-describing file formats compete in the Atmospheric Science Community –NASA AMES (ASCII) –NetCDF (binary) The current user community finds NetCDF more suitable
IBM Haifa Research Lab © 2008 IBM Corporation 9http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml 10 years later 20 years later Ingest AIP Load and apply transformation Access and view transformed data View AIP Demo Flow
IBM Haifa Research Lab © 2008 IBM Corporation 10http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml pds_view Video:
IBM Haifa Research Lab © 2008 IBM Corporation 11http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml Ingest AIP flow in PDS Preservation Engine PDS web service XAM XAM to OSD (VIM) OSD ingest (AIP) XUID AIP ID create XSet write fields (AIP data) commit create OSD objects write (XSet data) Unpack AIP Generate AIP ID Compute fixity Add ingest provenance event store (OID, XUID) mapping persistently store (XUID, AIP ID) mapping persistently
IBM Haifa Research Lab © 2008 IBM Corporation 12http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml 10 years later 20 years later Ingest AIP Load and apply transformation Access and view transformed data View AIP Demo Flow
IBM Haifa Research Lab © 2008 IBM Corporation 13http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml pds_ingest Video:
IBM Haifa Research Lab © 2008 IBM Corporation 14http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml
IBM Haifa Research Lab © 2008 IBM Corporation 15http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml Format Change Converting NetCDF to NASA-AMES –The MST NetCDF file version is no longer supported by the NetCDF foundations libraries, and files are not backwards compatible with this NetCDF version –The skills base of the atmospheric scientist in the future is heavily biased towards the use of ASCII-based file formats –The NASA-AMES format has become the preferred data format of the atmospheric scientists community Converting NetCDF to PNG –Provide additional quick image view of the data for users with limited access permissions
IBM Haifa Research Lab © 2008 IBM Corporation 16http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml Load transformation flow in PDS Preservation Engine PDS web service loadTransformation (AIP) AIP ID Register transformation Ingest transformation AIP
IBM Haifa Research Lab © 2008 IBM Corporation 17http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml Transform AIP flow in PDS PDS web service transformAip (targetAIPID, transformationAIPID) AIP ID Access transformation AIP Access target AIP Execute transformation Ingest result as new AIP, a version of the target AIP Preservation Engine
IBM Haifa Research Lab © 2008 IBM Corporation 18http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml 10 years later 20 years later Ingest AIP Load and apply transformation Access and view transformed data View AIP Demo Flow
IBM Haifa Research Lab © 2008 IBM Corporation 19http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml pds_xform Video:
IBM Haifa Research Lab © 2008 IBM Corporation 20http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml
IBM Haifa Research Lab © 2008 IBM Corporation 21http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml Access AIP flow in PDS Preservation Engine PDS web service XAM XAM to OSD (VIM) OSD access (AIP ID) read OSD objects AIP data AIP XSet data open XSet (XUID) read fields Lookup XUID by AIP ID Lookup OSD OID by XUID Validate AIP ID Validate fixity Add access provenance event Package AIP
IBM Haifa Research Lab © 2008 IBM Corporation 22http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml Demo Flow 10 years later 20 years later Ingest AIP Load and apply transformation Access and view transformed data View AIP
IBM Haifa Research Lab © 2008 IBM Corporation 23http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml pds_access Video:
IBM Haifa Research Lab © 2008 IBM Corporation Thank you! Haifa preservation team: Simona Cohen, Michael Factor, Aner Hamama, Ealan Henis, Dalit Naor, Petra Reshef, Shahar Ronen
1 STFC testbed. Testbed Aims Demonstrate complete solutions at different cost levels Produce an Analysis Methodology Produce Modelling Technique Produce.
1 The ESA SCIENTIFIC TESTBED in CASPAR S. ALBANI (ESA) CASPAR Workshop, Rome, september 2009.
Metadata for preservation: the Cedars perspective Michael Day UKOLN: UK Office for Library and Information Networking University of Bath
Digital disaster: are you prepared?, University College London, 23 June 2000 Michael Day, UKOLN Overview UKOLN is funded by Resource:
NATIONAL AERONAUTICS AND SPACE ADMINISTRATION ISO 14721:2003 OAIS - RM 6 July, 2004 Richard Ullman / 18th APAN, eScience Workshop, Cairns Austrailia 1.
1 Database Systems: Design, Implementation, and Management CHAPTER 10 Distributed Database Management System.
Long-Term Preservation. Technical Approaches to Long-Term Preservation the challenge is to interpret formats a similar development: sound carriers From.
The Reference Model for an Open Archival Information System (OAIS) Michael Day Digital Curation Centre UKOLN, University of Bath
Digital preservation: an introduction Michael Day UKOLN, University of Bath, UK University of the West of England,
1 The CASPAR Project Key Components - Overview Luigi Briguglio Engineering R&D Laboratory – Roma CASPAR FINAL WORKSHOP IN ROME SCUOLA SUPERIORE PUBBLICA.
Digital preservation Michael Day UKOLN, University of Bath, UK University of the West of England, MSc in Information.
WMO ET-ADRS: April WMO ET-ADRS Hierarchical Data Format (HDF) Manuel Fuentes (ECMWF) Erdem Erdi (Turkish State Meteorological Service)
Chapter 6 – Architectural Design 1Chapter 6 Architectural design Software Engineering Ian Sommerville, Software Engineering, 9 th Edition Pearson.
1 Ircam Artistic Testbed Jerome Barthelemy, Ircam.
Page 1 LAITS Laboratory for Advanced Information Technology and Standards Duh 7/10/03 The GMU Geospatial Grid Technology Development and Application Project.
Mitglied der 1 DGD 2.0: A Web-based Navigation Platform for the Visualization, Presentation and Retrieval of German Speech Corpora Joachim Gasch
SolidWorks Enterprise PDM and Microsoft SharePoint Interoperability Marc Young xLM Solutions, LLC.,
An Introduction to Repositories Thornton Staples Director of Community Strategy and Alliances Director of the Fedora Project.
Workshop on Metadata Standards and Best Practices November th, 2007 Session 4 The Data Documentation Initiative Technical Overview Pascal Heus Open.
MPEG-21 Multimedia Framework Dwight Borses MTS FAE SW Region OCCS Nov 25, 2002.
Integrating metadata schema registries with digital preservation systems to support interoperability Michael Day UKOLN, University.
DuraSpace: Digital Information All Ways, Always Pretoria, South Africa May 14 th, 2009.
Chapter 7 – Design and Implementation 1Chapter 7 Design and implementation Note: These are a modified version of Ch 7 slides available from the authors.
1 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
The OAIS Reference Model: current implementations Michael Day, UKOLN, University of Bath Chinese-European Workshop.
The PLANETS Testbed DPE, PLANETS and CASPAR 2nd Annual Conference Lisbon, 5–6 September, 2007 Max Kaiser, Austrian National Library
1 Rosetta at ETH Zurich – routes into the digital archive IGeLU 7th Conference Zurich, 11th to 13th September 2012 Dr. Matthias Töwe.
Pulling it all together… with thanks to Sheila Anderson.
Chapter 6 Architectural Design Slide 1 Chapter 6 Architectural Design.
A centre of expertise in data curation and preservation eScience Collaborative Workshop, Imperial College, 16 th October 2007 Funded by: This work is licensed.
© 2016 SlidePlayer.com Inc. All rights reserved.