INFSO-RI-508833 Enabling Grids for E-sciencE Grid & Data Preservation Boon Low System Development, EGEE Training National.

Slides:

Advertisements

Similar presentations

Panel 2 – Promoting Re-Use of Scientific Collections John Harrison SHAMAN Project University of Liverpool

Advertisements

Data Publishing Service Indiana University Stacy Kowalczyk April 9, 2010.

Joint Information Systems Committee Supporting Higher and Further Education Portals and the JISC Information Environment Strategy Chris Awre Programme.

Philip LordDigital Archiving Consultancy Alison Macdonald Digital Archiving Consultancy Liz LyonDigital Curation Centre David GiarettaDigital Curation.

The PREMIS Data Dictionary Michael Day Digital Curation Centre UKOLN, University of Bath JORUM, JISC and DCC.

A centre of expertise in data curation and preservation EAOLUG :: RSC :: Cambridge23 May 2006 Funded by: This work is licensed under the Creative Commons.

UKOLN is supported by: JISC Information Environment update Repositories and Preservation Programme meeting, October 24-25, 2006 Rachel Heery UKOLN

Digital Repositories: interoperability & common services Closing Remarks Dr Liz Lyon, UKOLN, University of Bath, UK

A centre of expertise in data curation and preservation Preserving Digital ArchivesLUCAS March 2006 Funded by: This work is licensed under the Creative.

Pulling it all together… with thanks to Sheila Anderson.

Digital Preservation Lifecycle Management Building a demonstration prototype for the preservation of large-scale multi-media collections Arcot Rajasekar.

Data Management: Metadata, Repositories and Curation Tony Mathys, Anne Robertson Eddie Boyle, Guy McGarva GeoForum, 4 th November, York.

Supporting further and higher education Supporting Digital Preservation and Asset Management in Institutions eSPIDA event University of Glasgow 11 February.

Background Chronopolis Goals Data Grid supporting a Long-term Preservation Service Data Migration Data Migration to next generation technologies Trust.

Digital Collections: Use, Value and Impact Lorna Hughes University of Wales Chair in Digital Collections, National Library of Wales Aberystwth University.

A centre of expertise in data curation and preservation MIS Seminar :: University of Edinburgh :: 2 October 2006 Funded by: This work is licensed under.

Preservation and Long-term access through Networked Services Adam Farquhar, The British Library iPres2006 Cornell University, October 2006.

DSpace Rea Devakos and Gabriela Mircea University of Toronto Libraries.

Robust Tools for Archiving and Preserving Digital Data Joseph JaJa, Mike Smorul, and Mike McGann Institute for Advanced Computer Studies Department of.

Co-funded by the European Union under FP7-ICT Alliance Permanent Access to the Records of Science in Europe Network Co-ordinated by aparsen.eu #APARSEN.

Australian Partnership for Sustainable Repositories AUSTRALIAN PARTNERSHIP FOR SUSTAINABLE REPOSITORIES Caul Meeting 2005/2 Brisbane 15.

Digital | Curation | Centre The UK Digital Curation Centre Michael Day UKOLN, University of Bath (with thanks to Peter Burnhill, Chris Rusbridge, et al.)

© HATII, University of Glasgow Introduction to the UK ’ s Digital Curation Centre Prof Seamus Ross Visiting Fellow at Oxford Internet Institute ,

Metadata for preservation Michael Day, UKOLN, University of Bath Chinese-European Workshop on Digital Preservation,

Documenting to preserve your data: metadata in support of digital preservation Michael Day, UKOLN, University of Bath

David Giaretta Associate Director (Development) Funders: DCC Development Digital Curation Centre a centre of expertise in data curation and preservation.

Supporting further and higher education The UK FAIR Programme: OAI in context Chris Awre OAI3, CERN, February 2004.

A hybrid approach of digital long term preservation to institutional repositories - A case study of DSpace/SRB Integration Ya-ning Arthur Chen, Feng-chien.

Peter Burnhill Director (Phase One) Funders: Aims & Organisation Digital Curation Centre a centre of expertise in data curation and preservation.

Production Data Grids SRB - iRODS Storage Resource Broker Reagan W. Moore

INFSO-RI Enabling Grids for E-sciencE V. Breton, 30/08/05, seminar at SERONO Grid added value to fight malaria Vincent Breton EGEE.

File format registries - a global infrastructure for local persistence Andreas Aschenbrenner, ERPANET.

Libraries, Archives, and Digital Preservation: The Reality of What We Must Do Leslie Johnston Acting Director, National Digital Information Infrastructure.

Life Cycle Models & Principles Jake Carlson Associate Professor of Library Science Data Services Specialist Purdue University Libraries.

The Canadian Information Network for Research in the Social Sciences and Humanities Tim Au Yeung and Mary Westell Libraries.

David Giaretta Associate Director (Development) for Chris Rusbridge (Director) Funders: Digital Curation Centre a centre of expertise in data curation.

Digital preservation activities at the NLW Sally McInnes 18 September 2009.

Rule-Based Preservation Systems Reagan W. Moore Wayne Schroeder Mike Wan Arcot Rajasekar Richard Marciano {moore, schroede, mwan, sekar,

National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center Persistent Management of Distributed Data Reagan W. Moore.

Microsoft Research Faculty Summit Natasa Milic-Frayling & Vijay Rajagopalan Microsoft Corporation.

Metadata for digital preservation: a review of recent developments Michael Day UKOLN, University of Bath ECDL2001, 5th European Conference.

OAIS Rathachai Chawuthai Information Management CSIM / AIT Issued document 1.0.

Breakout # 1 – Data Collecting and Making It Available Data definition “ Any information that [environmental] researchers need to accomplish their tasks”

Funded by: © AHDS Preservation in Institutional Repositories Preliminary conclusions of the SHERPA DP project Gareth Knight Digital Preservation Officer.

Digital Preservation across the technologies, strategies, open standards & interoperability aspects including the legal issues Pratik Shrivastava Scientist.

UKOLN is supported by: Introduction to UKOLN Dr Liz Lyon, Director UKOLN, University of Bath, UK Grand Challenge Meeting, June a centre.

1 st African Digital Curation Conference Social Sciences & Humanities February 2008.

26/05/2005 Research Infrastructures - 'eInfrastructure: Grid initiatives‘ FP INFRASTRUCTURES-71 DIMMI Project a DI gital M ulti M edia I nfrastructure.

DSpace - Digital Library Software

The Importance of Standards in Digital Preservation Tina Norris Kayla Payne Jennifer

April 14, 2005MIT Libraries Visiting Committee Libraries Strategic Plan Theme III Work to shape the future MacKenzie Smith Associate Director for Technology.

Preservation metadata and the Cedars project Michael Day UKOLN: UK Office for Library and Information Networking University of Bath

Portico’s “d-collections” preservation service Stephanie Orphan Positive trends in sustainability? Emerging approaches to archiving commercial databases.

Lifecycle Metadata for Digital Objects November 15, 2004 Preservation Metadata.

Institutional Repositories July 2007 DIGITAL CURATION creating, managing and preserving digital objects Dr D Peters DISA Digital Innovation South.

National Archives and Records Administration1 Integrated Rules Ordered Data System (“IRODS”) Technology Research: Digital Preservation Technology in a.

Infrastructure Breakout What capacities should we build now to manage data and migrate it over the future generations of technologies, standards, formats,

Cedars work on metadata Michael Day UKOLN, University of Bath Cedars Workshop Manchester, February 2002.

Rights Management for Shared Collections Storage Resource Broker Reagan W. Moore

Long-term preservation and access: the UK context Michael Day, UKOLN, University of Bath RCUK Workshop on Publication.

New Opportunities Fund Preservation Workshop March 15th 2002 Maggie Jones Cedars Project Manager.

Introduction to Research Data Management Joy Davidson and Sarah Jones Digital Curation Centre

Building Preservation Environments with Data Grid Technology Reagan W. Moore Presenter: Praveen Namburi.

Collection-Based Persistent Archives Arcot Rajasekar, Richard Marciano, Reagan Moore San Diego Supercomputer Center Presented by: Preetham A Gowda.

Joint Information Systems Committee Repositories Support Project Summer School 2008 Amber Thomas, JISC.

Data Grids, Digital Libraries and Persistent Archives: An Integrated Approach to Publishing, Sharing and Archiving Data. Written By: R. Moore, A. Rajasekar,

Institutional Repositories

JISC and SOA A view Robert Sherratt.

Robin Dale RLG OAIS Functionality Robin Dale RLG

Presentation transcript:

INFSO-RI Enabling Grids for E-sciencE Grid & Data Preservation Boon Low System Development, EGEE Training National e-Science Centre

Enabling Grids for E-sciencE INFSO-RI Grid and Data Preservation Grid Technologies for Digital Libraries, Athens Topics Digital curation and UK Digital Curation Centre General preservation issues Preservation and data grid DSpace + SRB project

The actions needed to maintain digital research data and other materials over their life-cycle, for current and future generations. These actions include digital archiving and preservation, and good practice in data creation and management. Also, providing the capacity for adding value to data to generate new sources of information and knowledge. Digital curation: a definition

Why a national centre? “Long-term curation and preservation of digital resources is seen as a challenge which is difficult if not impossible for individual institutions to resolve on their own due to the complexity and scale of the challenges involved.” - JISC circular, 6/03 “Scientists and researchers across the UK generate increasingly vast amounts of digital data, with further investment in digitisation and purchase of digital content and information. The scientific record and the documentary heritage created in digital form are at risk from technology obsolescence and by the fragility of digital media.” - JISC press release, 3/04

Digital Curation Centre Established in 2004 under JISC/EPSRC funding Continuing quality improvement in data curation & digital preservation practice –Initial focus: data as evidence for scholarly conclusions –wider remit: scholarly communication & e-Learning Working with data repositories, rather than being a data centre Centre of excellence in research & service –Programmes to address wider issues of data curation –Evaluation of tools, standards and policies –Focal point for digital curators with repository of tools and technical information Connecting communities via Associates Network –universities & research institutes –scientific data tradition & document tradition –international & cross-sectoral

DCC people (some of them…) Management & Co-ordination –Director Chris Rusbridge (University of Edinburgh) Community Support & Outreach –Led by Dr Liz Lyon (UKOLN, University of Bath) Service Definition & Delivery –Led by Professor Seamus Ross (HATII [ERPANET], University of Glasgow) Development –Led by Dr David Giaretta (Astronomical Software & Services, CCLRC) Research –Led by Professor Peter Buneman (Informatics, University of Edinburgh)

Enabling Grids for E-sciencE INFSO-RI Grid and Data Preservation Grid Technologies for Digital Libraries, Athens Evolving curation picture Source: JCSR e-Science Curation report

Enabling Grids for E-sciencE INFSO-RI Grid and Data Preservation Grid Technologies for Digital Libraries, Athens Evolving curation picture Source: JCSR e-Science Curation report

Enabling Grids for E-sciencE INFSO-RI Grid and Data Preservation Grid Technologies for Digital Libraries, Athens Preservation Technology changes needs to be addressed to ensure the long termed usage of archives Changes may stem from applications, OS environments, database systems, hardware and the encoding format of data Some approaches for preservations: –Emulation: recreating the application in new technology environment while preserving the original data – Migration: preserving usability instead of the original data, by transforming it into usable format suitable for new software, technology – Preserving data and application contexts such as schema / dtds, or operations applied on data Involves the maintenance of preservation metadata, e.g: – descriptive, authenticity, structural Manages content (the data to be archived) and context (metadata)

Enabling Grids for E-sciencE INFSO-RI Grid and Data Preservation Grid Technologies for Digital Libraries, Athens Preservation environment & grid Involves extracting data from its creation and application contexts and storing them in a preservation environment A preservation environment can be built upon the grid infrastructure Data grid provides mechanisms to manage the evolution of technology infrastructure Grid middleware such as the SRB can be used to provide abstraction capabilities, for example: –Logical name space for files stored in distributed locations –Storage repository abstraction For additional data grid capabilities, see: –Documentation of SRB project –

Enabling Grids for E-sciencE INFSO-RI Grid and Data Preservation Grid Technologies for Digital Libraries, Athens Storage repository abstraction Data applications Database ADatabase B ~200 GB Data applications Heterogeneous storage: file systems, databases, archives Grid broker, e.g. gLite, SRB single storage resource

Enabling Grids for E-sciencE INFSO-RI Grid and Data Preservation Grid Technologies for Digital Libraries, Athens Data grid topology “Grid Bricks”, grid storage building blocks on dedicated storage server e.g. 10 x 200GB drives = 2 Terabytes 200 GB Rack of storage servers Data grid - as a single logical storage e.g. 5 x 2TB = 10 Terabytes storage servers Multiple storage server racks (in a room) e.g. 5 x 10TB = 50 Terabytes storage servers Data applications broker Data applications broker

Enabling Grids for E-sciencE INFSO-RI Grid and Data Preservation Grid Technologies for Digital Libraries, Athens Data grids federation Data Grid A Data applications broker Data applications broker Data Grid B Federation provides mechanisms to organise and manage data on multiple data grids, to extend storage capacity Interactions among grids is facilitated by the brokers There various approaches in data grids federations, e.g.: –Applications can share data on Grid A and Grid B as an aggregated data storage –Data on a grid can also be replicated automatically on another grid

Enabling Grids for E-sciencE INFSO-RI Grid and Data Preservation Grid Technologies for Digital Libraries, Athens Data grids federation large scale federation, e.g. “snow-flake” federation approach

Enabling Grids for E-sciencE INFSO-RI Grid and Data Preservation Grid Technologies for Digital Libraries, Athens Federation approaches See “Data grids federation”

Enabling Grids for E-sciencE INFSO-RI Grid and Data Preservation Grid Technologies for Digital Libraries, Athens Example: DSpace + SRB project DSpace is an open source digital library system providing: –Content/metadata management –Collection/user/communities administration –Digital content ingestion (batch upload) –Indexing, search and discovery –Dissemination services (alerting) –OAI Harvesting –Web UI and API for cross application context development Jointly developed by: –MIT Libraries (MIT) –Hewlett-Packard (HP) DSpace + SRB (Storage Resource Broker) is a project by: –San Diego Super Computing Center (SDSC) –MIT Libraries (MIT) –UC San Diego Libraries (UCSD) –US National Archives and Records Administration (NARA)

Enabling Grids for E-sciencE INFSO-RI Grid and Data Preservation Grid Technologies for Digital Libraries, Athens Example: DSpace + SRB project Goal is to extends DSpace storage capability by using data grid, in addition the existing SQL database system Replace DSpace file system calls with access calls to data grid Uses METS based Archival Information Package (AIP) DSpace SQL Database DSpace Data grid digital collection Data grid digital collection

Enabling Grids for E-sciencE INFSO-RI Grid and Data Preservation Grid Technologies for Digital Libraries, Athens

Enabling Grids for E-sciencE INFSO-RI Grid and Data Preservation Grid Technologies for Digital Libraries, Athens

Enabling Grids for E-sciencE INFSO-RI Grid and Data Preservation Grid Technologies for Digital Libraries, Athens

Enabling Grids for E-sciencE INFSO-RI Grid and Data Preservation Grid Technologies for Digital Libraries, Athens

Enabling Grids for E-sciencE INFSO-RI Grid and Data Preservation Grid Technologies for Digital Libraries, Athens

Enabling Grids for E-sciencE INFSO-RI Grid and Data Preservation Grid Technologies for Digital Libraries, Athens

Enabling Grids for E-sciencE INFSO-RI Grid and Data Preservation Grid Technologies for Digital Libraries, Athens

Enabling Grids for E-sciencE INFSO-RI Grid and Data Preservation Grid Technologies for Digital Libraries, Athens

Enabling Grids for E-sciencE INFSO-RI Grid and Data Preservation Grid Technologies for Digital Libraries, Athens

Enabling Grids for E-sciencE INFSO-RI Grid and Data Preservation Grid Technologies for Digital Libraries, Athens

Enabling Grids for E-sciencE INFSO-RI Grid and Data Preservation Grid Technologies for Digital Libraries, Athens For further information Curation, preservation, data grid DSpace + SRB project: