Towards smart storage for repository preservation services Steve Hitchcock, David Tarrant, Adrian Brown 1, Ben O’Steen 2, Neil Jefferies 2 and Leslie Carr Preserv 2 Project School of Electronics and Computer Science, University of Southampton 1 The National Archives, Kew 2 Oxford University Library 2008: The Fifth International Conference on Preservation of Digital Objects, London, September 2008

Three-stage strategy for keeping your data safe Ability to move data freely, easily and instantly –OAI, ORE, Atom Reliable, trusted large-scale storage –Open Storage Risk profiling: invoke a range of selectable services –Smart storage

About institutional repositories Set up by institutions of higher education and research to manage and disseminate their digital intellectual outputs. IRs are a special type of Web site, typically based on some repository software that presents a database of records pointing to the objects deposited. The Preserv 2 project is investigating the provision of preservation services for IRs. IRs in flux Uncertainty in terms of target content - published papers, theses, research data, teaching materials - policy, rights, even locus of content and responsibility for long- term management. OAI-ORE (Object Reuse and Exchange) effectively frees the data from being captive to repository software. Commercial repository services, from software- specific services to digital library services or more general 'cloud' or network storage services. Photo: Flickr/cpikas Flickr/cpikas

IRs are Open source repository softwares Open access content Open archives using OAI-PMH to share data with e.g. discovery services. Open repositories, using OAI-ORE enables the easy movement of data between different types of repository software Photo: Flickr/RighteeFlickr/Rightee

A new ‘open’ How open storage supports preservation services Open storage, large-scale storage devices based on open source software Open storage averts the need for a repository layer to access first- class objects – these are objects that can be addressed directly –In turn, these digital objects can be distributed and/or replicated over many open storage platforms. –In turn, able to select storage with built-in preservation support –Resilient storage platforms may be viable for preservation services aimed at multiple repositories E.g. Sun Microsystems STK5800 (codenamed Honeycomb) Google Repository

Smart storage Smart storage combines an underlying passive storage approach with the intelligence provided through services. The key to realising smart storage is to enable the services to communicate and share information with the digital content sources they may be acting on. This is done through machine-level application programming interfaces (APIs) and protocols.

APIs, interfaces and the Web architecture Major services on the Web, such as deploy their own simple, but different, APIs, e.g. –Google Maps –Within the repository community, SWORD (Simple Web-service Offering Repository Deposit) –Open storage platforms such as Sun's STK5800 and the Amazon Simple Storage Service (S3) To take advantage of open storage, repositories have to be able to talk to these services through their APIs.

Smart storage example: format services Preservation methods affecting formats can be classified in three stages (‘seamless flow’): –Format identification and characterization (which format?) –Preservation planning and technology watch (format risk and implications) –Preservation action, migration, etc. (what to do with the format) Format-based services tend to be ad hoc processes for which some tools are available –E.g. PRONOM-DROID from The National Archives (UK) –PRONOM is an online registry of technical information, such as file format signatures –DROID is a downloadable file format identification tool that applies these signatures) These and other tools could be used in a more coordinated manner.

Smart storage DROID: concept

Smart storage DROID: scheduling/history Scheduling interface controls when a DROID classification needs to be performed. Preserv 2 has developed a scheduling service that uses the Darwin Calendar Server and iCalendar format. Provides a powerful scheduling service with many clients already available - Apple iCal, Mozilla Sunbird, and others - that can read and interpret the files so that past and future events can be reviewed.

Smart storage DROID: OAI-PMH interface An OAI-PMH interface to open storage discovers the latest objects to have been deposited and which are ready for format classification. Could also be performed by simpler RSS or Atom-based methods. The interface has since been expanded to allow export of OAI-ORE resource maps in both RDF and Atom formats.

Smart storage DROID: implementation E.g. iCal, Outlook, Sunbird DROID Messaging History Open storage OAI-PMH Web server HTTP Stores results of DROID events Calendar server Repository Atom? Schedule event Is event done? Get results of event url, date User interface Machine interface, API ImplementedTo be implemented Scheduler DROID-OAI harvester

Risk profiling The scheduler will invoke actions based on the results of scanning by DROID allied to decision-making tools that use intelligence from planning and technology watch tools, such as –PRONOM, –Plato preservation planning tool from the EC-funded Planets project, –and others. Photo: Flickr/yourbartender Flickr/yourbartender

Summary: smart storage in the storage scheme Binary stream File systemneed to store multiple streams with permissions Content addressableadds content validation and object identifiers, metadata required to locate an object Openadds error correction and recovery, places processing close to storage, solves some bandwidth problems Smartopens up the close-to-storage approach for application development, transition to 'cloud' storage How smart storage addresses current storage issues – see full paper

Storage can become smarter Openness, in its various forms, the ability to move data freely and easily, needs to be supplemented by decision-making that can be automated based on the supplied intelligence and information. In this way, open storage can become ‘smarter’. Thanks to