1. Secure Location-Independent Autonomic Storage Architectures GR/S44501/01 February 2004 - January 2007 Graham Kirby, Alan Dearle, Ron Morrison & Stuart Norcross School of Computer Science, University of St Andrews {graham, al, ron, stuart}@dcs.st-and.ac.uk

2. EPSRC e-Science 26/3/042 Project Aims Desirable features of a data storage system Desirable features of a data storage system unbounded capacity unbounded capacity zero latency & cost zero latency & cost total reliability total reliability location independence location independence simple interface simple interface complete security complete security complete historical archive complete historical archive Aim: a storage architecture approximating above, focusing on: Aim: a storage architecture approximating above, focusing on: simple interface for end user (file system) simple interface for end user (file system) abstracting over: abstracting over: user location user location physical devices physical devices provision of significant benefits with acceptable cost provision of significant benefits with acceptable cost

3. EPSRC e-Science 26/3/043 Potential Benefits Simplify user experience Simplify user experience ‘home directory’ ubiquitously available, irrespective of: ‘home directory’ ubiquitously available, irrespective of: machines and disks machines and disks physical location physical location firewalls firewalls data highly durable data highly durable no need for backup no need for backup simple data sharing simple data sharing uniform global name space uniform global name space Historical views Historical views data never over-written data never over-written

4. EPSRC e-Science 26/3/044 Potential Hurdles to User Adoption Speed and convenience must be close enough to that of a local disk Speed and convenience must be close enough to that of a local disk Users must be able to trust system Users must be able to trust system not to allow inappropriate access to data by other users not to allow inappropriate access to data by other users to be sufficiently reliable for serious evaluation to be sufficiently reliable for serious evaluation Need viable exit strategy Need viable exit strategy may require that system can reproduce effects of user’s existing backup regime may require that system can reproduce effects of user’s existing backup regime e.g. by maintaining a local copy of all data e.g. by maintaining a local copy of all data Financial cost Financial cost Critical mass of nodes and users required Critical mass of nodes and users required envisaged architecture relies on autonomic management of large numbers of nodes envisaged architecture relies on autonomic management of large numbers of nodes Storage overhead must be low enough Storage overhead must be low enough incurred through replication of data incurred through replication of data

5. EPSRC e-Science 26/3/045 User Control End users should deal only with very high-level configuration End users should deal only with very high-level configuration set broad goals regarding trade-offs (or ignore completely) set broad goals regarding trade-offs (or ignore completely) task of autonomic management system to try to achieve these goals task of autonomic management system to try to achieve these goals Examples of trade-offs Examples of trade-offs speed of reads and writes speed of reads and writes durability durability related to number and placement of replicas related to number and placement of replicas both absolute & time to converge both absolute & time to converge consistency consistency how long before updates to shared data are visible to others? how long before updates to shared data are visible to others? resource consumption resource consumption storage, bandwidth, computation storage, bandwidth, computation

6. EPSRC e-Science 26/3/046 Control Example

7. EPSRC e-Science 26/3/047 Control and Feedback Example

8. EPSRC e-Science 26/3/048 Implementation Approach File system interface File system interface SMB or NFS SMB or NFS Replication of files or fragments Replication of files or fragments erasure-resilient encoding erasure-resilient encoding Placement of data Placement of data controlled explicitly controlled explicitly Routing to data Routing to data abstracted by peer-to-peer overlay e.g. Tapestry abstracted by peer-to-peer overlay e.g. Tapestry Probes & gauges to monitor state of system Probes & gauges to monitor state of system publish/subscribe infrastructure e.g. Siena publish/subscribe infrastructure e.g. Siena Autonomic management elements Autonomic management elements attempt to map user goals and probe events into suitable low- level actions attempt to map user goals and probe events into suitable low- level actions

9. EPSRC e-Science 26/3/049 Challenges Core distributed storage infrastructure Core distributed storage infrastructure appropriate replication mechanisms appropriate replication mechanisms Autonomic management Autonomic management low-level policies low-level policies probe & gauge infrastructure probe & gauge infrastructure high-level views for users high-level views for users synthesising views from low-level events synthesising views from low-level events heuristics for adapting low-level policies to achieve high-level goals heuristics for adapting low-level policies to achieve high-level goals Evaluation Evaluation simulation, local cluster, PlanetLab simulation, local cluster, PlanetLab end-user adoption end-user adoption

10. EPSRC e-Science 26/3/0410 Conclusions Aim to design, implement and evaluate distributed storage system targeted at benefits to end-user Aim to design, implement and evaluate distributed storage system targeted at benefits to end-user very simple interface very simple interface ubiquitously available ubiquitously available highly durable highly durable append-only: historical views append-only: historical views Project details Project details http://www-systems.dcs.st-and.ac.uk/asa/ http://www-systems.dcs.st-and.ac.uk/asa/