GridPP9 – 5 February 2004 – Data Management DataGrid is a project funded by the European Union GridPP is funded by PPARC WP2+5: Data and Storage Management Gavin McCance – University of Glasgow Jens Jensen - RAL University of Glasgow GridPP9 NeSC, Edinburgh

GridPP9 – 5 February 2004 – Data Management – n° 2Gavin McCance – University of Glasgow EDG Summary Report WP2: Data Management

GridPP9 – 5 February 2004 – Data Management – n° 3Gavin McCance – University of Glasgow The team u Based at Glasgow University n William Bell n David Cameron n Tony Doyle n Gavin McCance n Paul Millar n Caitriana Nicholson u We formed close collaboration with WP2 of European DataGrid n Became an integral part of the WP2 team n All our products were delivered jointly with EDG

GridPP9 – 5 February 2004 – Data Management – n° 4Gavin McCance – University of Glasgow What we set out to do? u Initial work-plan was in two key areas: u Grid Optimisation n Simulating the Grid from the data-centric viewpoint n Applying the results from the simulations to develop algorithms for optimisation of replicas n Development of optimisation software for replica management u Metadata management n Development of secure metadata services for the experiments

GridPP9 – 5 February 2004 – Data Management – n° 5Gavin McCance – University of Glasgow What we actually did… u Grid Optimisation n Developed a data-centric Grid simulation [OptorSim] (fully documented final release available now) n Compared of a variety of replica optimisation techniques and algorithms (many publications) n Worked on developing and refining these algorithms n Exploitation of some of the early work, we developed the Replica Optimisation Service (ROS) component of the Replica Management System s provides file access cost estimation to the user and the WP1 resource broker.

GridPP9 – 5 February 2004 – Data Management – n° 6Gavin McCance – University of Glasgow What we actually did… u Metadata Management n Early production of the Spitfire browser service allowing remote access to RDBMS via a web browser n Early adoption of GSI security into Spitfire to allow secure use of the service using Grid certificates n Initial adoption of web services into EDG with the creation of the web services API version of Spitfire, allowing programmatic access to remote RDMBS n The web services technology built for Spitfire was subsequently factored out and used in all other WP2 services n The security built into Spitfire was subsequently factored out and led to the creation of the WP2 Java security modules

GridPP9 – 5 February 2004 – Data Management – n° 7Gavin McCance – University of Glasgow What we actually did… u Replica Management Services n As focus of EDG changed to “stability, stability, stability” and production, we changed with it n Helped factor-out the web services and security frameworks from Spitfire n Moved into the core replica management services s Replica Manager s Replica Location Service s Replica Metadata Catalogue s Replica Optimisation Service n Provided continued support for these products to EDG and LCG n Led the thorough performance testing of our products for WP2’s final EU deliverable

GridPP9 – 5 February 2004 – Data Management – n° 8Gavin McCance – University of Glasgow What we actually did… u WP2 Integration and Testing n Significant integration expertise developed s Paul Millar is the WP2 integration team member n Glasgow ran the WP2 CVS repository n We created the testing framework for WP2 software u WP2 Testbed support n Provided support for the WP2 mini-testbed s Machines managed at ScotGrid and at CERN u WP2 Management n Gavin McCance is WP2 deputy u Major contributions to WP2 deliverables

GridPP9 – 5 February 2004 – Data Management – n° 9Gavin McCance – University of Glasgow What we actually did… u Standards n Contribution of GGF Data Area documents u Publications n Work was fully reported: 15 or so papers published… OptimisationMetadataReplicationGGFDeployment 63231

GridPP9 – 5 February 2004 – Data Management – n° 10Gavin McCance – University of Glasgow EDG Effort Delivered u Substantial UK effort delivered to EDG : ~150% of contracted

GridPP9 – 5 February 2004 – Data Management – n° 11Gavin McCance – University of Glasgow EDG Summary Report WP5: Storage Management (Jens Jensen)

GridPP9 – 5 February 2004 – Data Management – n° 12Gavin McCance – University of Glasgow The team u Based at RAL n Tim Eaves n Tim Folkes n John Gordon n Jens Jensen n Glen Johnson n Tara Shah n Bonny Strong n Owen Synge n Regina Tam u UK WP5 managed the WP5 team at CERN, SARA and IN2P3.

GridPP9 – 5 February 2004 – Data Management – n° 13Gavin McCance – University of Glasgow What we did… u Developed Storage Element n uniform control interface to MSS: disk, ADS, CASTOR, HPSS u SE provides additional functionality for storage systems n e.g. directory hierarchies for mass storage with flat filenames u Modular design allows easy deployment in front of different backing mass storage systems u SE provides disk cache in front of mass storage u SE can provide secure access (authentication and authorisation) to the mass storage resource.

GridPP9 – 5 February 2004 – Data Management – n° 14Gavin McCance – University of Glasgow What we did… u Demonstrated 3 rd party transfers between different mass storage systems u Work on interfacing with Replica Management service and with the information services u LCG now requires SRM v1 to access storage n EDG SE is architected for easy migration to this u Significant work in standards development n Input into the SRM standards process

GridPP9 – 5 February 2004 – Data Management – n° 15Gavin McCance – University of Glasgow What we did… u Overall WP5 Management n John Gordon and Jens Jensen u WP5 Integration and Testing n Significant integration expertise developed s Owen Synge and Regina Tam were EDG integration team members s Regina Tam was on the EDG Software Quality team u EDG Architecture Team n Jens Jensen was WP5 member of Architecture team. u UK responsible for all WP5 deliverables

GridPP9 – 5 February 2004 – Data Management – n° 16Gavin McCance – University of Glasgow EDG effort delivered u Substantial UK effort delivered to EDG : ~150% of contracted

GridPP9 – 5 February 2004 – Data Management – n° 17Gavin McCance – University of Glasgow WP2 + WP5: What we learned u Software development process… n sometimes learned the hard way…! u Much feedback from our users on the software n Very useful for the design process in the next projects u The amount of effort required to make production quality services out of our software u Take all this forward into GridPP2!