Metadata Mòrag Burgon-Lyon University of Glasgow

31 st January 2005Metadata Contents Overview gLite Adoption Use Cases Schema OGSA-DAI Evaluation Further information

31 st January 2005Metadata Overview The Metadata group exists to examine commonalities across all the High Energy Physics experiments' metadata handling, at the technology, interface and schema level. The aim of the group is to ensure metadata services are deployed as much as possible using web services and other grid standards.

31 st January 2005Metadata Contents Overview gLite Adoption Use Cases Schema OGSA-DAI Evaluation Further information

31 st January 2005Metadata gLite Adoption Classes have been deployed as web services using Apache Axis Skeletons of web service clients have been produced in Java and Python –AttributeMetadataCatalog Specific methods, e.g. ‘queryByAttributes’, ‘remove’ Maps to AMI command e.g. AMI ListDataset –QueryMetadataCatalog Supports AMI and SQL Queries Uses SAX XML parser to interpret AMI response to produce a list of logical datasets Atlas have implemented the gLite QueryMetadataCatalog and AttributeMetadataCatalog interfaces as wrappers for an AMI backend:

31 st January 2005Metadata Contents Overview gLite Adoption Use Cases Schema OGSA-DAI Evaluation Further information

31 st January 2005Metadata Use Cases Reviewed existing Use Case documentation: –HEPCAL I & II –CDF Note 5858 / SAMGrid / D0 & CDF –BaBar Analysis Grid Application – Use Case and Requirements Document –Atlas – Catalog Services for Atlas Held discussions with representatives of the experiments, ARDA and EGEE.

31 st January 2005Metadata Use Cases Thirteen Core Use Cases of HEP Metadata “Unlucky for some?” ases/CoreUseCases_v10.pdf Data handling –Specify a new dataset –Read metadata for datasets –Update metadata for a dataset –Resolve physical data –Access data in a dataset

31 st January 2005Metadata Use Cases Analysis –Run a physics simulation program –Select a subset of a dataset –Run an algorithm over an input dataset Job handling –Submit a job to a Grid –Retrieve/Access the output of a job –Estimate the system resources cost of running a job –Monitor the progress of a job –Repeat a previous job

31 st January 2005Metadata Use Cases Conclusions: Highlighted occasional conflicting use-cases (for updating metadata). General consensus on 13 Core Use Cases. More feedback welcome.

31 st January 2005Metadata Contents Overview gLite Adoption Use Cases Schema OGSA-DAI Evaluation Further information

31 st January 2005Metadata Schema Looking at different approaches to database schema with two case studies: –AMI –SAM Aim to document the common elements and note the differences in both schema and approach.

31 st January 2005Metadata Schema AMI Schema: Stores Atlas production data for Data Challenges Simply supports different schema for different projects Allows schema to be changed easily User accesses multiple DB’s via generic commands and a router database

31 st January 2005Metadata Schema

31 st January 2005Metadata Schema AMI future plans – Implement the catalogues defined by David Adams and the ADA group: –Consists (roughly) of a selectableDataset catalogue with links to physics properties. –Related to a virtual dataset catalogue containing recipes for creating dataset instances, and a concrete dataset catalogue for finding files.

31 st January 2005Metadata Schema SAM Schema Mature system which has been in production for a number of years Stores real physics data Schema updates take between days and weeks to complete due to integration testing New code – Dimension Editor under development to allow run-time addition of new dimensions

31 st January 2005Metadata Schema

31 st January 2005Metadata Schema SAM future plans Dimension Editor development –Convert ‘SAM translate constraints’ commands into SQL queries –Optimise queries using ‘Chains’ and ‘Links’ tables. The Chains table describes the complete definition of join paths all the way back to the fact destination table. The Links table depicts the actual join details needed to follow the chains. –Allow the creation of new dimensions, built from queries, stored in the ‘Chains’ and ‘Links’ tables. Each time a new dimension is added, by walking through the paths of all Links on the Chains described the validity of a new table addition can be verified.

31 st January 2005Metadata Contents Overview gLite Adoption Use Cases Schema OGSA-DAI Evaluation Further information

31 st January 2005Metadata OGSA-DAI Evaluation Evaluation to establish appropriateness of OGSA-DAI for the HEP community. Initial investigations highlights: Pros –Web service (SOAP) access to database –Allows deployment of OGSA-DQP (Distributed Query Processor) Cons –Difficult installation due to poor packaging – mass deployment would be problematic –OGSA-DAI is heavy-weight in terms of dependencies

31 st January 2005Metadata OGSA-DAI Evaluation Initial conclusions –SOAP and DQP both attractive features. Migration to Globus Toolkit 4 might reduce the dependencies issue. Will keep an eye on future OGSA-DAI releases –Further evaluation required, especially through-put tests. Full document: dai.pdf

31 st January 2005Metadata Further Information GridPP Metadata pages: Metadata Wiki: Mailing List: