1. EGRIS-1 www.eu-eela.org E-infrastructure shared between Europe and Latin America AMGA Metadata Services: examples and usage scenarios Tony Calanducci INFN Catania Itacuruca, 04-15 December 2006

2. E-infrastructure shared between Europe and Latin America EGRIS-1 Outline AMGA Basics –Metadata concepts and AMGA terminology –Collection and Schema management –Queries –Command Line Tools and API Use AMGA to exchange data among running grid jobs Secure access to metadata

3. E-infrastructure shared between Europe and Latin America EGRIS-1 Metadata concepts and terminology –Entries – Entities/Objects which we are attaching metadata to –Metadata - List of attributes (including their values) associated with entries –Schema – A set of attributes –Attribute – key/value pair  Type – The type (int, float, string,…)  Name/Key – The name of the attribute  Value - Value of an entry's attribute –Collection – A set of entries associated with a schema (AMGA Collection are hierarchical organized) –Analogy to the RDBMS world: think of collections as tables, attributes as columns, entries as rows

4. E-infrastructure shared between Europe and Latin America EGRIS-1 Example: Movie Trailers Movie trailers files saved on Grid Storage Elements and registered into a LFC File Catalogue We have a LFN (Logical File Name) per movie files We want to add metadata to describe movie content. Possible schema: –Title -- varchar –Rutime -- int –Cast -- varchar –LFN -- varchar AMGA will be the repository of the movies’ metadata

5. E-infrastructure shared between Europe and Latin America EGRIS-1 gMOD: grid Movie On Demand gMOD provides a Video-On-Demand service User chooses among a list of video and the chosen one is streamed in real time to the video client of the user’s workstation For each movie a lot of details (Title, Runtime, Country, Release Date, Genre, Director, Case, Plot Outline) are stored and users can search a particular movie querying on one or more attributes Two kind of users can interact with gMOD: TrailersManagers that can administer the db of movies (uploading new ones and attaching metadata to them); GILDA VO users (guest) can browse, search and choose a movie to be streamed.

6. E-infrastructure shared between Europe and Latin America EGRIS-1 gMOD under the hood Built on top of gLite services: Storage Elements, sited in different place, physically contain the movie files LFC, the File Catalogue, keeps track in which Storage Element a particular movie is located AMGA is the repository of the detailed information for each movie, and makes possible queries on them The Virtual Organization Membership Service (VOMS) is used to assign the right role to the different users The Workload Management System (WMS) is responsible to retrieve the chosen movie from the right Storage Element and stream it over the network down to the user’s desktop or laptop

7. E-infrastructure shared between Europe and Latin America EGRIS-1 gMOD interactions VOMS LFC File Catalogue Metadata Catalogue WNWN WN CE Storage Elements User Genius Portal Workload Management System get Role AMGA

8. E-infrastructure shared between Europe and Latin America EGRIS-1 gMOD screenshot gMOD is accesible through the Genius Portal (https://glite-tutor.ct.infn.it)

9. E-infrastructure shared between Europe and Latin America EGRIS-1 Accessing AMGA services from UI/WNs Command Line Tools: –mdclient (interactive, like mysql & psql CLI) –mdcli (run one single command, useful into scripts) APIs: –C++ –Java –Python/Perl/PHP Client configuration file: –/opt/glite/etc/mdclient.config (default for all UI users) –$HOME/.mdclient.config (overrides the default one) –My suggestion, copy the default into $HOME/.mdclient.config and customize it –Useful options to set up:  Host = egris196.eela.ufrj.br  Login = NULL (or your AMGA username we’ve created: itacurucaXX)  UseSSL = require  AuthenticateWithCertificate = 1  UseGridProxy = 1

10. E-infrastructure shared between Europe and Latin America EGRIS-1 Accessing AMGA services from UI/WNs Be sure you have a valid proxy –with VOMS extensions if you set up Login = NULL (authenticated as the gilda generic user….all VO users will see your entries) –if you set up Login = yourusername the VOMS extensions will not take into account (so do not care about them) Start the AMGA client with –$ mdclient Try –whoami –pwd –help –grp_member

11. E-infrastructure shared between Europe and Latin America EGRIS-1 AMGA Basic Commands(I) Entries/Collection listing –dir –ls (ls -l) Entries/Collection permissions –acl_show entry | collection –stat entry | collection Collection and schema creation : –createdir collection_pathname  createdir /egris/yoursurname –cd collection_pathname) --- change the current collection  cd /egris/yoursurname –createdir trailers –cd trailers –addattr dir attr_name attr_type  addattr. Title varchar  addattr. Runtime int  addattr. Cast varchar  addattr. LFC varchar

12. E-infrastructure shared between Europe and Latin America EGRIS-1 AMGA Basic Commands (II) Inspecting collection’s schema: –listrattr fullpath_collectioname  (listrattr /egris/yoursurname/trailers Sequence managements: –they are used to create autoincremental IDs –sequence_create seq_name dir [increment] [start value]  sequence_create seq1 /egris/yoursurname/trailers –sequence_next seq_name  sequence_next /egris/yoursurname/trailers/seq1 Entries creations: –addentries entry_name (attribute_name value)+  cd /egris/yoursurname/trailers  addentry 1 Title 'Matrix' Runtime 110 Cast 'Keanu Reeves‘ LFN 'lfn:/grid/gilda/tony/matrix.mov‘  addentry 2 Title 'Notting Hill' Runtime 99 Cast 'Julia Roberts, Hugh Grant' LFN 'lfn:/grid/gilda/tony/notting.mov‘  addentry 3 Title 'Anger Management' Runtime 120 Cast 'Adam Sandlan, Jack Nicholson' LFN 'lfn:/grid/gilda/scardaci/anger.mov'

13. E-infrastructure shared between Europe and Latin America EGRIS-1 Other basic useful commands Set/edit attributes values: –setattr entry (attribute value)+ Get attributes values: –getattr pattern (attribute)+  getattr /egris/yoursurname/trailers/ Title LFN Removing attributes –removeattr dir attr_name Removing entries –rm pattern

14. E-infrastructure shared between Europe and Latin America EGRIS-1 Making queries AMGA provides a SQL-like selectattr statement: –selectattr attrs... condition  where: attrs is a space-separated list of attributes with the following format: collection_name:attribute_name (ex.: /egris/yoursurname/trailers:Title) condition rapresents one or more condition to select entries. Allowed operators are comparison operator (,=), logical operator (and, or), various operator (like, limit), etc..There is support also for aggregator operator (count) and order clause. ex: –selectattr /egris/yoursurname/trailers:Title.:LFN 'like(Title, "Anger%")' –selectattr /egris/yoursurname/trailers:Title.:LFN.:Cast 'like(Cast, "%ndle%") and Runtime > 80'

15. E-infrastructure shared between Europe and Latin America EGRIS-1 Updating attributes on condition selectattr and setattr can be used to update entries attributes’ values updateattr allows to do multiple updating on condition and ATOMICALLY –updateattr pattern attr expression [attr expression]... condition –ex:  getattr /egris/yoursurname/trailers/ Runtime >> 2 >> 99 >> 1 >> 111 >> 3 >> 121  updateattr /egris/yoursurname/trailers/ Runtime Runtime+1 'Runtime > 100‘  getattr /egris/yoursurname/trailers/ Runtime >> 2 >> 99 >> 1 >> 112 >> 3 >> 122

16. E-infrastructure shared between Europe and Latin America EGRIS-1 Access AMGA from WNs with mdcli All commands available through mdclient can be issued with mdcli. This will: –connect to the AMGA server defined into mdclient.config –run the command –exit to the shell Very useful in shell script (ex. the one running in the WN during job execution) Ex: – ID=`mdcli sequence_next /egris/yoursurname/seq1` – mdcli addentry /egris/yoursurname/${ID} LFN \'${LFN}\' Analyzed 0 – LFN=`mdcli selectattr $AMGA_HOME:LFN \'$AMGA_HOME:FILE = ${ID}\'` N.B.: If you need to use single quote, escape them with \

17. E-infrastructure shared between Europe and Latin America EGRIS-1 Use AMGA to exchange data among running jobs Suppose we have two sets of jobs: –Producers: they generate a file, store on a SE, register it onto the LFC File Catalogue assigning a LFN –Consumers: they will take a LFN, download the file and elaborate it AMGA can be used to share the information generated by the Producers, it could act as a “bag-of-LFNs” (bag- of-task model) from which Consumers can fetch file for further elaboration

18. E-infrastructure shared between Europe and Latin America EGRIS-1 Producers-Consumers example Set up of the collection containing the “tasks” –createdir /egris/youraccount/tasks –cd /egris/youraccount/tasks –addattr. LFN varchar(255) –addattr. Analyzed int –addattr. MagicNumber int –listattr. >> LFN >> varchar(255) >> Analyzed >> int >> MagicNumber >> int –sequence_create seq1 /egris/youraccount/tasks

19. E-infrastructure shared between Europe and Latin America EGRIS-1 Producers-Consumers example Producers script excerpts: –#Produce your data and save it on the SE and reg on LFC –LFN=‘lfn:/grid/gilda/tony/bomdia.txt’ –ID=`mdcli sequence_next /egris/yourname/tasks/seq1` –mdcli addentry /egris/yourname/tasks/${ID} LFN \'${LFN}\' Analyzed 0 Consumers script excerpts: –AMGA_HOME=/egris/yoursurname/tasks –ID=`mdcli selectattr $AMGA_HOME:FILE \'$AMGA_HOME:Analyzed = 0\' | head -1` –if [ $ID ]; then; echo “Task to be processed found“; else; exit 1; fi –MAGIC=`echo $RANDOM` –mdcli updateattr $AMGA_HOME/$ID Analyzed 1 MagicNumber $MAGIC \'Analyzed = 0\' –MAGIC2=`mdcli selectattr $AMGA_HOME:MagicNumber \'$AMGA_HOME:FILE = ${ID}\'` –if [ $MAGIC=$MAGIC2 ]; then LFN=`mdcli selectattr $AMGA_HOME:LFN \'$AMGA_HOME:FILE = ${ID}\'` –else; echo "Entry already processed by someone else“; exit 1; fi –lcg-cp -v --vo gilda $LFN file:$PWD/temp.dat –#Consume your data

20. E-infrastructure shared between Europe and Latin America EGRIS-1 Demo Thanks to Fabio for the creations of the scripts!!!

21. E-infrastructure shared between Europe and Latin America EGRIS-1 Secure your metadata AMGA allows to restrict access to a collection (or even to specific entries) using ACLs. This can be done thanks to the commands provided to manage groups of users. Here there are some useful commands: –whoami -- Print the name of the current user –user_listcred user -- Lists all possible credentials of a user. Only root can inspect other's credentials. –grp_member -- Lists the groups a user belongs to –acl_show colletionpath -- Shows all access controls for a directory –grp_create groupname -- Creates a new group. The group created will have the following name owner:groupName –grp_adduser groupname user -- Adds a user to a group –grp_show groupname -- Lists the members of a group –acl_add directory group rights -- Adds a new access control to a directory granting a group certain rights –acl_remove directory group -- Removes an access control for a group from a directory

22. E-infrastructure shared between Europe and Latin America EGRIS-1 Example (I) Query> whoami >> tcaland Query> user_listcred tcaland >> 1f86e515aa56dfb8b75ed7daaf63f52e97b8678 >> 'C = IT, O = GILDA, OU = Personal Certificate, L = INAF Catania, CN = Tony Calanducci, emailAddress = tony@calanducci.it', 'C = IT, O = GILDA, OU = Personal Certificate, L = INFN Catania, CN = Tony Calanducci, emailAddress = tony.calanducci@ct.infn.it', 'C = IT, O = GILDA, OU = Personal Certificate, L = ISCHIA, CN = ISCHIA15, emailAddress = roberto.barbera@ct.infn.it' Query> grp_member >> gilda:users Query> acl_show /egris06 >> root rwx >> gilda:users rx >> egris:users rwx Query> acl_show /egris06/tcaland >> tcaland rwx >> gilda:users rx >> egris:users rwx

23. E-infrastructure shared between Europe and Latin America EGRIS-1 Example (II) Query> grp_create myfriends Query> grp_adduser myfriends tcaland Query> grp_adduser myfriends emidio Query> grp_adduser myfriends valeria Query> grp_show myfriends >> tcaland >> emidio >> valeria Query> grp_member >> egris:users >> tcaland:myfriends Query> acl_add /egris06/tcaland myfriends rx Query> acl_show /egris06/tcaland >> tcaland rwx >> gilda:users rx >> egris:users rwx >> tcaland:myfriends rx Query> acl_remove /egris06/tcaland gilda:users Query> acl_show /ischia06/tcaland >> tcaland rwx >> egris:users rwx >> tcaland:myfriends rx

24. E-infrastructure shared between Europe and Latin America EGRIS-1 References AMGA Project Homepage –http://amga.web.cern.ch/amga/http://amga.web.cern.ch/amga/ AMGA 1.2.7 User Manual –http://amga.web.cern.ch/amga/downloads/amga- manual_1_2_7.pdfhttp://amga.web.cern.ch/amga/downloads/amga- manual_1_2_7.pdf Exercise documentation from ISSGC’06: –http://www.dma.unina.it/~murli/ISSGC06/glite/public_html/amga. htmlhttp://www.dma.unina.it/~murli/ISSGC06/glite/public_html/amga. html –http://www.dma.unina.it/~murli/ISSGC06/glite/public_html/summ aryExercise.htmhttp://www.dma.unina.it/~murli/ISSGC06/glite/public_html/summ aryExercise.htm AMGA GILDA Wiki pages: –https://grid.ct.infn.it/twiki/bin/view/GILDA/AMGAHandsOnhttps://grid.ct.infn.it/twiki/bin/view/GILDA/AMGAHandsOn –https://grid.ct.infn.it/twiki/bin/view/GILDA/AMGAAdvhttps://grid.ct.infn.it/twiki/bin/view/GILDA/AMGAAdv