1. Implementation of GLUE 2.0 support in the EMI Data Area Elisabetta Ronchieri on behalf of JRA1’s GLUE 2.0 Working Group INFN-CNAF 13 April 2011, EGI User Forum, Vilnius

2. EMI INFSO-RI-261611 Overview GLUE details Working Group Implementation State Future Work Conclusions

3. EMI INFSO-RI-261611 Overview GLUE details Working Group Implementation State Future Work Conclusions

4. EMI INFSO-RI-261611 GLUE goals GLUE describes the Grid infrastructure – GLUE: Grid Laboratory Uniform Environment Why is it used? – Service discovery – Monitoring – Brokering – Accounting – …

5. EMI INFSO-RI-261611 GLUE 1.3 GLUE 1.3 is currently widely deployed – In addition to Nordugrid schema, UNICORE What’s wrong with GLUE 1.3? – Protocol architecture: gridftp, gram… comparing with service architecture – Numerous details e.g., SE/CE relationships e.g., SRM is published as two objects (GlueSEControlProtocol and GlueService) – Not easily extensible

6. EMI INFSO-RI-261611 GLUE 1.3 GLUE 1.3 is currently widely deployed – In addition to Nodugrid schema from UNICORE What’s wrong with GLUE 1.3? – Protocol architecture: gridftp, gram… comparing with service architecture – Numerous details e.g., SE/CE relationships e.g., SRM is published as two objects (GlueSEControlProtocol and GlueService) – Not easily extensible GLUE 1.3 is no longer able to provide an accurate description of Grid resources

7. EMI INFSO-RI-261611 How is it used? How is it implemented now? – BDIIs – Info Providers – Clients, LDAP Site BDII Resource BDII Site BDII Resource BDII Provider Top BDII Resource BDII Resource BDII Top BDII

8. EMI INFSO-RI-261611 GLUE 2.0 It is an OGF standard (GFD-R-P.147) – Ratified March 2009 It is described as an abstract schema – A separate document describes LDAP rendering

9. EMI INFSO-RI-261611 GLUE 2.0 Storage Schema Storage Service Data Store Storage Manager Storage Share Storage End Point Storage Share Capacity Defined onMaps User to Storage Capacity Has Storage Access Protocol Offers Manages Has S. Burke, L. Field, and D. Horat, “Migration to the GLUE 2.0 information schema in the LCG/EGEE/EGI production Grid”, CHEP 2010, Taipei 1 2 3 4 5 6 7 8

10. EMI INFSO-RI-261611 Roll out Plan The plan was presented at CHEP 2010 (by S. Burke): 1.Define the abstract schema 2.Define the LDAP rendering 3.Implement the schema in the BDII and roll out 4.Write and deploy information providers 5.Update client tools to understand GLUE 2.0 6.Retire GLUE 1.3 Both GLUE versions will be published but with different bases: – “o=grid” for GLUE 1.3 – “o=glue” for GLUE 2.0

11. EMI INFSO-RI-261611 Roll out Plan The plan was presented at CHEP 2010 (by S. Burke): 1.Define the abstract schema 2.Define the LDAP rendering 3.Implement the schema in the BDII and roll out 4.Write and deploy information providers 5.Update client tools to understand GLUE 2.0 6.Retire GLUE 1.3 Both GLUE versions will be published but with different bases: – “o=grid” for GLUE 1.3 – “o=glue” for GLUE 2.0 Completed during EGEE Ready for EMI 1 Ready for EMI 2 Far in the future

12. EMI INFSO-RI-261611 Overview GLUE details Working Group Implementation State Future work Conclusions

13. EMI INFSO-RI-261611 The Working Group Purpose is – To ensure the consistent implementation of GLUE 2.0 across the Data Area Representatives are – SEs: dCache, DPM, StoRM – Clients: dCache, gLite, ARC, UNICORE Timetable is as follows (from DJRA1.2.1) – Common agreement on the interpretation of the GLUE 2.0 schema [PM6] – Publishing GLUE 1.3 data with GLUE 2.0 schema [PM 12] – EMI data components fully GLUE 2.0 compatible [PM 24] Useful links – https://twiki.cern.ch/twiki/bin/view/EMI/EmiJra1GLUE20 https://twiki.cern.ch/twiki/bin/view/EMI/EmiJra1GLUE20 – https://twiki.cern.ch/twiki/bin/view/EMI/EmiJRA1GLUE20FAQ

14. EMI INFSO-RI-261611 Working Group Decisions We are only considering current LDAP Information System We will validate published data using gstat 2.0 – Via “glue validator” provided by EMI infrastructure area https://tomtools.cern.ch/confluence/display/IS/GLUEValidator It performs generic checks for schema compliance This can be extended to encapsulate the decisions of the Working Group, e.g., to enforce the presence of certain attributes that are optional in the LDAP schema

15. EMI INFSO-RI-261611 Working Group Decisions (examples) Optional attributes are optional – Exception list: *size attributes How do we ensure unique IDs? – Implementer should pick a hostname which is associated with the service and reasonably persistent, and append the string ‘/data’. They can then append further information as they see fit. Note that this is not intended to be parsed. – e.g., srm.example.org/data/opaque1 Which Type is my service? – servicetype_t is an open enumeration. Use a type of org.emi-eu., e.g., org.emi-eu.dCache. What do the *Size attributes mean? – Where possible, the “Usage of Glue Schema v1.3 for WLCG Installed Capacity Information” should be considered, being the current authoritative position. This may require further discussion.

16. EMI INFSO-RI-261611 Overview GLUE Details Working Group Implementation State Future Work Conclusions

17. EMI INFSO-RI-261611 Work Completed GLUE 2.0 info providers implemented for all three Storage Elements (dCache, DPM and StoRM) – Publishing the same info as for GLUE 1.3 – DPM was installed on the SA2 testbed at CERN – StoRM is under installation on the SA2 testbed at INFN-CNAF – dCache is under installation on the SA2 testbed at DESY FTS updated to publish to GLUE 2.0

18. EMI INFSO-RI-261611 DPM The dpm-listspaces utility (maintained at LAL) operates as the DPM info provider It now has a “--glue2” option to output all relevant info in the GLUE 2.0 schema Main issue was understanding how to incorporate GLUE 1.3’s GlueVOInfo into GLUE 2.0 – The answer was multiple Storage Shares with a common SharingID

19. EMI INFSO-RI-261611 dCache GLUE info-provider The info-provider has two parts: 1.A generic framework for converting xml into ldif called xylophone 2.A dCache specific configuration for xylophone Xylophone is platform independent – Can run on any platform with XSLT or Java. The configuration is a domain- specific language: – Changing what is published is easy.

20. EMI INFSO-RI-261611 StoRM GLUE dynamic info-provider The dynamic info-provider is written as a bash shell script and configured by using yaim.

21. EMI INFSO-RI-261611 Storage Space Concepts Total Size (TS) is the total amount of storage of a given Type (e.g., online, nearline, and so on)

22. EMI INFSO-RI-261611 Storage Space Concepts Used Size (US) is the amount of storage occupied by stored data.

23. EMI INFSO-RI-261611 Storage Space Concepts Reserved Size (RS) is the amount of storage reserved for use by a specific user.

24. EMI INFSO-RI-261611 Storage Space Concepts Free Size (FS)=TS-US-RS

25. EMI INFSO-RI-261611 Storage Space Concepts FS contains Available Size and Unavailable Size

26. EMI INFSO-RI-261611 All info-providers in EMI-1 Support for GLUE 2.0 comes with EMI-1. Info-providers can publish GLUE 1.3, 2.0 or both in parallel: – GLUE 1.3 and GLUE 2.0 are on separate bases; – The default is to publish both. GLUE 2.0 base has more or less the same information as GLUE 1.3.

27. EMI INFSO-RI-261611 Overview GLUE Details Data GLUE 2.0 Working Group Work Plan Implementation State Future Work Conclusions

28. EMI INFSO-RI-261611 Future work Update clients – Gfal, FTS Extend existing information to make a better use of GLUE 2.0 Fix problems discovered when validating production data services

29. EMI INFSO-RI-261611 Overview GLUE Details Data GLUE 2.0 Working Group Work Plan Implementation State Future Work Conclusions

30. EMI INFSO-RI-261611 Conclusions Data PTs have delivered info-providers for GLUE 2.0 on time They keep backward compatibility by publishing both GLUE 1.3 and 2.0 Updating clients to query GLUE 2.0 is the next step The EMI project is on track to deliver GLUE 2.0

31. EMI INFSO-RI-261611 Acknowledgements GLUE 2.0 Working Group emi-jra1-data@eu- emi.euemi-jra1-data@eu- emi.eu dCache Product Team DPM Product Team StoRM Product Team

32. EMI is partially funded by the European Commission under Grant Agreement RI-261611 Thank you!