Implementation of GLUE 2.0 support in the EMI Data Area Elisabetta Ronchieri on behalf of JRA1’s GLUE 2.0 Working Group INFN-CNAF 13 April 2011, EGI User Forum, Vilnius

EMI INFSO-RI Overview GLUE details Working Group Implementation State Future Work Conclusions

EMI INFSO-RI Overview GLUE details Working Group Implementation State Future Work Conclusions

EMI INFSO-RI GLUE goals GLUE describes the Grid infrastructure – GLUE: Grid Laboratory Uniform Environment Why is it used? – Service discovery – Monitoring – Brokering – Accounting – …

EMI INFSO-RI GLUE 1.3 GLUE 1.3 is currently widely deployed – In addition to Nordugrid schema, UNICORE What’s wrong with GLUE 1.3? – Protocol architecture: gridftp, gram… comparing with service architecture – Numerous details e.g., SE/CE relationships e.g., SRM is published as two objects (GlueSEControlProtocol and GlueService) – Not easily extensible

EMI INFSO-RI GLUE 1.3 GLUE 1.3 is currently widely deployed – In addition to Nodugrid schema from UNICORE What’s wrong with GLUE 1.3? – Protocol architecture: gridftp, gram… comparing with service architecture – Numerous details e.g., SE/CE relationships e.g., SRM is published as two objects (GlueSEControlProtocol and GlueService) – Not easily extensible GLUE 1.3 is no longer able to provide an accurate description of Grid resources

EMI INFSO-RI How is it used? How is it implemented now? – BDIIs – Info Providers – Clients, LDAP Site BDII Resource BDII Site BDII Resource BDII Provider Top BDII Resource BDII Resource BDII Top BDII

EMI INFSO-RI GLUE 2.0 It is an OGF standard (GFD-R-P.147) – Ratified March 2009 It is described as an abstract schema – A separate document describes LDAP rendering

EMI INFSO-RI GLUE 2.0 Storage Schema Storage Service Data Store Storage Manager Storage Share Storage End Point Storage Share Capacity Defined onMaps User to Storage Capacity Has Storage Access Protocol Offers Manages Has S. Burke, L. Field, and D. Horat, “Migration to the GLUE 2.0 information schema in the LCG/EGEE/EGI production Grid”, CHEP 2010, Taipei

EMI INFSO-RI Roll out Plan The plan was presented at CHEP 2010 (by S. Burke): 1.Define the abstract schema 2.Define the LDAP rendering 3.Implement the schema in the BDII and roll out 4.Write and deploy information providers 5.Update client tools to understand GLUE Retire GLUE 1.3 Both GLUE versions will be published but with different bases: – “o=grid” for GLUE 1.3 – “o=glue” for GLUE 2.0

EMI INFSO-RI Roll out Plan The plan was presented at CHEP 2010 (by S. Burke): 1.Define the abstract schema 2.Define the LDAP rendering 3.Implement the schema in the BDII and roll out 4.Write and deploy information providers 5.Update client tools to understand GLUE Retire GLUE 1.3 Both GLUE versions will be published but with different bases: – “o=grid” for GLUE 1.3 – “o=glue” for GLUE 2.0 Completed during EGEE Ready for EMI 1 Ready for EMI 2 Far in the future

EMI INFSO-RI Overview GLUE details Working Group Implementation State Future work Conclusions

EMI INFSO-RI The Working Group Purpose is – To ensure the consistent implementation of GLUE 2.0 across the Data Area Representatives are – SEs: dCache, DPM, StoRM – Clients: dCache, gLite, ARC, UNICORE Timetable is as follows (from DJRA1.2.1) – Common agreement on the interpretation of the GLUE 2.0 schema [PM6] – Publishing GLUE 1.3 data with GLUE 2.0 schema [PM 12] – EMI data components fully GLUE 2.0 compatible [PM 24] Useful links – –

EMI INFSO-RI Working Group Decisions We are only considering current LDAP Information System We will validate published data using gstat 2.0 – Via “glue validator” provided by EMI infrastructure area It performs generic checks for schema compliance This can be extended to encapsulate the decisions of the Working Group, e.g., to enforce the presence of certain attributes that are optional in the LDAP schema

EMI INFSO-RI Working Group Decisions (examples) Optional attributes are optional – Exception list: *size attributes How do we ensure unique IDs? – Implementer should pick a hostname which is associated with the service and reasonably persistent, and append the string ‘/data’. They can then append further information as they see fit. Note that this is not intended to be parsed. – e.g., srm.example.org/data/opaque1 Which Type is my service? – servicetype_t is an open enumeration. Use a type of org.emi-eu., e.g., org.emi-eu.dCache. What do the *Size attributes mean? – Where possible, the “Usage of Glue Schema v1.3 for WLCG Installed Capacity Information” should be considered, being the current authoritative position. This may require further discussion.

EMI INFSO-RI Overview GLUE Details Working Group Implementation State Future Work Conclusions

EMI INFSO-RI Work Completed GLUE 2.0 info providers implemented for all three Storage Elements (dCache, DPM and StoRM) – Publishing the same info as for GLUE 1.3 – DPM was installed on the SA2 testbed at CERN – StoRM is under installation on the SA2 testbed at INFN-CNAF – dCache is under installation on the SA2 testbed at DESY FTS updated to publish to GLUE 2.0

EMI INFSO-RI DPM The dpm-listspaces utility (maintained at LAL) operates as the DPM info provider It now has a “--glue2” option to output all relevant info in the GLUE 2.0 schema Main issue was understanding how to incorporate GLUE 1.3’s GlueVOInfo into GLUE 2.0 – The answer was multiple Storage Shares with a common SharingID

EMI INFSO-RI dCache GLUE info-provider The info-provider has two parts: 1.A generic framework for converting xml into ldif called xylophone 2.A dCache specific configuration for xylophone Xylophone is platform independent – Can run on any platform with XSLT or Java. The configuration is a domain- specific language: – Changing what is published is easy.

EMI INFSO-RI StoRM GLUE dynamic info-provider The dynamic info-provider is written as a bash shell script and configured by using yaim.

EMI INFSO-RI Storage Space Concepts Total Size (TS) is the total amount of storage of a given Type (e.g., online, nearline, and so on)

EMI INFSO-RI Storage Space Concepts Used Size (US) is the amount of storage occupied by stored data.

EMI INFSO-RI Storage Space Concepts Reserved Size (RS) is the amount of storage reserved for use by a specific user.

EMI INFSO-RI Storage Space Concepts Free Size (FS)=TS-US-RS

EMI INFSO-RI Storage Space Concepts FS contains Available Size and Unavailable Size

EMI INFSO-RI All info-providers in EMI-1 Support for GLUE 2.0 comes with EMI-1. Info-providers can publish GLUE 1.3, 2.0 or both in parallel: – GLUE 1.3 and GLUE 2.0 are on separate bases; – The default is to publish both. GLUE 2.0 base has more or less the same information as GLUE 1.3.

EMI INFSO-RI Overview GLUE Details Data GLUE 2.0 Working Group Work Plan Implementation State Future Work Conclusions

EMI INFSO-RI Future work Update clients – Gfal, FTS Extend existing information to make a better use of GLUE 2.0 Fix problems discovered when validating production data services

EMI INFSO-RI Overview GLUE Details Data GLUE 2.0 Working Group Work Plan Implementation State Future Work Conclusions

EMI INFSO-RI Conclusions Data PTs have delivered info-providers for GLUE 2.0 on time They keep backward compatibility by publishing both GLUE 1.3 and 2.0 Updating clients to query GLUE 2.0 is the next step The EMI project is on track to deliver GLUE 2.0

EMI INFSO-RI Acknowledgements GLUE 2.0 Working Group emi.eu dCache Product Team DPM Product Team StoRM Product Team

EMI is partially funded by the European Commission under Grant Agreement RI Thank you!