1. www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 GLUE 2: Deployment and Validation Stephen Burke egi.eu EGI OMB March 26 th 2013

2. www.egi.eu EGI-InSPIRE RI-261323 Overview Current state of GLUE 2 deployment GLUE 2 support by services –Current and expected GLUE 2 profile document Validation –glue-validator –Nagios Next steps GLUE 2 Deployment - EGI OMB2

3. www.egi.eu EGI-InSPIRE RI-261323 Deployment status All EGI sites now publish GLUE 2 –Helped by the gLite 3 retirement campaign –Remaining configuration problems all fixed –Should pay attention to new sites – part of certification? –OSG currently has no plans for GLUE 2 Currently have 325 sites, 4306 endpoints, 51 endpoint types GLUE 2 Deployment - EGI OMB3

4. www.egi.eu EGI-InSPIRE RI-261323 Service publication All gLite services (+dCache) publish from EMI 2 –Deployment will advance as EMI 1 is retired –Argus only publishes in GLUE 2 ARC integration is part of the EMI 3 top BDII –Also has performance improvements, e.g. cache size and indexing Unicore publishing should come with EMI 3 –Not documented? IGE (GridFTP, gsissh, gram) publishes from the 3.1 release GLUE 2 Deployment - EGI OMB4

5. www.egi.eu EGI-InSPIRE RI-261323 EMIR New service from EMI – service registry –http://eu-emi.github.com/emiregistry/documentation/registry- 1.2.1/emir-manual.htmlhttp://eu-emi.github.com/emiregistry/documentation/registry- 1.2.1/emir-manual.html In EMI 3 all services can be configured to register with it Has been under test for a while – feedback is positive –http://www.isgtw.org/announcement/european-middleware- initiative-registry-solution-federated-online-serviceshttp://www.isgtw.org/announcement/european-middleware- initiative-registry-solution-federated-online-services Need to consider how to deploy and use it in EGI GLUE 2 Deployment - EGI OMB5

6. www.egi.eu EGI-InSPIRE RI-261323 Client support WMS now has support for matchmaking with GLUE 2 attributes –Data matching not yet supported, but less important –https://wiki.italiangrid.it/twiki/bin/view/WMS/WMS34ExtendedR eleaseNoteshttps://wiki.italiangrid.it/twiki/bin/view/WMS/WMS34ExtendedR eleaseNotes GFAL 2 has support for data management tools –No documentation? ginfo tool in development as a replacement for lcg-info(sites) –http://cern.ch/lfield/ginfo-1.0.0-1.noarch.rpmhttp://cern.ch/lfield/ginfo-1.0.0-1.noarch.rpm –Will release to EPEL after feedback from WLCG GLUE 2 Deployment - EGI OMB6

7. www.egi.eu EGI-InSPIRE RI-261323 EGI Profile The draft profile document was presented at the August OMB Defines detailed semantics, extends the schema definitions Classifies attributes by use case and importance –Service Discovery, Service Selection, Monitoring, Oversight, Diagnostic –Mandatory, Recommended, Desirable, Optional, Undesirable –Various implications: need for accuracy, update rate, latency, caching... Detailed validation criteria –FATAL, ERROR, WARNING, INFO GLUE 2 Deployment - EGI OMB7

8. www.egi.eu EGI-InSPIRE RI-261323 Profile status Detailed comments received from four people, plus more minor comments from several others –Comments mainly asking for better/more detailed explanations Detailed comments and replies uploaded to DocDB page –https://documents.egi.eu/document/1324https://documents.egi.eu/document/1324 Updated document with changes tracked: version 0.9 –Fairly small changes in substance Will do one final review myself, update and create version 1.0 –Hopefully within a couple of weeks GLUE 2 Deployment - EGI OMB8

9. www.egi.eu EGI-InSPIRE RI-261323 glue-validator Existing python tool which just checked the basic schema structure Enhanced by Maria Alandes Pradillo –BDII developer and WLCG Information Officer Now implements essentially all validation checks in the profile document –Currently 129 separate tests –Easy to add more Standalone tool, but can output in Nagios format –Flexible configuration GLUE 2 Deployment - EGI OMB9

10. www.egi.eu EGI-InSPIRE RI-261323 Middleware validation Started by looking for middleware bugs –Initially using the EMI 3 certification testbed Bugs already submitted for several gLite services –Many services were OK! Continue with ARC and Unicore Also need to look at IGE Record progress on a wiki page –https://twiki.cern.ch/twiki/bin/view/EGEE/ISproviders#GLUE_2_ 0_Validationhttps://twiki.cern.ch/twiki/bin/view/EGEE/ISproviders#GLUE_2_ 0_Validation GLUE 2 Deployment - EGI OMB10

11. www.egi.eu EGI-InSPIRE RI-261323 Production BDII status Querying the lcg-bdii Top level BDII –273 ERROR messages All related to wrong totals published for the GLUE2StorageShareCapacity and GLUE2StorageServiceCapacity objects –30541 WARNING messages Mainly due to wrong types and mandatory attributes not present –31425 INFO messages Mainly recommended attributes not present and default values (i.e. MaxCPUTime, MaxWallClockTime) Note that information provider validation is ongoing: –Some errors won’t be considered if they come from bugs in the information providers GDB, 13th March 2013

12. www.egi.eu EGI-InSPIRE RI-261323 Nagios A previous version of glue-validator is already running on midmon.egi.eu as a test –Some initial problems fixed –Central deployment makes it easier to update Known middleware bugs will be masked in Nagios –Sites should only get bugs for things they can fix Need to move the latest version to midmon and check that the output looks sane Can then consider raising alarms – in a few weeks? GLUE 2 Deployment - EGI OMB12

13. www.egi.eu EGI-InSPIRE RI-261323 Other uses glue-validator should also be used in middleware testing and certification Could be run inside the BDII to catch problems before publication –Needs further experience A new version of gstat is in development to support GLUE 2 – useful to display the validation status for the whole Grid –Some tests intrinsically global, e.g. IDs and references GLUE 2 Deployment - EGI OMB13

14. www.egi.eu EGI-InSPIRE RI-261323 Next steps - deployment Continue to deploy EMI 2 services Consider pushing deployment of EMI 3 top BDII –Should be an easy upgrade –ARC integration, several performance improvements Test EMIR and consider deployment strategy Test WMS, GFAL and ginfo Test gstat 3 when available NB: GOC DB is also moving to use the GLUE 2 info model –XML rendering but similar to LDAP GLUE 2 Deployment - EGI OMB14

15. www.egi.eu EGI-InSPIRE RI-261323 Next steps - validation Review profile document and release “final” version –May well need updates after experience –Document is versioned –Compliance with a given version should be published Release new version of glue-validator and integrate in Nagios Submit bugs to developers and sites as necessary Document known bugs/features GLUE 2 Deployment - EGI OMB15

16. www.egi.eu EGI-InSPIRE RI-261323 Summary All sites now publish GLUE 2 information All services can publish First versions of clients ready or on the way Profile document defines detailed validation rules, glue-validator tests them Submit bugs to developers and sites –Test and improve –Should soon be in a better state than we ever had with GLUE 1 This is the year of GLUE 2! GLUE 2 Deployment - EGI OMB16