1. DataGrid is a project funded by the European Union CHEP 2003 24-28 March 2003 R-GMA 1 R-GMA: First results after deployment Steve Fisher (EDG - WP3) s.m.fisher@rl.ac.uk https://edms.cern.ch/document/376535/

2. CHEP 2003 24-28 March 2003 R-GMA 2 Who we are u Heriot-Watt, Edinburgh n Andrew Cooke, Werner Nutt u IBM-UK n James Magowan, (Manfred Oevers), Paul Taylor u INFN n Roberto Barbera, Giuseppe Save, Gennaro Tortone u Queen Mary, University of London n Roney Cordenonsi, (Ari Datta) u CCLRC n Linda Cornwall, Abdeslem Djaoui, Steve Fisher, Robin Middleton u PPARC n Rob Byrom, Laurence Field, Steve Hicks, Manish Soni, Antony Wilson, (Xiaomei Zhu), Jason Leake u SZTAKI, Hungary n Peter Kacsuk, Norbert Podhorszki u Trinity College Dublin n Brian Coghlan, Stuart Kenny, David O’Callaghan, (John Ryan)

3. CHEP 2003 24-28 March 2003 R-GMA 3 R-GMA u Uses the Grid Monitoring Architecture from Global Grid Forum u R-GMA is a relational implementation u Applied to both information and monitoring u Creates impression that you have one RDBMS per Virtual Organisation Producer Consumer Registry Information flow Meta-data flow

4. CHEP 2003 24-28 March 2003 R-GMA 4 Relational Approach u Not a general distributed RDBMS system, but a way to use the relational model in a distributed environment where global consistency is not important. u Producers announce:SQL “CREATE TABLE” publish:SQL “INSERT” u Consumers collect:SQL “SELECT” u Some producers, the Registry and Schema make use of RDBMS as appropriate – but what is central is the relational model.

5. CHEP 2003 24-28 March 2003 R-GMA 5 Producers u DataBaseProducer – Supports History Queries n Information not lost n Supports joins n Clean up strategy u StreamProducer – Supports Continuous Queries n In memory data structure n Can define minimum retention period u ResilientStreamProducer – Supports Continuous Queries n Like the StreamProducer but won’t lose data if system crashes n So slightly slower u LatestProducer – Supports Latest Queries n Just holds the latest information for any “primaryish” key n Supports joins u CanonicalProducer – Supports anything n Offers anything as relations

6. CHEP 2003 24-28 March 2003 R-GMA 6 Archiver (Re-publisher) u It is a combined Consumer-Producer u You just have to tell it what to collect and it does so on your behalf u Re-publishes to any kind of “Insertable” (i.e. not to the CanonicalProducer)

7. CHEP 2003 24-28 March 2003 R-GMA 7 Schema & Contributions CPULoad (Global Schema) CountrySiteFacilityLoadTimestamp UKRALCDF0.319055711022002 UKRALATLAS1.619055611022002 UKGLACDF0.419055811022002 UKGLAALICE0.519055611022002 CHCERNALICE0.919055611022002 CHCERNCDF0.619055511022002 CPULoad (Producer 3) CHCERNATLAS1.619055611022002 CHCERNCDF0.619055511022002 CPULoad (Producer 1) UKRALCDF0.319055711022002 UKRALATLAS1.619055611022002 CPULoad (Producer 2) UKGLACDF0.419055811022002 UKGLAALICE0.519055611022002

8. CHEP 2003 24-28 March 2003 R-GMA 8 The Mediator u Producers, associated with views on a virtual data base. u Queries posed against the virtual data base u The Mediator must: n find the right Producers n combine information from them u Can now merge information from several producers u The final mediator will take “any” SQL statement and do the right thing

9. CHEP 2003 24-28 March 2003 R-GMA 9 R-GMA Tools u R-GMA CLI n Command Line Interface (similar to MySQL) n Supports single query and interactive modes u R-GMA Browser n JSP application dynamically generating web pages n Supports pre-defined and user-defined queries u Pulse n R-GMA Java client-based GUI n Supports streaming and simple graphical displays

10. CHEP 2003 24-28 March 2003 R-GMA 10 A user application: CMS u BOSS for job tracking on local farm n It currently forks the executable and parses stdout to publish info directly to an SQL DB n They publish to one table per job type and one table which is common to all job types u They are now ready to publish via R-GMA instead n Providing a scaleable Grid solution

11. CHEP 2003 24-28 March 2003 R-GMA 11 GIN and GOUT (Gadget IN and Gadget OUT) R-GMA Consumers LDAP InfoProvider GIN LDAP Server LDAP InfoProvider CircularBuffer Producer GIN Consumer (CE) Consumer (SE) Consumer (SiteInfo) RDBMS DataBase Producer GOUT Consumer API Archiver CircularBuffer Producer R-GMA

12. CHEP 2003 24-28 March 2003 R-GMA 12 CE and SE Tables ComputingElement dn CEId TotalCPUs FreeCPUs TotalJobs RunningJobs …… CloseStorage Element dn CEId CloseSE …… StorageElement status dn SEId SEfreespace …… “Select a ComputingElement with at least 1 free CPU that also has a CloseStorageElement with at least 1000 MB of free space” SELECT DISTINCT ComputingElement.CEId FROM ComputingElement, CloseStorageElement,StorageElementStatus WHERE ComputingElement.FreeCPUs > 0 AND (ComputingElement.CEId = CloseStorageElement.CEId AND CloseStorageElement.CloseSE = StorageElementStatus.SEId AND StorageElementStatus.SEfreespace > 1000)

13. CHEP 2003 24-28 March 2003 R-GMA 13 u All Grid Services u OGSA Factories, GSH, GSR u Registry includes HandleMapper u SQL as Service Data Element Query Language Consumer Factory Producer Instance OGSIfied R-GMA Sensor Producer API Application Consumer API Schema Registry Consumer Instance Producer Factory

14. CHEP 2003 24-28 March 2003 R-GMA 14 Other technicalities – no time today u Soft-state Registration and the Registry n Registry records existence of Producers and Consumers n Registry holds last contact time and ‘expiry’ time n Producers and Consumers periodically refresh their time stamps n Scheduled removal of entries that have timed-out u Registry & schema distribution n Will have one logical registry and schema per VO n Each logical registry will have multiple physical “copies” n Self healing algorithm u Security u etc …

15. CHEP 2003 24-28 March 2003 R-GMA 15 Performance u By design: n Very flexible - to avoid bottlenecks n Powerful queries allow a single query to be made u Performance and Optimisation n Use NetLogger and profiling tools to identify possible bottlenecks

16. CHEP 2003 24-28 March 2003 R-GMA 16 Results u It has only just been deployed in the EDG development testbed and we do not yet have the results which the title of this talk implied.

17. CHEP 2003 24-28 March 2003 R-GMA 17 Summary and the future u R-GMA is a combined Grid information and monitoring system u Just deployed in the EDG development testbed u Focusing on reliability, stability and performance for the rest of the project (9 months) Thanks to the EU and our national funding agencies for their support of this work