1. EGEE-III INFSO-RI-222667 Enabling Grids for E-sciencE www.eu-egee.org EGEE and gLite are registered trademarks Wojciech Lapka SAM Team CERN EGEE’09 Conference, 21 - 25 September 2009, Barcelona Regional Grid Monitoring Introduction & database components

2. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Outline Introduction to the new Service Availability Monitoring System Description of the Database Components –Aggregated Topology Provider (ATP) –Metric Description Database (MDDB) –Metric Results Store (Metric Store) 2

3. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Outline Introduction to the new Service Availability Monitoring System Description of the Database Components –Aggregated Topology Provider (ATP) –Metric Description Database (MDDB) –Metric Results Store (Metric Store) 3

4. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 SAM – existing architecture 4

5. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 SAM - enhanced architecture 5

6. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Data Flow 6

7. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Data Flow 7

8. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Data Flow 8

9. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Data Flow 9

10. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Data Flow 10

11. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Data Flow 11

12. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Data Flow 12

13. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Data Flow 13

14. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Data Flow 14

15. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 MyEGEE portal & iGoogle 15

16. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Outline Introduction to the new Service Availability Monitoring System Description of the Database Components –Aggregated Topology Provider (ATP) –Metric Description Database (MDDB) –Metric Results Store (Metric Store) 16

17. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Databases - ATP 17 What will be tested? ? ? How it will be tested? What to do with test results? ?

18. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Databases - ATP 18 What will be tested? ? ? How it will be tested? What to do with test results? Aggregated Topology Provider

19. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Databases - ATP What information is provided by the ATP? –Topology information containing:  Projects (WLCG) and grid infrastructures (EGEE, OSG, NDGF)  Sites, Services, VOs and their groupings  Downtimes  A history of the above Why do we need it? –For availability re-calculations, history of grid topology is needed –We couldn’t name groups of arbitrary grid resources (e.g. ATLAS clouds) –Single authoritative information source with topology information 19

20. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 ATP - why do we need it? 20 Current flow of Grid topology data across various monitoring tools:

21. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 ATP - why do we need it? 21 Streamlined grid topology data flow using the ATP:

22. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 ATP – data sources 22 BDII OSG IM GOCDB CIC Portal ATP sync OSG topology & downtimes EGEE topology & downtimes Installed capacity VO cards Aggregated Topology Provider Gstat 2.0 VO / service mappings Alice Voboxes WLCG MOU Portal Project feeds VO feeds

23. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 ATP – status What do we have today? –MySQL and Oracle version –Synchronizer –A programmatic interface to retrieve ATP information (XML/JSON): 23

24. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 ATP – status What needs to be added? –History tables to record changes in topology information –Programmatic Interface - parameterised queries (similar to SAM PI) 24

25. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Databases 25 What will be tested? ? ? How it will be tested? What to do with test results? Aggregated Topology Provider

26. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Databases - MDDB 26 What will be tested? ? How it will be tested? What to do with test results? Aggregated Topology Provider Metric Description Database

27. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Databases - MDDB What information is provided the MDDB? –Metrics which are used to test Grid infrastructure –Profiles – combination of metrics for computation of different availabilities and configuration of Nagios installations Why do we need it? –More flexible availability calculations:  Example: CMS would like to test Tier-1 and Tier-2 sites differently –Maintain a history of which metrics and calculations were valid at each point in time 27

28. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 MDDB - Architecture 28 CENTRAL MDDB Local Cache MDDB Sync

29. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 MDDB - Status What do we have today? –MySQL and Oracle version –Integration with ATP –Web User Interface –A programmatic interface to retrieve MDDB information (JSON) What needs to be added? –Synchronizer between Central DB and local (ROC) caches –Interface for populating and querying profiles –Profiles: Mapping with grid resources 29

30. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Databases 30 What will be tested? ? How it will be tested? What to do with test results? Aggregated Topology Provider Metric Description Database

31. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Databases – Metric Store 31 What will be tested? How it will be tested? What to do with test results? Aggregated Topology Provider Metric Description Database Metric Results Store

32. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Databases – Metric Store What information is provided by the Metric Store? –Metric results for service end-points for the grid infrastructure –Status changes for service end-points in the infrastructure What do we have today? –MySQL and Oracle versions:  Integration with MDDB and ATP  Per-service status change calculation for Profiles  Data loader –Data from 11 ROCs is being loaded to Central Metric Store:  Some of the records rejected (Mainly due to service end-points not defined correctly in GOCDB) 32

33. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Metric Store – status What needs to be added: –MySQL – tuning of DB (e.g. table partitioning) –Programmatic Interface - parameterised queries –Purging mechanism –Alerting mechanism integrated with Nagios (e.g. when not enough metric results received in given period of time) 33

34. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Central Metric Store Population 34 Active & Passive Checks Results Metric & Profile Definition Service Definition

35. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Outline Introduction to the new Service Availability Monitoring System Description of the Database Components –Aggregated Topology Provider (ATP) –Metric Description Database (MDDB) –Metric Results Store (Metric Store) Publicity 35

36. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Publicity - Demo Watch our demo and vote for it: –Tuesday 16:30-17:00 –Wednesday lunch –http://tinyurl.com/EgeeSAM (YouTube)http://tinyurl.com/EgeeSAM –http://www.youtube.com/watch?v=PADq2x8q0kwhttp://www.youtube.com/watch?v=PADq2x8q0kw 36

37. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Acknowledgments Thanks to the following people for their contributions: –James Casey (CERN) –Emir Imamagic (SRCE) –Pradyumna Joshi (BARC) –Rajesh Kalmady (BARC) –Vaibhav Kumar (BARC) –Steve Traylen (CERN) SAM Team at CERN: –John Shade –David Collados –Karolis Eigelis –Judit Novak –Konstantin Skaburskas 37

38. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Summary New enhanced SAM system, based on Nagios - a very popular powerful open-source tool, will: –Simplify transition to the EGI era –Help site administrators with fabric monitoring ATP, acting as a single authoritative information aggregator, will simplify the job of assimilating grid resource information MDDB will allow flexible availability calculations Metric Results Store will help MyEGEE portal in displaying of the test results. Demo: http://tinyurl.com/EgeeSAMhttp://tinyurl.com/EgeeSAM 38

39. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Thank you! 39 Questions? egee3-operations-automation- discuss @cern.ch