2. CrossGrid WP3 Task 3.3 Grid Monitoring Trinity College Dublin (TCD, AC14 - CR11) Brian Coghlan, Stuart Kenny CYFRONET Academic Computer Centre, Krakow ( CYFRO, CO1) Xxx Yyy ICM, University of Warsaw (ICM, AC2 - C01) Xxx Yyy Poznan 28-JAN-2002

3. CrossGrid Task 3.3 Task 3.3 Presentation Will concentrate on: Goals Technology to be developed Technology required

4. Poznan 28-JAN-2002 CrossGrid Task 3.3 Task 3.3 Target Audience Application-level:Programs Tools Services

5. Poznan 28-JAN-2002 WP3 Portals (3.1) Roaming Access (3.1) Grid Resource Management (3.2) Grid Monitoring (3.3) Optimisation of Data Access (3.4) Tests and Integration (3.5) Applications WP1 End Users WP1, WP2, WP5Testbed WP4 Performance evaluation tools (2.4) WP3 CrossGrid Technical Annex Fig. WP3-1

6. Poznan 28-JAN-2002 CrossGrid Task 3.3 WP2 Application source code Grid Monitoring (3.3) Performance measurement MPI verification (2.2) Benchmarks (2.3) Automatic analysis Analytical model Visualization Applications (WP1) executing on Grid testbed (WP4 ) Performance analysis (2.4) CrossGrid Technical Annex Fig. WP2-1

7. Poznan 28-JAN-2002 CrossGrid Task 3.3 Task 3.3 We MUST get it RIGHT Very central function

8. Poznan 28-JAN-2002 CrossGrid Task 3.3 Task 3.3 Milestones PM 1-3 Definition of requirements PM3 : Deliverable D3.1 [ALL] PM 4-6 Designing of architecture, interfaces and security issue PM6 : Deliverable D3.2 (report) [ALL] PM 6 First testbed set-up on selected sites PM6 : Deliverable D3.2 [ALL] PM 6-12 Implementation of 1 st prototype (running on local grid) PM12 : Deliverables D3.3 (prototype and report) PM 13-24 Second development phase PM13 (beginning) full testbed prototype PM 13-15 Testing, feedback collecting PM13 : Extraction of raw performance data locally for WP2.4 PM 16-18 Refinement, implementation of WP3.3 PM18 : Deliverable D3.5 Extraction of raw performance data on Grid for WP2.4 PM 18-24 Refinement, implementation of WP3.3. ??? PM 16-24 Refinement, implementation of other tasks within WP3 PM24 : Deliverable D3.6 (prototype and report) PM 25-33 3 rd development phase PM33 (end) : Milestone M3.5 Final version of the programming environment … and so on

9. Poznan 28-JAN-2002 CrossGrid Task 3.3 Task 3.3 Philosophy ‘A good Engineer is a person who makes a design that works with as few original ideas as possible’ Freeman Dyson [via Yannis Perros] So, where possible, reuse ideas from: Globus GGF DataGrid Elsewhere

10. Poznan 28-JAN-2002 CrossGrid Task 3.3 Task 3.3 Technology to be Developed OMIS Interface RGMA Local Monitor Performance Data Storage RGMA Producer Service Manager RGMA Consumer Performance Tools Application Application Monitor Non-invasive Monitoring Invasive Monitoring Non-invasive Monitoring fromTechnicalAnnex

11. Poznan 28-JAN-2002 CrossGrid Task 3.3 Invasive Monitoring Copyright S.M.Fisher / RAL

12. Poznan 28-JAN-2002 CrossGrid Task 3.3 Non-invasive Monitoring

13. Poznan 28-JAN-2002 CrossGrid Task 3.3 Task 3.3 Technology Required: Choices Homebrew Globus Sockets – current technology Globus: MDS - current technology GGF: GMA - guidelines only DataGrid: R-GMA - evolving technology Elsewhere: Jiro - new technology

14. Poznan 28-JAN-2002 CrossGrid Task 3.3 Homebrew: Globus Sockets Supported by Globus toolset Includes GSI security http://www.globus.org/ Quick solution Does not give access to grid information system

15. Poznan 28-JAN-2002 CrossGrid Task 3.3 Globus: Meta-Directory Service (MDS) Hierarchical Schema Queries constrained to schema Can have parallel schemas Based on Lightweight Directory Access Protocol (LDAP) Good for static data Less good for dynamic data Now in 2 nd generation (MDS2 released OCT-2001) Now uses LDAP v3

16. Poznan 28-JAN-2002 CrossGrid Task 3.3 MDS: DataGrid DataGrid WP3 has delivered: Coordination of GRIS/GIIS work OpenLDAP caching backend (FTREE) GRIS/GIIS infrastructure maintain as necessary Little effort anticipated FTREE maintain as necessary Little effort anticipated Copyright S.M.Fisher / RAL

17. Poznan 28-JAN-2002 CrossGrid Task 3.3 MDS: LDAP LDAP reflects hierarchical database technology LDAP is efficient if you know the query in advance: you can build your database to respond very rapidly For other queries, it could be very inefficient: LDAP query language cannot give results based on computation on two different objects in the structure Relational database technology proposed by Codd (30yrs ago) as solution to inflexibility of hierarchical & network data bases

18. Poznan 28-JAN-2002 CrossGrid Task 3.3 GGF: Grid Monitoring Architecture (GMA) Open schema Distributed architecture Allows for dynamic data GGF is a Forum GMA is a set of guidelines GGF will not ‘approve’ anything Several prototype implementations

19. Poznan 28-JAN-2002 CrossGrid Task 3.3 GMA: Architecture Consumers can themselves be Producers Consumer Producer Registry register register lookup

20. Poznan 28-JAN-2002 CrossGrid Task 3.3 GMA: Data Model / Protocols GGF do NOT specify any data model GGF do NOT specify protocols between consumers & producers An implementer MUST: First choose a suitable data model Then select suitable protocols The chosen data model MUST have the power to support ALL required queries

21. Poznan 28-JAN-2002 CrossGrid Task 3.3 DataGrid: Relational GMA (R-GMA) Trying to do “better” than MDS An implementation of GMA Relational schema Queries less constrained than MDS Designed to handle dynamic data 1 st prototype released in NOV-2001 Based on servlet technology HTTP queries / XML result-sets Can be set up to access MDS data

22. Poznan 28-JAN-2002 CrossGrid Task 3.3 R-GMA: Data Model DataGrid have chosen a RELATIONAL data model Not general distributed RDBMS system, but a way to use relational model in a distributed environment where ACID (Atomicity, Consistency, Isolation and Durability) properties not important Producersannounce:SQL “CREATE TABLE” publish:SQL “INSERT” Consumerscollect:SQL “SELECT” Viewed as one huge logical data base, partitioned according to certain criteria (specified by WHERE clause as a predicate)

23. Poznan 28-JAN-2002 CrossGrid Task 3.3 R-GMA: Data Quality Difficult to know when to get rid of archived data Information may no longer be ‘up to date’ BUT For historical analysis this is not a criteria for pruning Only the consumer of data can devise a suitable strategy Only they know why they are collecting the data Producer of data is no judge of its continued worth Time-to-Live (TTL) of questionable value

24. Poznan 28-JAN-2002 CrossGrid Task 3.3 R-GMA: Philosophy Information vs. Monitoring From the user’s point of view there there is little or no difference between “plain” information and monitoring information. Arguments about rates of change [slowly=Information whereas rapidly=Monitoring] are unconvincing. Any measurement or fact represented as a tuple. Add a time stamp to the tuple and the information can be stored for later analysis as monitoring information. At most the difference is 1 field - the time stamp. Time is the common element Datagrid use R-GMA not only for monitoring but also as the basis of an information system Copyright S.M.Fisher / RAL

25. Poznan 28-JAN-2002 CrossGrid Task 3.3 R-GMA: Implementation Consumer Servlet Registry API Consumer Servlet Registry API Consumer Servlet Registry API Consumer Servlet and Mediator Registry API Sensor Code Producer API Application Code Consumer API ProducerServlet Registry API Registry Servlet Schema API Schema Servlet Event Dictionary DBProducer DBProducerServlet Application Code Archiver API Archiver Servlet Consumer API Consumer API Consumer API Consumer API Copyright S.M.Fisher / RALlookup subscribe

26. Poznan 28-JAN-2002 CrossGrid Task 3.3 R-GMA: Progress API in both JAVA and C++ Basic sensors (i.e. cpuload) for producing information Integration with MDS: R-GMA sensor queries MDS GRIS and republishes information Some basic displays to consume data Demo code to get people started Released into Testbed1 of EU-DataGrid Copyright S.M.Fisher / RAL

27. Poznan 28-JAN-2002 CrossGrid Task 3.3 Jiro TM Technology New technology http://www.jiro.com/ For distributed resource management Part of ‘Federated Management Architecture’ ICM will discuss

28. Poznan 28-JAN-2002 CrossGrid Task 3.3 Task 3.3: Subtasks Task 3.3 (CYFRO / TCD / ICM) Requirements Specification Task 3.3.1 (CYFRO - 38 MM) Invasive Monitoring: ‘Autonomous monitoring system for on-line and automatic performance analysis ‘ performance analysis ‘ Task 3.3.2 (TCD – 32 MM) Non-invasive Monitoring: ‘SQL-query-based tool support and interfaces to Grid application programming environment’ application programming environment’ Task 3.3.3 (ICM - 28 MM) Jiro: ‘Grid Jiro-based services for Grid infrastructure monitoring’

29. Poznan 28-JAN-2002 CrossGrid Task 3.3 Task 3.3: To Be Discussed Source Code Portability C / Java / Perl / Python ? API definition For Task 3.3.1 Open Source Licensing GPL ?

30. Poznan 28-JAN-2002 CrossGrid Task 3.3 WP3 Task 3.3 THE END