1. Dave Kant D.Kant@rl.ac.uk LCG Monitoring and Accounting Dave Kant CCLRC e-Science Centre, UK HEPSYSMAN April 2005

2. 2 Introduction Overview of some of the monitoring tools in action in the LHC Computing Grid –GOCDB –GPPMON –GRIDICE –GSTAT –CERTIFICATION TESTING –REAL TIME GRID MONITOR Accounting Use Case Future Plans

3. 3 Monitoring the LCG Grid is a Challenge! Number of participating sites is growing every day: August 2003 => 12 sites ~100 CPUs October 2004 => 83 sites ; ~8000 CPUs April 2005 => 138 site; ~14000 CPUs 4TB Disk Grid Operations Centre Monitor the operational status of sites; Fault detection Problem Management Identify problems; escalate; track;

4. 4 With so many sites participating, there is a requirement for operational information in order to manage a grid environment What are the core grid services e.g. RBs/SEs/BDIIs the VOs are using for data challenges. Who do we contact when there is a security incident? Require a toolkit test specific core services. We have to concentrate on functional behaviour of services e.g. If an RB sends your job to a CE, then we must assume the RB is working fine. Is this the only test of a RB? Not all the tests that we perform are effective at finding problems. We must develop tests which simulate the life cycle of real applications in a Grid environment. …and lots more Monitoring Challenges

5. 5 GOC Configuration Database GOC GridSite MySQL Resource Centre Resources & Site Information EDG, LCG-1, LCG-2, … ce se bdii rb Monitoring Services Operations Maps Configure other Tools Organisation Structure - People/Institites/Projects Secure services - News - Self Certification - Accounting http://goc.grid-support.ac.uk/gridsite/gocdb Secure Database Management via HTTPS / X.509 Store a Subset of the Grid Information system People, Contact Information, Resources Scheduled Maintenance RC SQL https SERVERSERVER GOC DB can also contain information that is not present in the IS such as: Scheduled maintenance; News; Organisational Structures; Geographic coordinates for maps.

6. 6 Operations Map – Job Submission Tests GPPMON Displays the results of tests against sites. Test: Job Submission Job is a simple test of the grid middleware components e.g. Gatekeeper service, RB service, and the Information System via JDL requirements. This kind of test deals with the functional behaviour core grid services – do simple jobs run. They are lightweight tests which run hourly. However, they have certain limitations e.g. Dteam VO; WN reach (specialised monitoring queues).

7. 7 Operations Map – Certificate Lifetime GPPMON Displays the results of tests against sites. Test:Certificate Lifetime Many grid services require a valid certificate for security. By probing the host certificates on CEs and SEs at sites with a simple SSL client service, we can identify certificates which are due to expire and send an early warning to them. A predictive tool!

8. 8 GRIDICE – Architecture A different kind of monitoring tool – processes / low level metrics / grid metrics Developed by the INFN-GRID Team http://infnforge.cnaf.infn.it/gridicehttp://infnforge.cnaf.infn.it/gridice Data harvest via discovery service (postgreSQL) Measurement service monitoring sensor agents probe process table, memory, cpu Publication service

9. 9 GRIDICE – Global View Display shows the processes belonging to the Broker service. Problems are flagged List of Sites Resource Usage CPU#, Load, Storage, Job Info Different Views of the data: Site / VO / Geographic

10. 10 GridIce Job Monitoring Recently deployed version 1.6.3 on to LCG which features job monitoring: Queued, Running, Finished organised in different ways (site, Vo etc) XML views of data

11. 11 GRIDICE – Expert View Display shows the processes belonging to the Broker service. Problems are flagged Node Processes

12. 12 Ganglia Monitoring http://gridpp.ac.uk/ganglia Can use Ganglia to monitor a cluster Scalable distributed monitoring system for clusters and grids using RRD for storage and visualisation. RAL Tier-1 Centre LCG PBS Server displays Job status for each VO Get a lot for little effort

13. 13 Federating Cluster Information Can also use Ganglia to monitor clusters of clusters Ganglia/R-GMA integration through Ranglia.

14. 14 GIIS Monitor Developed by MinTsai (GOC Taipei) Tool to display and check information published by the site GIIS (sanity checks, fault detection) http://goc.grid.sinica.edu.tw/gstat/

15. 15 Regional Monitoring EGEE is made up of regions. Each region contains many computing centres. Regional Operational Centres is a focus for operations. USA Dealing with the complexities of managing a grid.

16. 16  http://goc.grid-support.ac.uk/roc_map/map.php http://goc.grid-support.ac.uk/roc_map/map.php  Provide ROCs with a package to monitor the resources in the region Tailored Monitoring GUIs to create organisations and populate them with sites  Hierarchical view of Resources Example UK Particle Physics GridPP Materialised Path encoding Regional Monitoring Maps EGEE (1) France (1.1) UK/I (1.2) GridPP (1.2.1) LondonT2 IMPERIAL QMUL ScotGrid Edinburgh S.E.E (1.3)

17. 17 Site Functional Tests (SFT) In terms of middleware, the installation and configuration of a site is quite a complicated procedure. –When there is a new release, sites don’t upgrade at the same time –Some upgrades don’t always go smoothly –Unexpected things happen (who turned of the power?) –Day-to-day problems; robustness of service under load? Its necessary to actively hunt for problems Site certification testing is by CERN deployment team on a daily basis. First step toward providing this service involves running a series of replica manager tests which register files onto the grid, move them around, delete them; and 3 rd party copies from remote SE. Unlike the simple job submission tests implemented in GPPMON, these tests are more heavy weight and attempt simulate the life cycle of real applications.

18. 18 Certification Test Results http://lcg-testzone-reports.web.cern.ch/lcg-testzone-reports/cgi-bin/listreports.cgi

19. 19 Aggregator RSSReader (Windows Client) GOC generates RSS feeds which clients can pull using an RSS aggregator. How can we integrate feeds and ticketing systems? Syndication of Monitoring Information

20. 20 Real Time Grid Monitor http://www.hep.ph.ic.ac.uk/e-science/projects/demo/index.html A Visualisation tool to track jobs currently running on the grid. Applet queries the logging and bookkeeping service to get information about grid jobs. Why are jobs failing? Why are jobs queued at sites while others are empty?

21. 21 Problems with existing tools Lots of monitoring tools have described – they have a few things in common: - all the information which they generate is hidden away or difficult to access - limited interfaces: the data can only be accessed in specific ways Therefore, its difficult to build “on-demand” services to allow communities “Players” to interact with the data. Examples include a)Job Accounting service : to allow an Organisation to compare resources usage for each VO b)Certification Testing service: Secure service to allow a site administrator to run the certification test suite against their site through a RB of their choice? The idea is for the services to collect information and put it into a common repository such as an RGMA Archiver. In this way, the information can be shared and accessible to all. Services (EGEE parlance: ROC and CIC services) munch the data and present it to the community. Example: GIIS is that its hard to drill down to the information you want e.g How much CPU in GridPP today? How much disk in the UKI ROC? The new paradigm solves this problem by allowing the data to be aggregated in different ways.

22. 22 Monitoring Paradigm A Better way to unify monitoring information. GOC Services collect information and publish into an archiver. ROC/CIC Services provide a means for the community to interact with this information on-demand. GOC provides services tailored to the requirements of the community. Information Repository (RGMA) Accounting Monitoring GSTAT Testing ROC Services Self Certification CIC Services Communities VOs ROCs EGEE Sites Organisations GOC Services

23. 23 GOC UseCase Job Accounting An accounting package for LCG has been developed by the GOC at RAL There are two main parts –the accounting data-gathering infrastructure based on R-GMA which brings the data to a central point –a web portal to allow on-demand reports for a variety of players.

24. 24 Requirements A historical record of grid usage to identify the use of individual sites by VOs as a function of time To demonstrate the total delivery of resources by that site to the Grid Aggregated views of the collected data by: –VO –Country – a requirement of LCG which has a country-based structure –EGEE Region – for use by EGEE Regional Operations Centre (ROC) A presentation front-end to the data to allow the selection on-demand of the views described above for different VOs and periods of time. To present the data as –A graphical view for interpretation –A tabular view for precision To support sites that already had their own methods of data collection by allowing arbitrary data collection techniques and insertion of the data in the standard schema into the central database.

25. 25 Requirements It was not an explicit requirement that user information be captured but we included this in the design as we were sure this would be a secondary requirement This is a reporting system, not a charging mechanism. The information is under the control of the site, so it does not meet the requirement of a charging system to be digitally signed and irrefutable. Information is gathered centrally, not under the control of the VO

26. 26 Design Information collected at each site from batch logs, gatekeeper logs etc Information joined at site level to select grid jobs and stored in database on R-GMA MON box at site. Information published through R-GMA and collected centrally in an R-GMA archive at GOC Web site presents various views of this data for presentation Structure of Grid taken from GOC DB – the grid configuration database. Only normalised cpu time collected

27. 27 APEL – Accounting Processor for Event Logs

28. 28 How APEL Works? PBS/LSF log processed daily on site CE to extract required data, filter acts as R-GMA DBProducer - > PbsRecords table Gatekeeper log processed daily on site CE to extract required data, filter acts as R-GMA DBProducer -> GkRecords table Message log processed daily on site CE to extract required data, filter acts as R-GMA DBProducer - > MessageRecords table Site GIIS interrogated daily on site CE to obtain SpecInt and SpecFloat values for CE, acts as DBProducer -> SpecRecords table, one dated record per day These three tables joined daily on MON to produce LcgRecords table. As each record is produced program acts as StreamProducer to send the entries to the LcgRecords table on the GOC site. Site now has table containing its own accounting data; GOC has aggregated table over whole of LCG. Interactive and regular reports produced by site or at GOC site as required.

29. 29 Job Records In via RGMA RGMA MON SQL QUERY TO Accounting Server 1 Query / Hour On-Demand Accounting Pages based on SQL queries to summary data 1 Record per Grid Job (Millions of records expected) Summary data refreshed every hour (Max records about 100K per year) Home Page User queries Graphs GOC

30. 30 Description Web allows information to be selected by –VO, time range, (Whole Grid, Country, EGEE Region, site) Also shows information on data collected

31. 31 Web form to apply selection criteria on the data Aggregate data across an organisation structure (Default= All ROCs) Select VOs (Default = All) Select date range

32. 32 VO Index Summed CPU (Seconds) consumed by resources in selected Region Selected Date Range

33. 33 List of Sites Belonging to the Selected ROC A breakdown of the resource usage per Site, per VO, per Month

34. 34 65 Sites publishing data to GOC (April 2005) Over 1.3 Million Job records ~ 50K records per week http://goc.grid-support.ac.uk/ /

35. 35 GOC Accounting Services http://goc.grid-support.ac.uk/gridsite/accounting/index.html BaseCpuSeconds Aggregated across EGEE Each Site, per VO, per Month Simple interface to customise views of data: VO, time frame and Region (default = EGEE) Each Region, per VO, per Month On Demand Services to EGEE Community Other Distributions Normalised CPU # Jobs

36. 36 Provide Interface to the Data Driven by User Requirements Materialised Path Library Tier-1 View Regional View Country View

37. 37 … Including Graphing Features

38. 38 Number of Sites per Country Publishing Accounting Records to GOC

39. 39 GridPP Accounting Status – April 2005 Sites that have never published or have not published recently. CAVENDISH-LCG2 -- never published Dublin-CSTCDIE -- never published DURHAM – last published 18 th Feb 2005 IC-LCG2 -- last published 9 th April 2005 RAL-LCG2 – last published 16 th March 2005 HP-Bristol -- never published Lancs-LCG2 – never published LivHep-LCG2 – never published QMUL-eScience – never published RHUL-LCG2 -- never published ScotGrid-Glas – last published 17 th Jan 2005 UCL-CCC – last published 12 th Feb 2005 UCL-HEP – never published Contact Dave if you need advise about installing Apel D.Kant@rl.ac.uk Tel: 01235 778178D.Kant@rl.ac.uk

40. 40 Batch System Support APEL supports PBS (Released) and LSF (Testing) Implementations are separate and independent of one another. Currently LCG2_4_0 has PBS support only. Re-factoring to a single package with plug-in batch specific components is currently in progress. What is the current status about LSF Support? LSF currently comes in three flavours (version 4, 5 and 6), each has a different usage record format New RPM edg-rgma-apel-lsf has been released to CERN for testing. Expect a release in the 2_4_1 tag next Month.

41. 41 Issues 1.Which RPM Version? Latest version on http://goc.grid-support.ac.uk/gridsite/accountinghttp://goc.grid-support.ac.uk/gridsite/accounting 3.4.44 for LCG2_4_0 Change Log 3.4.37 to 3.4.43 oApel 3.4.43 (April 6 th ) Startup script modified for RGMA 2_4_0 s/w release oApel 3.4.42 (Mar 20 th ) Improved core functionality oBetter handling of dn suppression oCheck flexible archiver on-line before attempting to send job records oApel 3.4.41 (Feb 2 nd ) Minor fix to SQL script oApel 3.4.40 (Jan 17 th )  Normalisation issue (see later)  CatchAll specInt/specFloat set to value in GIIS rather than 0 oApel 3.4.39 (Dec 16 th ) Current PBS log excluded from archive oApel 3.4.38 (Nov 19 th )  Bug in “reprocess” option during Join  Added “cleanAll” option oApel 3.4.37 (Oct 14 th )  grant mechanism to allow GK and CE to connect to MySQL database

42. 42 Issues 2.VO Filtering National Grid VOs activities running on same infrastructure as EGEE/LCG Privacy reasons why sites don’t want to publish National VO data to GOC APEL does not discriminate between the VOs Develop a solution? What can we suggest today? GOCDB can hide resources APEL made the requirement to exclude Local work not published but non LCG work does come through. Whats the model 1 CE per VO…what do people do? Don’t need to install Apel on non-LCG VO CEs SARA-LCG2, IISAS-Bratislava GridPP?

43. 43 Issues 3.Development of Tests to Check the Accounting Service Is site accounting working? Is the GOC listening for new data? Is the RGMA Registry working? GSTAT oGOC Flexible archiver service listens for accounting producers oIf the service is down, no data can be sent to the GOC! oUse the service every 5 minutes to update a timestamp in a test record in the accounting database. GSTAT can query table, look at the timestamp and compare with the current date/time. o3 rd party to use the flexi service. oUse RGMA to compare records in the site database and GOC Site Functional Tests oCan check the RPM version installed on the CE Testing the Whole Thing instead of the Pieces oInvestigate an Apel “heart-beat” oSite cron writes a test record every hour and publishes to GOC

44. 44 Issues 4.Which Log Files Should Site Administrators Backup? To build accounting records, we need to process data from THREE log file sources. This is a mandatory requirement in order to reconstruct what has been done during the 2004 period. /var/log/globus-gatekeeper* oMatch between grid-user dn to GramScriptJobId /var/spool/pbs/server_priv/accounting/* oLocal jobID and details of resources consumed oNo distinction between grid jobs and non-grid jobs. /var/log/messages* oMap GramScriptJobID to local JobID This is how we separate grid jobs from local user jobs which run on the local fabric. If the site has deleted its messages files, we may be able to work around this by matching local unix groups in the batch logs. Accounting records formed in this way will not contain the dn of the grid-user.

45. 45 Issues 5.siteName Changes –Recent problem with presenting data from the French ROC where CCIN2P3 was renamed to IN2P3-CC via GOCDB portal –All records associated with the site are updated in order for SQL queries to match the new siteName. 6.Namespace Convention? –Naming scheme to identify data belonging to large sites which provide services for different communities etc. –NIKHEF: lcgprod.nikhef.nl, lcg2prod.nikhef.nl, edgapptb.nikhef.nl –*SiteName* is a bad choice because we get multiple hits o*IC-LCG2* gives multiple matches PIC-LCG2 and IFIC-LCG2 –Request sites stick to the convention *.SiteName oh1.desy.de, zeus.desy.de

46. 46 Issues 7.Normalisation –We want to perform a reasonably sensible first order estimate to account for the differences in worker node performance. –Homogeneous vs Heterogeneous –PBS Job Records don’t have any information about the worker node benchmarks, so we must insert one manually –PBS Farms setup in different ways; can lead to an error in the normalisation calculation (Blindman vs internal normalisation) –Histories - What SpecInts do we use in order to process archived Job Records? –LSF Job Records have a CPU_FACTOR (1 - 4) in the Job Record. oWhat does a value of 1 correspond to? oDifferent “calibration” value at each site oConversion table? oCan the site publish a weighted specInt2000 for the farm?

47. 47 Issues 8.Service Reliability & Hardening –If flexible archiver is down, sites unable to publish data to GOC Update: 3.4.42/43 Apel core checks if flexible archiver service is available before attempting to publish data. GOC publishes a test record every 5 minutes to check the service is alive: automatic service recovery mechanism now in place –Investigate running multiple flexible archiver services 1 per GOC or 1 per ROC? At the moment, the archiver service listens for all producers rather than producers belonging to a ROC. –Single point of failure if registry is down? Multiple registry replicas supported in the RC1 (gLite) release? Update: Multiple registries supported in LCG2_4_0 ?

48. 48 Future Plans 1.Integration into gLite Framework? Work started 2.“Apel” for Storage Capturing billing information for dcache Cron runs, publish recent data into R-GMA SE snapshot e.g “df” of filesystem oUse of disk and tape oCron runs on SE which is a script; but script tailored for different SE e.g. dcache, tapestore etc 3.Web Services Interface to accounting data How would such a thing work? Any UseCases?

49. 49 Accounting Issues 1.A stable release of accounting package has been certified and tested at CERN; Should sites wait for the official release of press ahead independently? 2.Package supports PBS only; initial implementation for LSF. 80 sites advertising 313 Job managers: - 300 PBS (91% of sites) - 3 CONDOR (KFKI, FNAL, TRIUMF) -7 LSF (GSI, LNL, CERN). 3.Accounting requires the R-GMA infrastructure to be deployed at the site. 4.The VO associated with a user’s DN is not available in the batch or gatekeeper logs. It will be assumed that the group ID used to execute user jobs, which is available, is the same as the VO name. 5.The global jobID assigned by the Resource Broker is not available in the batch or gatekeeper logs. This global jobID cannot therefore appear in the accounting reports. The RB Events Database contains this, but that is not accessible nor is it designed to be easily processed. [Andrea Guarise: JRA1 proposal]

50. 50 Accounting Issues 6.Most sites keep GK/Batch logs but throw away message log files after 9 weeks due to default log rotation. 7.At present the logs provide no means of distinguishing sub-clusters of a CE which have nodes of differing processing power. Changes to the information logged by the batch system will be required before such heterogeneous sites can be accounted properly. At present it is believed all sites are homogeneous.

51. 51 Summary Accounting Information gathering infrastructure has been developed It has been through the C&T cycle and should be deployed in the next release. A web portal for display of this information has been developed (work in progress) This is an EGEE deliverable (DSA1.3) The display infrastructure can be deployed for other monitoring information. Development towards on-demand services to provide the community with up-to-date information, aggregated at different levels. Development of Visualisation tools to enhance our understanding of the grid.