MND review

Main directions of work  Development and support of the Experiment Dashboard Applications - Data management monitoring - Job processing monitoring - Monitoring of the distributed sites and services  Development and support of the high level view monitoring applications showing LHC distributed computing in action : - Dashboard Google Earth - SiteView  Development and support of the Dashboard framework. Dashboard framework is used not only by the Dashboard applications but by other projects as well. ES Group Meeting

Progress report. Experiment Dashboard applications  Usage of all Experiment Dashboard applications by the LHC experiments is steadily growing. Example: For CMS server # unique visitors from March to April increased more than 20% Volume of accessed data from March to April increased increased more than 30% Same trend for other Dashboard servers  Several application are shared by all LHC experiments : Dashboard SAM portal, Site Status Board ES Group Meeting

4  Prove to scale well though the load on the servers had considerably increased after start of data taking. In some cases when scalability issues were discovered, they were promptly addressed (see next slides)  New functionality requested by the experiments is enabled quickly (see next slides).  Started to work on migration of the Dashboard SAM portal to the new SAM structure Progress report. Experiment Dashboard applications

Progress report. Performance and scalability improvements  ATLAS DDM example During ATLAS reprocessing week in April, ATLAS was moving 3 times more files than during STEP09, while STEP was already 1.5 times higher than foreseen in ATLAS computing model (in terms of files). This sudden unforeseen increase caused delay in statistics generation in ATLAS DDM Dashboard. Statistics generation procedures were modified to use bulk operations and to run logically independent components in parallel. This allowed to solve the problem and cope with high file transfer rate.  CMS Job monitoring example Due to modification of the CRAB server configuration, update rate of the Dashboard job monitoring DB had increased 3-4 times compared to situation in 2009 and beginning of The Dashboard DB collectors were modified to run in multithreaded mode which allowed to handle much higher update rate without delay.  Site Status Board (SSB) SSB performance was considerably improved by running collectors in parallel and by optimizing DB access ES Group Meeting

Progress report. New functionality  On request of the CMS Analysis Support team new Analysis support UI and new API for accessing job failure information had been developed  Added new distributions for CPU and wall clock consumption and CPU efficiency to the historical job monitoring application  Enabled information flow for job monitoring data from GANGA to Dashboard via MSG. Next step is to instrument PANDA jobs in order to get complete picture of ATLAS job processing in Dashboard and to adapt for ATLAS all job monitoring applications developed for CMS  Started migration of the current Dashboard SAM portal to the new SAM structure ES Group Meeting

Experiment Dashboard applications. Future development  Continue to develop generic job monitoring which can be used by any VO. Currently prototyped for ATLAS. Use MSG as a message bus. This includes common library for data reporting, common job monitoring schema, data collectors and set of user interfaces.  Continue migration of the SAM Dashboard portal to the new SAM structure. We call new portal Site Usability portal.  Develop flexible statistics display for ATLAS DDM which will have several additional views and filters. Use existing backend, leverage new frontend using jQuery/JSON  Integrate SSB with Google calendar for site/service downtime  Enable reporting of ALICE data transfer information to MSG. This would allow to make this data available for GridView. ES Group Meeting

Progress report. High level view of the LHC computing activities on the WLCG infrastructure.  Dashboard Google Earth monitor proved to play an important role as a dissemination tool. ES Group Meeting TeV Media Event ATLAS CMS CCC and Openlab

Dashboard Google Earth monitor was demonstrated ES Group Meeting  Event organized by CERN for schoolchildren in Meyrin  WLCG booth at the EGEE User Forum  WLCG demo at The Third International Scalable Computing Challenge (SCALE 2010)  And of course CCC

Dashboard Google Earth clients (last month) ES Group Meeting  CERN  PIC  DESY  JINR  SARA  U. Toronto  U. Melbourne  Merck (the pharmaceutical company!)  Plus connections from ISPs from: New Zealand, Romania, Portugal, Norway, Finland, France,...

Dashboard Google Earth development  Two major releases were done during last two months Main improvements: -Changed the algorithm for computing trajectory of data links shown at the screen. -Enable caching inside the collector is order to increase the performance and decrease load on the information source. -Make collectors safe from the exceptions coming from the VO- specific information sources. -Fix logging of various components  Future development: Code consolidation - Faster - More robust - More generic in order to enable it for other VOs if required ES Group Meeting

Dashboard framework  Migrated from cvs to svn Future plans -Support for SLC5 and multiple versions of python -Create ‘Dashboard host management’ tool ES Group Meeting

Summary  Experiment Dashboard proved to become an important instrument used by the LHC experiments in offline distributed computing. Usage of the Dashboard applications is steadily growing. Current effort aims to provide reliable Dashboard service and to improve it’s performance and scalability.  Some of Dashboard applications are generic and can be used other user communities  Dashboard Google Earth became an important dissemination tool presented at various events to show LHC distributed computing in action ES Group Meeting