1. Enabling Grids for E-sciencE www.eu-egee.org Analysis of the ATLAS Rome Production Experience on the LCG Computing Grid Simone Campana, CERN/INFN EGEE User Forum, CERN (Switzerland) March 1 st – 3 rd 2006

2. Enabling Grids for E-sciencE Simone.Campana@cern.ch EGEE User Forum, CERN (Switzerland) – March 1 st – 3 rd,2006 2 Outline  The ATLAS Experiment The Computing Model and the Data Challenges  The LCG Computing Grid Overview and architecture  The ATLAS Production System  The Rome Production on LCG Report and numbers of the production Achievements with respect to DC2 Standing Issues and possible improvements  Conclusions

3. Enabling Grids for E-sciencE Simone.Campana@cern.ch EGEE User Forum, CERN (Switzerland) – March 1 st – 3 rd,2006 3 The ATLAS Experiment View of LHC @ CERN ATLAS ATLAS A ToroidaL ApparatuS for LHC

4. Enabling Grids for E-sciencE Simone.Campana@cern.ch EGEE User Forum, CERN (Switzerland) – March 1 st – 3 rd,2006 4 The ATLAS Computing Model NOTSINGLE  The ATLAS computing can NOT rely on a SINGLE computer center model Amount of required resources is too large For the 1 st year of data taking  50.6 MSI2k of CPU  16.9 TB of space on Tape  25.4 TB of space on Disk GRID paradigm  ATLAS decided to embrace the GRID paradigm High level of decentralization multi-tier  Sites are organized in a multi-tier structure Hierarchical model ROLE Tiers are defined by ROLE in the ATLAS computing  Tier-0 at CERN Record RAW data Distribute second copy to Tier-1s Calibrate anddo first-pass reconstruction  Tier-1 centers Manage permanent storage – RAW, simulated, processed Capacity for reprocessing, bulk analysis  Tier-2 centers Monte Carlo event simulation End-user analysis Virtual Organization  In Grid terminology ATLAS is a Virtual Organization Tier-1 Tier-0 Online filter farm RAW ESD AOD Reconstruction farm RAW ESD AOD MC Analysis farm Re-reconstruction farm Tier-2 Analysis farm Monte Carlo farm SelectedESD, AOD RAW ESD AOD ESD, AOD RAW MC RAW ESD AOD ESD, AOD

5. Enabling Grids for E-sciencE Simone.Campana@cern.ch EGEE User Forum, CERN (Switzerland) – March 1 st – 3 rd,2006 5 Data Challenges  Data Challenge  Data Challenge: validation of the Computing and Data Model and test the complete software suite Full simulation and reprocessing of data as if coming from the detector Same software and computing infrastructure to be employed for data taking  ATLAS ran two major Data Challenges DC1 in 2002-2003 (with direct access to local resources + NorduGrid, see later) DC2 in July - December 2004 (completely in GRID environment)  Large scale production  Large scale production in January – June 2005 Friendly called “Rome Production” provide data for physics studies for the ATLAS Rome Workshop in June 2005. Can be considered totally equivalent to a real Data Challenge Can be considered totally equivalent to a real Data Challenge  Same methodology  Large number of events produced Offered a unique opportunity to test improvements in the production framework, the Grid middleware and the reconstruction software. LCGNorduGridOSG  ATLAS resources span three different grids: the LCG, NorduGrid and OSG  In this talk I will present the “Rome production” experience on the LHC Computing Grid infrastructure

6. Enabling Grids for E-sciencE Simone.Campana@cern.ch EGEE User Forum, CERN (Switzerland) – March 1 st – 3 rd,2006 6 The LCG Infrastructure May 2005 140 Grid sites 34 countries 12000 CPUs 8 PetaBytes

7. Enabling Grids for E-sciencE Simone.Campana@cern.ch EGEE User Forum, CERN (Switzerland) – March 1 st – 3 rd,2006 7 LCG architecture Logging and Bookkeeping  The Logging and Bookkeeping service keeps the state information of a job allows the user to query its status Computing Element  Each Computing Element is the front-end to a local batch system manages a pool of Worker Nodes where the job is eventually executed.  Limited User Credentials  Limited User Credentials (Proxies) can be automatically renewed through a Proxy Service. Resource Broker  A set of services running on the Resource Broker machine match job requirements to the available resources schedule the job for execution to an appropriate Computing Element track the job status allow to retrieve their job output. The Workload Management System responsible for the management and monitoring of jobs

8. Enabling Grids for E-sciencE Simone.Campana@cern.ch EGEE User Forum, CERN (Switzerland) – March 1 st – 3 rd,2006 8 LCG Architecture  Allows the user to move files in and out of the Grid, replicate files among different Storage Elements and locate files. Storage Elements  Files are stored in Storage Elements Disk only or with tape backend.  A number protocols allows data transfer GridFTP is the most commonly used central catalogue  Files are registered in a central catalogue Replica Location Service keeps information about file location and about some file metadata. The Data Management System  Provides information about the Grid resources and their status.  Info generated on every service And published by the GRIS  Propagated in a hierarchical structure GIIS at every site BDII as central collector. The Information System

9. Enabling Grids for E-sciencE Simone.Campana@cern.ch EGEE User Forum, CERN (Switzerland) – March 1 st – 3 rd,2006 9 LCG architecture  Accounting logs resource usage and traces user jobs  Monitoring Services visualize and record the status of LCG resources Different systems in place  R-GMA  GridICE ...

10. Enabling Grids for E-sciencE Simone.Campana@cern.ch EGEE User Forum, CERN (Switzerland) – March 1 st – 3 rd,2006 10 The ATLAS production system executors  The executors offer an interface to the underlying Grid middleware. The LCG executor, Lexor, provides an interface to the native LCG WMS. clients tools  File upload/download rely on Grid-specific clients tools Data Management System  The ATLAS Data Management System (DonQuijote) ensures high-level data management across different Grids. job monitoring  The job monitoring is performed through Grid-specific tools. In LCG, information collected from the production database and the GridICE server are merged and published through an interactive web interface. central database  An ATLAS central database holds Grid- neutral information about jobs. “supervisor”  A “supervisor” agent distributes jobs to Grid-specific agents called “executors” follows up their status, validates them in case of success or flags them for resubmission.

11. Enabling Grids for E-sciencE Simone.Campana@cern.ch EGEE User Forum, CERN (Switzerland) – March 1 st – 3 rd,2006 11 Task Flow for ATLAS production Hits MCTruth Digits (RDO) MCTruth Bytestream Raw Digits ESD Geant4 Reconstruction Pile-up Bytestream Raw Digits Bytestream Raw Digits Hits MCTruth Digits (RDO) MCTruth Physics events Events HepMC Events HepMC Hits MCTruth Digits (RDO) MCTruth Geant4 Digitization Digits (RDO) MCTruth Bytestream Raw Digits Bytestream Raw Digits Bytestream Raw Digits Events HepMC Hits MCTruth Geant4 Pile-up Digitization Mixing Reconstruction ESD Pythia Event generation Detector Simulation Digitization (Pile-up) Reconstruction Event Mixing Byte stream Events HepMC Min. bias Events Piled-up events Mixed events With Pile-up ~5 TB 20 TB 30 TB 20 TB 5 TB TB Volume of data for 10 7 events

12. Enabling Grids for E-sciencE Simone.Campana@cern.ch EGEE User Forum, CERN (Switzerland) – March 1 st – 3 rd,2006 12 Hits MCTruth Digits (RDO) MCTruth ESD AOD Geant4 Reconstruction Pile-up Hits MCTruth Digits (RDO) MCTruth Events HepMC Events HepMC Geant4 Digitization Pythia Event generation Detector Simulation Digitization (Pile-up) part of the full chain  Only part of the full chain has been used/tested for Rome Production No ByteStream No Event Mixing  The reconstruction has been performed on digitized events and only partially on piled-up events But in fact …

13. Enabling Grids for E-sciencE Simone.Campana@cern.ch EGEE User Forum, CERN (Switzerland) – March 1 st – 3 rd,2006 13 Rome Production experience on LCG 8 concurrent instances of Lexor  In average 8 concurrent instances of Lexor were active on the native LCG-2 system.  Four people were controlling the production process checking for job failures interacting with the middleware developers and the LCG Experiment Integration Support team.  The production for the Rome workshop consisted of: 380k jobs A total of 380k jobs submitted to the native LCG-2 WMS  109k simulation jobs  106k digitization jobs  125k reconstruction jobs  40k pile-up jobs 1.4M files An total of 1.4M files stored in LCG Storage Elements  corresponding to an amount of data of about 45TB clear improvement  This is a clear improvement with respect to DC2 where 91.5k jobs in total ran on LCG-2 and no reconstruction was performed.

14. Enabling Grids for E-sciencE Simone.Campana@cern.ch EGEE User Forum, CERN (Switzerland) – March 1 st – 3 rd,2006 14 Rome Production experience on LCG Number of jobs per day Data Challenge 2 Rome Production

15. Enabling Grids for E-sciencE Simone.Campana@cern.ch EGEE User Forum, CERN (Switzerland) – March 1 st – 3 rd,2006 15 Rome Production experience on LCG  Jobs distributed to 45 different computing resources  Ratio generally proportional to the size of the cluster indicates an overall good job distribution.  No site in particular ran large majority of jobs. The site with the largest number of CPU resources (CERN), contributed for about 11% of the ATLAS production. Other major sites ran between 5% and 8% of the jobs each.  Achievement toward a more robust and fault-tolerant system does not rely on a small number of large computing centers. The percentage of ATLAS jobs run at each LCG site

16. Enabling Grids for E-sciencE Simone.Campana@cern.ch EGEE User Forum, CERN (Switzerland) – March 1 st – 3 rd,2006 16 Improvements: the Information System unstable  An unstable Information System can affect production in many aspects. Jobs might not match the full set of resources and flood a restricted number of sites  causing overload of some site services and leaving other available resources unused. Data management commands might fail to transfer input and output files  waste a large amount of CPU cycles  cause an overhead for the submission system and the production team  For the Rome production, several aspects were improved Fixes in BDII software BDII deployed as Load Balanced Service  Multiple backends under a DNS switch single point of failure  This reduced the single point of failure effect for both job submission and data management during the job execution.

17. Enabling Grids for E-sciencE Simone.Campana@cern.ch EGEE User Forum, CERN (Switzerland) – March 1 st – 3 rd,2006 17 Improvements: Site Configuration main source of failures during DC2  The main source of failures during DC2. No procedure in place during DC2  Treated on a case-by-case basis Unthinkable in the long timescale since  LCG infrastructure counts a very large number of resources  LCG grows very rapidly and is widely distributed.  Many sites started a careful monitoring of the number of job failures and developed automatic tools to identify problematic nodes. automatic tools  The LCG Operation team developed a series of automatic tools for site sanity controls See next slide

18. Enabling Grids for E-sciencE Simone.Campana@cern.ch EGEE User Forum, CERN (Switzerland) – March 1 st – 3 rd,2006 18 Improvements: Site Configuration  The Site Functional Tests running every day at every site test the correct configuration of the Worker Nodes and the interaction with Grid services. Now, can include VO specific tests and allows a VO specific view  The GIIS monitor checks the consistency of the information published by the site in the IS almost real time.

19. Enabling Grids for E-sciencE Simone.Campana@cern.ch EGEE User Forum, CERN (Switzerland) – March 1 st – 3 rd,2006 19 Improvements: Site Configuration  Freedom of Choosing Resources Allows the user to exclude mis-configured resources from the BDII  Generally based on SFT  Possible to whitelist or blacklist a resource Can exclude separately Storage and Computing Resources of the same site

20. Enabling Grids for E-sciencE Simone.Campana@cern.ch EGEE User Forum, CERN (Switzerland) – March 1 st – 3 rd,2006 20 Improvements: WMS and others highly automated  The LCG Workload Management System is highly automated designed to reduce the human intervention at the minimum consists in a complex set of services interacting with external components. unreliabilityduring DC2 This complexity caused a certain unreliability of the WMS during DC2. system became more robust  The system became more robust before the Rome production several bug fixes and optimizations in the WMS workflow. heterogeneous and dynamic nature  The heterogeneous and dynamic nature of a Grid environment implies a certain level of unreliability. ATLAS application improved to cope with such unreliability. lot of experience  The production team and the LCG operation and support teams gathered a lot of experience during DC2 and benefited from this experience at the time of Rome Production.

21. Enabling Grids for E-sciencE Simone.Campana@cern.ch EGEE User Forum, CERN (Switzerland) – March 1 st – 3 rd,2006 21 Issues (and possible improvements)  Failure Rates and Causes are shown in table  Still failure rate is quite high (~ 48%) Different failures imply different amounts of resource losses  Obviously the most serious reason is Data Management SystemCausesRate WMSTotal1.6% DMDownload input files26.4% DMUpload output files0.8 % ATLAS/LCGApplication-crash9.1% ATLASProxy Expired0.3% LCGSite misconfiguration0.9% Unclassified9%

22. Enabling Grids for E-sciencE Simone.Campana@cern.ch EGEE User Forum, CERN (Switzerland) – March 1 st – 3 rd,2006 22 Data Management: issues Reliable File Transfer  NO Reliable File Transfer service in place during Rome production. Data movement performed through LCG DM client tools. timeout and retry LCG DM tools did not provide timeout and retry capabilities  WORKAROUND: a timeout and possible retry was implemented in Lexor at some point consistency LCG DM tools not always ensure consistency between files in the SE and entries in the catalog  If the catalog is down or unreachable or the operation is killed prematurely  WORKAROUND: manual cleanup was needed mass storage systems  Data access on mass storage systems was very problematic Data need to be moved (staged) from tape to disk before being accessed. The middleware could not ensure the existence/persistency of data on disk WORKAROUND: manual pre-staging of files was carried out by the production team strategy for file distribution  The ATLAS strategy for file distribution must be (re)thought. Output files chaotically spread around 143 different Storage Elements. A replication schema for frequently accessed file was not in place Complicates analysis of the reconstructed samples and the production itself.

23. Enabling Grids for E-sciencE Simone.Campana@cern.ch EGEE User Forum, CERN (Switzerland) – March 1 st – 3 rd,2006 23 Data Management: improvements  Timeout and Retrynatively  Timeout and Retry capabilities introduced natively in the LCG DM tools. Also improved to guarantee atomic operations. LCG File Catalog  A new catalog, the LCG File Catalog, has been developed More stable Easier problem tracking Better performance and reliability Storage Resource Manager  The Storage Resource Manager interface introduced as a front-end to every SE. agreed on between experiments and middleware developers standardize storage access and management Offers more functionality for MSS access reliable File Transfer Service  A reliable File Transfer Service developed within the EGEE project. SERVICE It is a SERVICE Allows to replicate files between SEs in a reliable way. Built on top of gridFTP and SRM Capable to deal with data transfer from/to MSS

24. Enabling Grids for E-sciencE Simone.Campana@cern.ch EGEE User Forum, CERN (Switzerland) – March 1 st – 3 rd,2006 24 Data Management: improvements  FTSSRM  FTS and SRM SEs have been intensively tested during Service Challenge 3 (ongoing) Throughput exercise started in July 2005 and continuing at a low rate even now. Data Distribution from CERN T0 to several T1s  Some issues have been addressed Many issues already fixed Others being fixed by Service Challenge 4 (April 2006) positive feedback  In general, very positive feedback from experiments

25. Enabling Grids for E-sciencE Simone.Campana@cern.ch EGEE User Forum, CERN (Switzerland) – March 1 st – 3 rd,2006 25 Strategy for files distribution already in place  The new ATLAS DDM is already in place Fully tested and employed during SC3  Data throughput from CERN to ATLAS Tier 1s Target (80MB/s sustained for a week) fully achieved  ATLAS DM is being now integrated with ATLAS Production System  ATLAS DM is being now integrated with ATLAS Production System. DistributedATLAS Data Management  New Distributed ATLAS Data Management system logical dataset  Enforce concept of “logical dataset” collection of files being moved and located as a unique entity. Subscription Model  Dataset Subscription Model the site declare the interest in holding a dataset agents  ATLAS agents trigger migration of files Integrated with LFC, FTS and SRM

26. Enabling Grids for E-sciencE Simone.Campana@cern.ch EGEE User Forum, CERN (Switzerland) – March 1 st – 3 rd,2006 26 The Workload Management: issues performance  The performance of the WMS for job submission and handling generally acceptable in normal conditions  … but degrade under stress. WORKAROUND: several RBs dedicated to ATLAS and with different hardware solutions have been deployed enhanced WMS  The EGEE project will provide an enhanced WMS Possibility of bulk submission, bulk matching and bulk queries Improved communication with Computing Elements at sites submission speed  Possible improvement of job submission speed and job dispatching. promising results  Some preliminary tests show promising results Still several issue must be clarified

27. Enabling Grids for E-sciencE Simone.Campana@cern.ch EGEE User Forum, CERN (Switzerland) – March 1 st – 3 rd,2006 27 Monitoring: issues and improvements VO specific information  Lack of VO specific information about jobs at the sites GridICE sensors deployed in every site, but not correctly configured everywhere.  Partial information, difficult to interpret. Queries Queries to the ATLAS Production Database could cause an excessive load. error diagnostics  The error diagnostics should be improved performed parsing executor log files and querying the DB  should be formalized in proper tools  Real-timeinspection  Real-time job output inspection would have been helpful especially to investigate causes of hanging jobs. global job monitoring system  An ATLAS team is building a global job monitoring system Based on the current tools Possibly integrating new components (R-GMA etc …)

28. Enabling Grids for E-sciencE Simone.Campana@cern.ch EGEE User Forum, CERN (Switzerland) – March 1 st – 3 rd,2006 28 Conclusions overall successful exercise  The Rome Production on the LCG infrastructure has been an overall successful exercise Exercised the ATLAS production system Contributed to the testing of the ATLAS Computing and Data model Stress-tested the LCG infrastructure … and produced a lot of simulated data for the physicists!!!  Must be seen as the consequence of several improvements in the Grid middleware In the ATLAS components In the LCG operations several components need improvements  Still, several components need improvements both in terms of reliability and performance Production still requires a lot of human attention addressed to the relevant parties  Issues have been addressed to the relevant parties and a lot of work has been done since Rome Production Preliminary tests show promising improvements Will be evaluated fully in Service Challenge 4 (April 2006)