José M. Hernández CIEMAT Grid Computing in the Experiment at LHC Jornada de usuarios de Infraestructuras Grid January 2012, CIEMAT, Madrid

José Hernández The CMS Experiment at the LHC 20 January 2012 Grid Computing in CMS 2 The Large Hadron Collider p-p collisions, 7 TeV, 40 MHz The Compact Muon Solenoid Precision measurements Search for new phenomena

José Hernández LHC: a challenge for computing  The Large Hadron Collider at CERN is the largest scientific instrument on the planet  Unprecedented data handling scale  40 MHz event rate (~1 GHz collision rate) → 100TB/s → online filtering to ~300 Hz (~300 MB/s) → ~3 PB/year (10 7 secs data taking/year)  Need large computing power to process data  Complex events  Many interesting signals << Hz  Thousands of scientists around the world access and analyze the data  Need computing infrastructure able to store, move around the globe, process, simulate and analyze data at the Petabyte scale [O(10) PB/year] 3

José Hernández The LHC Computing Grid 4 LCG: 300+ centers, 50+ countries, ~100k CPUs, ~ 100PB disk/tape, 10k users  The LHC Computing Grid provides the distributed computing infrastructure  Computing resources (CPU, storage, networking)  Computing services (data and job management, monitoring, etc)  Integrated to provide a single LHC computing service  Using Grid technologies  Transparent and reliable access to heterogeneous computing resources geographically distributed via internet  High capacity wide area networking

José Hernández The CMS Computing Model  Distributed computing model for data storage, processing and analysis  Grid technologies (Worldwide LHC Computing Grid, WLCG)  Tiered architecture of computing resources  ~20 Petabytes of data (real and simulated) every year  About 200k jobs (data processing, simulation production and analysis) per day

José Hernández WLCG network infrastructure 20 January 2012 Grid Computing in CMS 6  T0-T1 and T1-T1 interconnected via LHCOPN (10 Gpbs links)  T1-T2 and T2-T2 using general research networks  Dedicated network infrastructure (LHCONE) being deployed

José Hernández Grid services in WLCG  Middleware providers: gLite/EMI, OSG, ARC  Global services: data transfers and job management, authentication / authorization, information system  Compute (gateway, local batch system, WNs) and storage (gridftp servers, disk servers, mass storage system) elements at the sites  Experiment specific services 20 January 2012 Grid Computing in CMS 7

José Hernández CMS Data and Workload Management  Experiment-specific DMWM services on top of basic Grid services  Pilot-based WMS  Data bookkeeping, location and transfer systems  Data pre-located  Jobs go to data  Experiment software pre-installed at sites 20 January 2012 Grid Computing in CMS 8 Production System (WMAgent) Analysis System (CRAB) Data Bookkeeping & location system (DBS) Data Transfer System (PhEDEx) gLite WMS File Transfer System CE SE Local batch system Mass storage system CMS Services Grid Services Sites Operators Users Pilot-based WMS

José Hernández CMS Grid Operations - Jobs  Large scale data processing & analysis  ~50k used slots, 300k jobs/day  Plots correspond Aug 2011 – Jan January 2012 Grid Computing in CMS 9

José Hernández Spanish contribution to CMS Computing Resources 10  Spain contributes with ~ 5% of the CMS computing resources PIC Tier-1  ~1/2 average Tier-1  3000 cores, 4 PB disk, 6 PB tape IFCA Tier-2  ~ 2/3 average Tier-2 (~3% T2 resources)  1000 CPUs, 600 TB disk CIEMAT Tier-2  ~ 2/3 average Tier-2 (~3% T2 resources)  1000 cores, 600 TB disk

José Hernández Contribution from Spanish sites 20 January 2012 Grid Computing in CMS 11  ~5 % of total CPU delivered for CMS CPU delivered Feb 2011 – Jan 2012

José Hernández CMS Grid Operations - Data  Large scale data replication  1-2 GB/s throughput CMS-wide  ~1 PB/week data transfers  Full mesh 50+ sites T0 T1 T1 T2 T2 20 January 2012 Grid Computing in CMS 12 1 GB/s Production transfers debug transfers 1 GB/s

José Hernández Site monitoring/readiness 20 January 2012 Grid Computing in CMS 13

José Hernández Lessons learnt 20 January 2012 Grid Computing in CMS 14  Porting the production and analysis applications to the Grid was easy  Package job wrapper and user libraries into input sandbox  Experiment software pre-installed at the sites  Job wrapper sets up environment, runs the job, stages out output  When running at large scale in WLCG, additional services are needed  Job and data management services on top of Grid services  Data bookkeeping and location  Monitoring

José Hernández Lessons learnt 20 January 2012 Grid Computing in CMS 15  Monitoring is essential  Multi-layer complex system (experiment, Grid, site layers)  Monitor workflows, services, sites  Experiment services should be robust  Deal with (inherent) Grid unreliability  Be prepared for retries, cool-off  Pilot-based WMS  gLite BDII and WMS not reliable enough  Smaller overhead, verify node environment, global priorities, etc  Isolating users from the Grid; Grid operations team  Lots of manpower needed to operate the system  Central operations team (~20 FTE)  Contacts at sites (50+)

José Hernández Future developments 20 January 2012 Grid Computing in CMS 16  Dynamic data placement/deletions  Most of the pre-located data not really accessed much  Investigating automatic replication of hot data, deletion of cold data  Replicate data when accessed by jobs and cache locally  Remote data access  Jobs go to free slots and access data remotely  CMS has improved a lot read performance over WAN  At the moment only used as fail-over and overflow  Service to asynchronously copy user data  Remote stage out from WN is a bad idea  Multi-core processing  More efficient use of multi-core nodes, savings in RAM, many less jobs to handle

José Hernández Future developments 20 January 2012 Grid Computing in CMS 17  Virtualization of WNs/Cloud computing  Decouple node OS and application environment using VMs or chroot  Allow use of opportunistic resources  CERN VMFS for experiment software

José Hernández Summary 20 January 2012 Grid Computing in CMS 18  CMS has been very successful in using the LHC Computing Grid at large scale  Lot of work to make the system efficient, reliable and scalable  Some developments in the pipeline to make CMS distributed computing more dynamic and transparent