IT Department and The LHC Computing Grid Visit of Mr. François Gounand Conseiller de l’Administrateur Général du CEA Conseiller pour les très grandes installations de Recherche au Ministère de la Recherche Frédéric Hemmer Deputy Head, IT Department October 4, 2006 The LHC Computing Grid – October 2006

Outline IT Department in brief Fabrics The LCG Project Beyond LCG The Challenge The (W)LCG Project The LCG Infrastructure The LCG Service Beyond LCG Real-Time LCG monitor The LHC Computing Grid – October 2006

Services provided by IT Dept (I) Basic Services Campus and external networking, Internet Exchange Point Productivity tools (Windows, Linux, Mac, exchange, office tools, other applications) PC-shop via Stores, Printing, backups, phones, faxes Security, user support Administrative Computing Engineering Support Databases (Oracle), EDMS, maths tools Electrical and mechanical engineering Simulation The LHC Computing Grid – October 2006

Services provided by IT Dept (II) Computing for Physics Software process support, database applications Interactive and batch services (Linux & Solaris) Central data recording, mass storage Linux support, system management Control systems (SCADA, PLCs, Fieldbuses, …) Major projects: LCG, EGEE, openlab Tier0/Tier1 centre at CERN Data challenges, grid deployment & operations, middleware Advanced high-speed networks, transatlantic connections Service Challenges Collaboration with industry The LHC Computing Grid – October 2006

A few IT Department numbers Around 280 staff positions Adding fellows, associates, students and visitors, we are nearly 500 people in total Materials budget is ~50 MCHF in 2006 with the largest fraction devoted to LHC computing, dropping to ~40MCHF/year ~3000 central computers, 1’500 TB disk storage, 10PB tape storage capacity (being increased for LHC) ~15000 accounts The LHC Computing Grid – October 2006

CERN openlab Concept Partner/contributor sponsors latest hardware, software and brainware (young researchers) CERN provides experts, test and validation in Grid environment Partners: 500’000 €/ year, 3 years Contributors: 150’000 €, 1 year Current Activities Platform competence centre Grid interoperability centre Security activities Joint events The LHC Computing Grid – October 2006

Computer Center The LHC Computing Grid – October 2006

Computer Centre Upgrade 2006 2004 2005 The LHC Computing Grid – October 2006

Computer Centre Automation Developed Systems Management Tools Quattor Automated installation, configuration & management of clusters Lemon Computer Center Monitoring System Leaf Hardware and State Management Castor Advance Storage Manager moving data from disk to tape and vice-versa The LHC Computing Grid – October 2006 9

The LHC Computing Grid The LHC Computing Grid – October 2006

New frontiers in data handling ATLAS experiment: ~150 million channels @ 40MHz ~ 10 million Gigabytes per second Massive data reduction on-line Still ~1 Gigabyte per second to handle The LHC Computing Grid – October 2006

The Data Challenge The accelerator will be completed in 2007 and run for 10-15 years LHC experiments will produce 10-15 million Gigabytes of data each year (about 20 million CDs!) LHC data analysis requires a computing power equivalent to ~100,000 of today's fastest PC processors. Requires many cooperating computer centres, CERN providing only ~20% of the computing resources The LHC Computing Grid – October 2006

(extracted by physics topic) Data Handling and Computation for Physics Analysis reconstruction detector event filter (selection & reconstruction) analysis processed data event summary data raw data batch physics analysis event reprocessing simulation analysis objects (extracted by physics topic) event simulation interactive physics analysis les.robertson@cern.ch The LHC Computing Grid – October 2006

LCG Service Hierarchy Tier-0 – the accelerator centre Data acquisition & initial processing Long-term data curation Data Distribution to Tier-1 centres Canada – Triumf (Vancouver) France – IN2P3 (Lyon) Germany –Karlsruhe Italy – CNAF (Bologna) Netherlands – NIKHEF/SARA (Amsterdam) Nordic countries – distributed Tier-1 Spain – PIC (Barcelona) Taiwan – Academia SInica (Taipei) UK – CLRC (Oxford) US – FermiLab (Illinois) – Brookhaven (NY) Tier-1 – “online” to the data acquisition process  high availability Managed Mass Storage –  grid-enabled data service All re-processing passes Data-heavy analysis National, regional support Tier-2 – ~100 centres in ~40 countries Simulation End-user analysis – batch and interactive Services, including Data Archive and Delivery, from Tier-1s The LHC Computing Grid – October 2006

Summary of Computing Resource Requirements ~ 100K of today’s fastest processors Summary of Computing Resource Requirements All experiments - 2008 From LCG TDR - June 2005 CERN All Tier-1s All Tier-2s Total CPU (MSPECint2000s) 25 56 61 142 Disk (PetaBytes) 7 31 19 57 Tape (PetaBytes) 18 35 53 CPU Disk Tape The LHC Computing Grid – October 2006

WLCG Collaboration The Collaboration 4 LHC experiments ~120 computing centres 12 large centres (Tier-0, Tier-1) 38 federations of smaller “Tier-2” centres Growing to ~40 countries Memorandum of Understanding Agreed in October 2005, now being signed Resources Commitment made each October for the coming year 5-year forward look The LHC Computing Grid – October 2006

The LHC Computing Grid – October 2006

The new European Network Backbone LCG working group with Tier-1s and national/ regional research network organisations New GÉANT 2 – research network backbone  Strong correlation with major European LHC centres Swiss PoP at CERN The LHC Computing Grid – October 2006

LHC Computing Grid Project - a Collaboration Building and operating the LHC Grid – a global collaboration between The physicists and computing specialists from the LHC experiments The national and regional projects in Europe and the US that have been developing Grid middleware The regional and national computing centres that provide resources for LHC The research networks Researchers Computer Scientists & Software Engineers Service Providers The LHC Computing Grid – October 2006

LCG depends on 2 major science grid infrastructures … The LCG service runs & relies on grid infrastructure provided by: EGEE - Enabling Grids for E-Science OSG - US Open Science Grid The LHC Computing Grid – October 2006

LCG Service planning Pilot Services – stable service from 1 June 06 2006 cosmics LHC Service in operation – 1 Oct 06 over following six months ramp up to full operational capacity & performance 2007 LHC service commissioned – 1 Apr 07 first physics 2008 full physics run The LHC Computing Grid – October 2006

Service Challenge 4 (SC4) the Pilot LHC Service from June 2006 A stable service on which experiments can make a full demonstration of their offline chain DAQ  Tier-0  Tier-1 data recording, calibration, reconstruction Offline analysis - Tier-1  Tier-2 data exchange simulation, batch and end-user analysis And sites can test their operational readiness LCG services -- monitoring  reliability Grid services Mass storage services, including magnetic tape Extension to most Tier-2 sites Target for service by end September – Service metrics  90% of MoU service levels Data distribution from CERN to tape at Tier-1s at nominal LHC rates The LHC Computing Grid – October 2006

CERN Tier-1 Data Distribution Sustaining ~900 MBytes/sec The data rate required during LHC running for all four experiments is 1.6 GBytes/sec – -- which was demonstrated during a test period in April ATLAS alone has moved 1 PetaByte of data during its data challenge between 19 June and 7 August CMS has moved 3.8 PetaBytes of data between sites during a four months period this week The LHC Computing Grid – October 2006

CMS Data Transfers The LHC Computing Grid – October 2006

Production Grids for LHC what has been achieved Basic middleware A set of baseline services agreed and initial versions in production Pro-active grid operation – distributed across several sites All major LCG sites active ~50K jobs/day > 10K simultaneous jobs during prolonged periods on the EGEE grid Reliable data distribution service demonstrated at 1.6 GB/sec CERNTier-1s mass storage to mass storage = nominal LHC data rate The LHC Computing Grid – October 2006

Impact of the LHC Computing Grid in Europe LCG has been the driving force for the European multi-science Grid EGEE (Enabling Grids for E-sciencE) EGEE is now a global effort, and the largest Grid infrastructure worldwide Co-funded by the European Commission (~130 M€ over 4 years) EGEE already used for >20 applications, including… Bio-informatics Education, Training Medical Imaging The LHC Computing Grid – October 2006

The EGEE Project Infrastructure operation Middleware Currently includes >200 sites across 40 countries Continuous monitoring of grid services & automated site configuration/management http://gridportal.hep.ph.ic.ac.uk/rtm/launch_frame.html Middleware Production quality middleware distributed under business friendly open source licence User Support - Managed process from first contact through to production usage Training Documentation Expertise in grid-enabling applications Online helpdesk Networking events (User Forum, Conferences etc.) Interoperability Expanding interoperability with related infrastructures The LHC Computing Grid – October 2006

EGEE Grid Sites : Q1 2006 sites EGEE: Steady growth over the lifetime of the project CPU EGEE: > 180 sites, 40 countries > 24,000 processors, ~ 5 PB storage The LHC Computing Grid – October 2006

Applications on EGEE More than 20 applications from 7 domains Astrophysics MAGIC, Planck Computational Chemistry Earth Sciences Earth Observation, Solid Earth Physics, Hydrology, Climate Financial Simulation E-GRID Fusion Geophysics EGEODE High Energy Physics 4 LHC experiments (ALICE, ATLAS, CMS, LHCb) BaBar, CDF, DØ, ZEUS Life Sciences Bioinformatics (Drug Discovery, GPS@, Xmipp_MLrefine, etc.) Medical imaging (GATE, CDSS, gPTM3D, SiMRI 3D, etc.) Multimedia Material Sciences … The LHC Computing Grid – October 2006

Example: EGEE Attacks Avian Flu EGEE used to analyse 300,000 possible potential drug compounds against bird flu virus, H5N1. 2000 computers at 60 computer centres in Europe, Russia, Taiwan, Israel ran during four weeks in April - the equivalent of 100 years on a single computer. Potential drug compounds now being identified and ranked Neuraminidase, one of the two major surface proteins of influenza viruses, facilitating the release of virions from infected cells. Image Courtesy Ying-Ta Wu, AcademiaSinica. The LHC Computing Grid – October 2006

Example: Geocluster industrial application The first industrial application successfully running on EGEE Developed by the Compagnie Générale de Géophysique (CGG) in France, doing geophysical simulations for oil, gas, mining and environmental industries. EGEE technology helps CGG to federate its computing resources around the globe. The LHC Computing Grid – October 2006

European e-Infrastructure Coordination Evolution European e-Infrastructure Coordination EGEE EGEE-II EDG EGEE-III Testbeds Utility Service Routine Usage The LHC Computing Grid – October 2006

The LHC Computing Grid – October 2006