May 2004Sverre Jarp1 Preparing the computing solutions for the Large Hadron Collider (LHC) at CERN Sverre Jarp, openlab CTO IT Department, CERN

May 2004Sverre Jarp2 Short overview of CERN

May 2004Sverre Jarp3 150m depth Accelerators and detectors in underground tunnels and caverns

May 2004Sverre Jarp4 CERN in numbers Financed by 20 European countries Special contributions also from other countries: USA, Canada, China, Japan, Russia, etc CHF (650 M€) budget to cover operation + new accelerators 2,200 staff (and diminishing) 6,000 users (researchers) from all over the world broad visitor and fellowship program

May 2004Sverre Jarp5 CERN User Community Europe: 267 institutes, 4603 users Elsewhere: 208 institutes, 1632 users

May 2004Sverre Jarp6 Computing at CERN

May 2004Sverre Jarp reconstruction simulation analysis interactive physics analysis batch physics analysis batch physics analysis detector event summary data raw data event reprocessing event reprocessing event simulation event simulation analysis objects (extracted by physics topic) event filter (selection & reconstruction) event filter (selection & reconstruction) processed data Data Management and Computing for Physics Analysis

May 2004Sverre Jarp8 High Energy Physics Computing Characteristics Independent events (collisions of particles) trivial (read: pleasant) parallel processing Bulk of the data is read-only versions rather than updates Meta-data in databases linking to “flat” files Compute power measured in SPECint (not SPECfp) But good floating-point is important Very large aggregate requirements: computation, data, input/output Chaotic workload – research environment - physics extracted by iterative analysis, collaborating groups of physicists  Unpredictable  unlimited demand

May 2004Sverre Jarp9 SHIFT architecture (Scalable Heterogeneous Integrated Facility) Tape server Disk server Batch server Interactive server Batch and disk SMP Network - Ethernet AFS In 2001 SHIFT won the 21st Century Achievement Award issued by Computerworld Disk server Batch server Tape server

May 2004Sverre Jarp10 CERN’s Computing Environment (today) High- throughput computing (based on reliable “commodity” technology) More than 1500 (dual Xeon processor) PCs with Red Hat Linux About 3 Petabytes of data (on disk and tapes)

May 2004Sverre Jarp11 IDE Disk servers Cost-effective disk storage: < 10 CHF/GB (mirrored)

May 2004Sverre Jarp12 The LHC Challenge

May 2004Sverre Jarp13 Large Hadron Collider A completely new particle accelerator The largest superconductor installation in the world Same tunnel as before; 27 km of magnets Super-fluid Helium cooled to 1.9°K Two counter-circulating proton beams in a field of 8.4 Tesla Collision energy: TeV Simulation tool: Sixtrack

May 2004Sverre Jarp14 CMS ATLAS LHCb Accumulating data at 10 Petabytes/year (plus replicated copies) Requirements: Storage – Raw recording rate: up to 1 GB/s per experiment 2 Petabytes of disk/year Total HEP processing needs – 50,000 (100,000) of today’s fastest processors The Large Hadron Collider (LHC) has 4 Detectors:

May 2004Sverre Jarp15 The Large Hadron Collider (LHC) goal: All charged tracks with pt > 2 GeV Reconstructed tracks with pt > 25 GeV (+30 minimum bias events) Find new physics, such as the Higgs particle, and get the Nobel price ! selectivity: 1 in person in a thousand world populations

May 2004Sverre Jarp16 LHC Computing Grid Project Goal of the project: To prepare, deploy and operate the computing environment for the experiments to analyse the data from the LHC detectors Phase 1 – development of common applications, libraries, frameworks, prototyping of the environment, operation of a pilot computing service Phase 2 – acquire, build and operate the LHC computing service The Grid is just a tool towards achieving this goal

May 2004Sverre Jarp17 RAL IN2P3 BNL FZK CNAF PIC ICEPP FNAL Computing Model (simplified!!) Tier-0 – the accelerator centre Filter  raw data Reconstruction  summary data (ESD) Record raw data and ESD Distribute raw and ESD to Tier-1 Tier-1: Permanent storage and management of raw, ESD, calibration data, meta-data, analysis data and databases  grid-enabled data service Data-heavy analysis Re-processing raw  ESD National, regional support “online” to the data acquisition process high availability, long-term commitment managed mass storage Tier-2: Well-managed disk storage – grid-enabled Simulation End-user analysis – batch and interactive High performance parallel analysis (PROOF ) USC NIKHEF Krakow CIEMAT Rome Taipei TRIUMF CSCS Legnaro UB IFCA IC MSU Prague Budapest Cambridge Tier-1 small centres Tier-2 desktops portables

May 2004Sverre Jarp18 RAL IN2P3 BNL FZK CNAF USC PIC ICEPP FNAL NIKHEF Krakow Taipei CIEMAT TRIUMF Rome CSCS Legnaro UB IFCA IC MSU Prague Budapest Cambridge Data distribution ~70 Gbits/sec

May 2004Sverre Jarp19 LCG Basics Getting the data from the detector to the grid requires sustained data collection and distribution keeping up with the accelerator To achieve the required levels of performance, reliability, resilience at minimal cost (people, equipment) we also have to work on scalability and performance of some of the basic computing technologies: cluster management mass storage management high performance networking  Workshop 3  Workshop 1  Workshop 2  Workshop 5

May 2004Sverre Jarp20 SW Rep Fabric Automation at CERN Node Cfg Cache SW Cache SPMA SWRep CDB OraMon NCMMSA SMS HMS LEMON LEAF Configuration Installation Fault & hardware Management Monitoring Includes technology developed by DataGrid

May 2004Sverre Jarp21 WAN connectivity Itanium-2 single stream: 5.44 Gbps 1.1 TB in 30 mins We now have to get from an R&D project (DATATAG) to a sustained, reliable service – GEANT, ESNET,.. Microsoft enters the stage: Multiple streams: 6.25 Gbps 20 April 2004

May 2004Sverre Jarp22 Preparing for 2007 The LCG installation is on a tight schedule Due to the need for deployment and development in parallel first data Initial service in operation Decisions on final core middleware Demonstrate core data handling and batch analysis Installation and commissioning

May 2004Sverre Jarp23 CERN openlab

May 2004Sverre Jarp24 openlab: The technology focus of CERN/IT Industrial Collaboration: Enterasys, HP, IBM, and Intel and Oracle are our partners Voltaire (with Infiniband switches) just joined Technology aimed at the LHC era: Network switches at 10 Gigabits ~ 100 rack-mounted HP servers 64-bit computing: Itanium-2 processors StorageTank storage system w/28 TB ~1 GB/s throughput

May 2004Sverre Jarp25 64-bit porting status Ported: Castor (data management subsystem) GPL. Certified by authors. ROOT (C++ data analysis framework) Own license. Binaries both via gcc and ecc. Certified by authors. CLHEP (class library for HEP) GPL. Certified by maintainers. GEANT4 (C++ Detector simulation toolkit) Own license. Certified by authors. CERNLIB (all of CERN’s FORTRAN software) GPL. In test. Zebra memory banks are I*4 ALIROOT (entire ALICE software framework) LCG-2 software from VDT/EDG GPL-like license. Being ported: CMS ORCA (part of CMS framework)

May 2004Sverre Jarp26 CERN “Where the Web was born…” ® CERN is busily preparing for the first arrival of LHC data in 2007 New and exciting technologies are needed to manage the data Seamlessly, around the globe Together with our partners (EU, industry, other Physics Labs, other sciences) we expect to come up with interesting proofs-of-concept and technological spin-off ! High Throughput Computing is “on the move” ! People Motivation Technology Science Innovation