“Replica Management in LCG” Workshop on Spatiotemporal Databases for Geosciences, Biomedical sciences and Physical sciences “Replica Management in LCG” James Casey, Grid Deployment Group, CERN E-Science Institute, Edinburgh, 2nd November 2005 James.Casey@cern.ch

Talk Overview LHC and the Worldwide LCG Project LHC Data Management Architecture Replica Management Components Storage Catalog Data Movement User APIs & tools CERN Grid Deployment

The LHC Experiments CMS ATLAS LHCb CERN Grid Deployment

The Atlas Detector The ATLAS collaboration is The ATLAS detector is ~2000 physicists from ~ 150 universities and labs from ~ 34 countries distributed resources remote development The ATLAS detector is 26m long, stands 20m high, weighs 7000 tons has 200 million read-out channels 20 m ~ a 5 story building CERN Grid Deployment

equivalent to 2 PetaBytes/sec Atlas detector Data Acquisition Multi-level trigger Filters out background Reduces data volume Record data 24 hours a day, 7 days a week Equivalent to writing a CD every 2 seconds 40 MHz interaction rate equivalent to 2 PetaBytes/sec Level 1 - Special Hardware Level 2 - Embedded Processors Level 3 – Giant PC Cluster 160 Hz (320 MB/sec) Data Recording & Offline Analysis CERN Grid Deployment

Worldwide LCG Project - Rationale Satisfies the common computing needs of the LHC experiments Need to support 5000 scientists at 500 institutes; Estimated project lifetime: 15 years; Processing requirements: 100,000 CPUs (2004 units); Traditional, centralised approached ruled out in favour of a globally distributed grid for data storage and analysis: Costs of maintaining and upgrading a distributed system more easily handled - individual institutes and organisations can fund local computing resources and retain responsibility for these, while still contributing to the global goal. No single points of failure. Multiple copies of data and automatic reassigning of tasks to available resources ensures optimal use of resources. Spanning all time zones also facilitates round-the-clock monitoring and support. From http://lcg.web.cern.ch/LCG/overview.html CERN Grid Deployment

LCG Service Deployment Schedule CERN Grid Deployment

(extracted by physics topic) Data Handling and Computation for Physics Analysis reconstruction detector event filter (selection & reconstruction) analysis processed data event summary data raw data batch physics analysis event reprocessing simulation analysis objects (extracted by physics topic) event simulation interactive physics analysis les.robertson@cern.ch CERN Grid Deployment

WLCG Service Hierarchy Tier-0 – the accelerator centre Data acquisition & initial processing Long-term data curation Distribution of data  Tier-1 centres Canada – Triumf (Vancouver) France – IN2P3 (Lyon) Germany – Forschungszentrum Karlsruhe Italy – CNAF (Bologna) Netherlands – NIKHEF (Amsterdam) Nordic countries – distributed Tier-1 Spain – PIC (Barcelona) Taiwan – Academia Sinica (Taipei) UK – CLRC (Didcot) US – FermiLab (Illinois) – Brookhaven (NY) Tier-1 – “online” to the data acquisition process  high availability Managed Mass Storage –  grid-enabled data service Data intensive analysis National, regional support Tier-2 – ~100 centres in ~40 countries Simulation End-user analysis – batch and interactive Les Robertson CERN Grid Deployment

How much data in one year? Balloon (30 Km) How much data in one year? CD stack with 1 year LHC data! (~ 20 Km) Storage Space Data produced is ~15PB/year Space provided at all tiers is ~80PB Network bandwidth 70 Gb/s to the big centres Direct dedicated lightpaths to all centres Used only for Tier-0 -> Tier-1 data distribution Number of files ~ 40 million files assuming 2GB files and it runs for 15 years Concorde (15 Km) Mt. Blanc (4.8 Km) CERN Grid Deployment

Data Rates to Tier-1s for p-p running Centre ALICE ATLAS CMS LHCb Rate into T1 (pp) MB/s ASGC, Taipei - 8% 10% 100 CNAF, Italy 7% 13% 11% 200 PIC, Spain 5% 6.5% IN2P3, Lyon 9% 27% GridKA, Germany 20% RAL, UK 3% 15% 150 BNL, USA 22% FNAL, USA 28% TRIUMF, Canada 4% 50 NIKHEF/SARA, NL 23% Nordic Data Grid Facility 6% Totals 1,600 This is 135TB/day of actual data to be distributed These rates must be sustained to tape 24 hours a day, 100 days a year. Extra capacity is required to cater for backlogs / peaks. This is currently our biggest data management challenge. CERN Grid Deployment

Problem definition in one line… “…to distribute, store and manage the high volume of data produced as the result of running the LHC experiments and allow subsequent ‘chaotic’ analysis of the data” Data comprises of Raw data ~90% Processed data ~10% “relational” metadata ~1% “middleware-specific” metadata ~ .001% Main problem is movement of raw data To the Tier-1 sites as an “online” process - volume To the analysis sites – chaotic access pattern We’re really dealing with the non-analysis use cases right now CERN Grid Deployment

Replica Management Model Write-Once/Read-Many files Avoid issue of replica consistency No mastering User accesses data via a logical name Actual filename on storage system is irrelevant No strong authorization on storage itself All users in a VO are considered the same No usage of user identity on MSS Storage uses unix permissions Different users represent different “roles” e.g experiment production managers group == VO Simple user-initiated replication model upload/replicate/download cycle CERN Grid Deployment

Replica Management Model All replicas are considered the same a replica is “close” if in same network domain Explicitly made close to a particular cluster By the information system Or local environment variables This is basically the model inherited from European DataGrid (EDG) Data Management software Although all the software has been replaced! CERN Grid Deployment

Replica Management components Each file has a unique Grid ID. Locations corresponding to the GUID are kept in the Replica Catalog. Users select data via metadata. This is in the Experiment Metadata Catalog. The file transfer service provides reliable asynchronous 3rd party file transfer. Experiment Metadata Catalog Transfer Service Client tools Replica Catalog Blue: WP2 services The client interacts with the grid via the experiment framework, and LCG APIs. Files have replicas stored at many Grid sites on Storage Elements. Storage Element Storage Element CERN Grid Deployment

Software Architecture Layered Architecture Experiments hook in at whatever layer they require Focus on Core Services Experiments integrate into their own replication framework Not possible to provide a generic data management model for all four experiments Provide C/python/perl APIs and some simple CLI tools Data Management model still based on EDG model Suggest change is trying to introduce a better security model But our users don’t really care about it, only the performance penalty it gives them ! CERN Grid Deployment

Software Architecture LCG software model heavily influenced (by) EDG First LCG middleware releases came directly out of the EDG project Globus 2.4 is used as a basic lower layer Gridftp for data movement Globus GSI Security model and httpg for web service security We are heavily involved in EGEE We take components out of the EGEE gLite release and integrate them into the LCG release And we write our own components we need to But should be a very last resort! (LCG Data Management team is ~2FTE) CERN Grid Deployment

Layered Data Management APIs Experiment Framework User Tools Data Management (Replication, Indexing, Querying) lcg_utils Cataloging Storage Data transfer GFAL Component Specific APIs Classic SE File Transfer Service Globus Gridftp EDG LFC SRM CERN Grid Deployment

Summary : What can we do? Store the data Find the data Access the data Managed Grid-accessible storage Including interface to MSS Find the data Experiment metadata catalog Grid replica catalogs Access the data LAN “posix-like” protocols gridftp on the WAN Move the data Asynchronous high bandwidth data movement Throughput more important that latency CERN Grid Deployment

Storage Model We must manage storage resources in an unreliable distributed large heterogeneous system We must make the MSS at Tier-0/Tier-1 and the disk based storage appear the same to the users Long lasting data intensive transactions Can’t afford to restart jobs Can’t afford to loose data, especially from experiments Heterogeneity Operating systems MSS - HPSS, Enstore, CASTOR, TSM Disk systems – system attached, network attached, parallel Management Issues Need to manage more storage with less people CERN Grid Deployment

Storage Resource Manager (SRM) Collaboration between LBNL, CERN, FNAL, RAL, Jefferson Lab Became the GGF Grid Storage Management Working Group http://sdm.lbl.gov/srm-wg/ Provides a common interface to Grid Storage Exposed as a Web Service Negotiable transfer protocols (Gridftp, gsidcap, RFIO, …) We use three different implementations CERN CASTOR SRM – for CASTOR MSS DESY/FNAL dCache SRM LCG DPM – disk only lightweight SRM for Tier-2s CERN Grid Deployment

SRM / MSS by Tier1 Centre SRM MSS Tape H/W Canada, TRIUMF dCache TSM France, CC-IN2P3 HPSS STK Germany, GridKA LTO3 Italy, CNAF CASTOR STK 9940B Netherlands, NIKHEF/SARA DMF Nordic Data Grid Facility DPM N/A Spain, PIC Barcelona Taipei, ASGC UK, RAL ADS CASTOR(?) USA, BNL USA, FNAL ENSTOR CERN Grid Deployment

Catalog Model Experiments own and control the metadata catalog All interaction with grid files is via a GUID (or LFN) obtained from their metadata catalog Two models for tracking replicas Single global replica catalog LHCb Central metadata catalog stores pointers to site local catalogs which contain the replica information ALICE/ATLAS/CMS Different implementations used LHC File Catalog (LFC), Globus RLS, experiment developed catalogs This is a “simple” problem, but we keep revisting it CERN Grid Deployment

Accessing the Data Grid File Access Layer (GFAL) lcg_util originally a low-level I/O interface to Grid Storage provides “posix-like” I/O abstraction Now provides: File Catalog abstraction Information system abstraction Storage Element Abstraction (EDG SE, EDG ‘Classic’ SE, SRM v1) lcg_util Provides a replacement for the EDG Replica Manager Provides both direct C library calls and CLI tools Is a thin wrapper on top of GFAL Has extra experiment requested features compared to the EDG Replica Manager CERN Grid Deployment

Managed Transfers gLite File Transfer Service (FTS) is a fabric service It provides point to point movement of SURLs Aims to provide reliable file transfer between sites, and that’s it! Allows sites to control their resource usage Does not do ‘routing’ Does not deal with GUID, LFN, Dataset, Collections Provides Sites with a reliable and manageable way of serving file movement requests from their VOs Users with an async reliable data movement interface VO developers with a pluggable agent-framework to monitor and control the data movement for their VO CERN Grid Deployment

Summary LCG will require a large amount of data movement Production use-cases demand high-bandwidth distribution of data to many sites in a well-known pattern Analysis use cases will provide chaotic, unknown replica access patterns We have a solution for the first problem This is out our main focus Tier-1’s are “online” to the experiment The second is under way The accelerator is nearly upon us And then it’s full service until 2020 ! CERN Grid Deployment

Thank you http://lcg.web.cern.ch/LCG/ CERN Grid Deployment

Backup Slides CERN Grid Deployment

Computing Models CERN Grid Deployment

CMS Computing Model Overview CERN Grid Deployment

LHCb Computing Model Overview CERN Grid Deployment

Data Replication CERN Grid Deployment

Types of Storage in LCG-2 3 “classes” of storage at sites Integration of large (tape) MSS (at Tier 1 etc) – Responsibility of site to make the integration Large Tier 2’s – sites with large disk pools (100s Terabytes, many fileservers), need a flexible system dCache provides a good solution Needs effort to integrate and manage Sites with smaller disk pools (1 – 10 Terabytes), less available management effort Need a lightweight (install, manage) solution LCG Disk Pool Manager is a solution for this problem CERN Grid Deployment

Catalogs CERN Grid Deployment

Catalog Model Experiment responsibility to keep metadata catalog and replica catalog (either local or global) in sync LCG tools only deal with global case, since each local case is different LFC is able to be used as either a local or global catalog component Workload Management picks sites with replica by querying a global Data Location Interface (DLI) Can be provided by either Experiment metadata catalog Global grid replica catalog (e.g LFC) CERN Grid Deployment

LCG File Catalog Provides a filesystem-like view of grid files Hierarchical namespace and namespace operations Integrated GSI Authentication + Authorization Access Control Lists (Unix Permissions and POSIX ACLs) Fine grained (file level) authorization Checksums User exposed transaction API CERN Grid Deployment