1. Andrew C. Smith, 11 th May 2007 EGEE User Forum 2 - DIRAC Data Management System User Forum 2 Data Management System “Tell me and I forget. Show me and I remember. Involve me and I understand.” Chinese proverb

2. Andrew C. Smith, 11 th May 2007 EGEE User Forum 2 - DIRAC Data Management System Still to come… LHCb Computing Model (simplified) Requirements for Data Management System Introduction to DIRAC DIRAC Data Management System (DMS) Core DIRAC DM Components Bulk Transfer Framework Data Driven Automated Transfers Reliable Data Management Overview of EGEE resources used

3. Andrew C. Smith, 11 th May 2007 EGEE User Forum 2 - DIRAC Data Management System Reconstruction JobStripping Job Reconstructed RAW File (rDST) Stripped File (DST)RAW Physics FileRAW Replication LHCb Computing Model (Simplified) DST Broadcast

4. Andrew C. Smith, 11 th May 2007 EGEE User Forum 2 - DIRAC Data Management System DM Requirements in Numbers 2GB RAW file every ~30s Upload to Castor at 40MB/s on 1GB dedicated link. Each RAW file replicated from Castor to 1 of LHCb’s 6 Tier1s using shared 10GB links. Aggregated 40MB/s. Each Stripped DST produced is replicated to all Tier1s using dedicated network. Each Tier1 (on average) ~11MB/s in AND out..

5. Andrew C. Smith, 11 th May 2007 EGEE User Forum 2 - DIRAC Data Management System Introduction to DIRAC DIRAC is LHCb’s Grid Workload and Data Management System Initial incarnation as LHCb production system Since evolved into a generic Community Grid Solution Either stand-alone environment or Community Overlay Grid System (COGS) Architecture based on Services and Agents Implementing Service Oriented Architecture VO specific utilities can be tailored as required Demonstrated 10k concurrently running jobs Management of O(10M) Data files and replicas See Stuart Paterson’s talk for more on WMS

6. Andrew C. Smith, 11 th May 2007 EGEE User Forum 2 - DIRAC Data Management System DIRAC Core Data Management System FileCatalogueB FileCatalogueA SE Service SRMStorage RFIOStorage StoragePlugInX StorageElement ReplicaManager LCG File Catalogue User Interface WMS DM Agents Core DM Clients Physical Storage Core DM Components The main components are: Replica Manager File Catalogues Storage Element and access plug-ins

7. Andrew C. Smith, 11 th May 2007 EGEE User Forum 2 - DIRAC Data Management System DM Core Components Replica Manager provides logic for DM operations Interaction with StorageElement File upload/download/removal to/from Grid, File replication across SEs Interaction with File Catalog API File/replica registration/removal, Obtain replica information Logging of operations returned to client StorageElement is an abstraction of a Storage facility Access provided by plug-in modules for access protocols Current plug-ins: srm, gridftp, bbftp, sftp, http File Catalogue API All file catalogues offer same interface Can be used interchangeably LCG File Catalog (LFC), ProcessingDB…. VO specific resources easily integrated

8. Andrew C. Smith, 11 th May 2007 EGEE User Forum 2 - DIRAC Data Management System Other Key Components Data Management requests stored in RequestDB XML containing parameters for DM operation Operation type, LFN, etc…. Requests obtained and placed through RequestDB Service Transfer Agent polls RequestDB Service for work (multi-threaded) Contacts Replica Manager to perform DM operation Full log of operations returned Retries based on logging info Until success Redundacy built-in Transfer Agent Replica Manager RequestDB Svc Request Database ToDo Failed

9. Andrew C. Smith, 11 th May 2007 EGEE User Forum 2 - DIRAC Data Management System Data Mover DIRAC @ Online Gateway Transfer Agent Replica Manager Online Storage ADTDB LFC LHCb ONLINE SYSTEM RequestDB Svc Online Run Database Request Database CERN-IT File movement Request movement rfcp RFIO Plugin RAW Upload to Castor ToDo Done FC API

10. Andrew C. Smith, 11 th May 2007 EGEE User Forum 2 - DIRAC Data Management System Bulk Data Transfers gLite File Transfer Service (FTS) Provides point-to-point reliable bulk transfers Channel architecture SURLs at SRM X to SURLs at SRM Y Utilizing high throughput dedicated networks Network resources pledged to WLCG CERN-Tier1s Tier1-Tier1 matrix DIRAC DM System Interfaced to FTS Use FTS CLI to submit and monitor jobs DIRAC DM System Scheduling and placement of transfers Preparing source and target SURLs Transfer Agent Replica Manager EGEE FTS Svc SRM/G-U-C

11. Andrew C. Smith, 11 th May 2007 EGEE User Forum 2 - DIRAC Data Management System DIRAC DM components developed to perform data driven management AutoDataTransferDB (AdtDB) contains pseudo file catalogue Offers API to manipulate catalogue entries Based on ‘transformations’ contained in the DB Transformations defined for each DM operation to be performed Defines source and target SEs File mask (based on LFN namespace) Number of files to be transferred in each job Can select files of given properties and locations Replication Agent manipulates AdtDB API Checks active files in AdtDB Applies mask based on file type Checks the location of file Files which pass mask and match SourceSE selected for transformation Once threshold number of files found FTS jobs created ReplicationAgent logic generalised to support multiple transformation types Data Driven Production Management

12. Andrew C. Smith, 11 th May 2007 EGEE User Forum 2 - DIRAC Data Management System RAW Replication Data Mover DIRAC @ Online Gateway Transfer Agent Replica Manager Online Storage ADTDB LFC LHCb ONLINE SYSTEM RequestDB Svc Online Run Database Request Database CERN-IT File movement Request movement rfcp RFIO Plugin ToDo Done FC API When file uploaded to Castor registered in AdtDB This is the hook to data driven replication

13. Andrew C. Smith, 11 th May 2007 EGEE User Forum 2 - DIRAC Data Management System Transfer Agent Replica Manager RequestDB Svc Request Database DIRAC DM System File MovementRequest movement AdtDB Replication Agent Tier1 SRM WLCG FTS Svc SRM/G-U-C RAW Replication II LFC After replication registration LFC and ProcessingDB ProcessingDB drives data driven reconstruction and stripping jobs

14. Andrew C. Smith, 11 th May 2007 EGEE User Forum 2 - DIRAC Data Management System Reliable Data Management LHCb dedicated VO Box provided at Tier1s DIRAC instance installed RequestDB service TransferAgent Provides failover mechanism File upload from WN to associated SE If fails alternative SE chosen, ‘move’ request put to VO box Also provided initial mechanism for DST distribution DST uploaded to associated Tier1SE ‘Replication’ requests put to VOBoxes Proven capable of 100MB/s integrated across all Tier1s.

15. Andrew C. Smith, 11 th May 2007 EGEE User Forum 2 - DIRAC Data Management System Site (Country)Tape FilesTape Used (TB)Disk FilesDisk Used (TB) CERN (CH) 1660232191.5431650359.10 CNAF (IT) 13222919.6816057729.73 GRIDKA (DE) 11000114.5917000831.83 IN2P3 (FR) 316044.9329421447.36 PIC (ES) 14252019.3712872223.89 RAL (UK) 27481637.1221510840.74 SARA (NL) 408495.7016114530.33 Use of Resources During LHCb’s DC06 DIRAC’s DM System Stored 3.8M files at CERN + Tier1s 292TB of tape 262TB of disk +registration in the LCG File Catalogue

16. Andrew C. Smith, 11 th May 2007 EGEE User Forum 2 - DIRAC Data Management System DIRAC core DMS extensible, reliable, redundant VO specific resources plug-able 5 years of experiencing managing LHCb data Data driven operations to meet LHCb computing model Initial upload of RAW physics files Replication to Tier1s Broadcast of DSTs In the last year DIRAC DMS handled 3.8M files/replicas 292TB of tape 262TB of disk Summary

17. Andrew C. Smith, 11 th May 2007 EGEE User Forum 2 - DIRAC Data Management System Questions…?