1. L.M.Barone – INFN Rome 16-20 October 2000 ACAT2000 - FNAL Management of Large Scale Data Productions for the CMS Experiment Presented by L.M.Barone Università di Roma & INFN

2. L.M.Barone – INFN Rome 16-20 October 2000 ACAT2000 - FNAL The Framework The CMS experiment is producing a large amount of MC data for the development of High Level Trigger algorithms (HLT) for fast data reduction at LHC Current production is half traditional (Pythia + CMSIM/Geant3) half OO (ORCA using Objectivity/DB)

3. L.M.Barone – INFN Rome 16-20 October 2000 ACAT2000 - FNAL The Problem Data size ~ 10 6 - 10 7 events, 1 MB/ev ~ 10 4 files (typically 500 evts/file) Resource dispersion many production sites CERN,FNAL,Caltech, INFN etc. Dealing with actual MC productions and not with 2005 data taking

4. L.M.Barone – INFN Rome 16-20 October 2000 ACAT2000 - FNAL The Problem (cont’d) Data Relocation data produced in site A are stored centrally (CERN); site B may need a fraction of them; combinatorics increasing Objectivity/DB does not make life easier (but the problem would exist anyway)

5. L.M.Barone – INFN Rome 16-20 October 2000 ACAT2000 - FNAL ORCA Production 2000 Signal Zebra files with HITS ORCA Digitization (merge signal and MB) Objectivity Database HEPEVT ntuples CMSIM HLT Algorithms New Reconstructed Objects MC Prod. ORCA Prod. HLT Grp Databases ORCA ooHit Formatter Objectivity Database MB Objectivity Database Catalog import Objectivity Database Objectivity Database ytivitcejbOesabataD Mirrored Db’s (US, Russia, Italy..)

6. L.M.Barone – INFN Rome 16-20 October 2000 ACAT2000 - FNAL The Old Days Question: how was it done before ? A mix of ad hoc scripts/programs with a lot of manual intervention... but the problem was smaller and less dispersed

7. L.M.Barone – INFN Rome 16-20 October 2000 ACAT2000 - FNAL Requirements for a Solution Solution must be as automatic as possible  decrease manpower Tools should be independent from data type and from site Network traffic should be optimized (or minimized ?) Users need complete information on data location

8. L.M.Barone – INFN Rome 16-20 October 2000 ACAT2000 - FNAL Present Status Job creation is managed by a variety of scripts in different sites Job submission again goes through diverse methods, from UNIX commands to LSF or Condor File transfer has been managed up to now by Perl scripts  not generic, not site independent

9. L.M.Barone – INFN Rome 16-20 October 2000 ACAT2000 - FNAL Present Status (cont’d) The autumn 2000 production round is a trial towards standardization  same layout (OS, installation)  same scripts (T.Wildish) for non Objy data transfer  first use of GRID tools (see talk by A.Samar)  validation procedure for production sites

10. L.M.Barone – INFN Rome 16-20 October 2000 ACAT2000 - FNAL Collateral Activities Linux + CMS software automatic installation kit (INFN) Globus installation kit (INFN) Production monitoring tools with Web interface

11. L.M.Barone – INFN Rome 16-20 October 2000 ACAT2000 - FNAL What is missing ? Scripts and tools are still too specific and not robust enough  need practice on this scale Information service needs a clear definition in our context and then an effective implementation (see later) File replication management is just appearing and needs careful evaluation

12. L.M.Barone – INFN Rome 16-20 October 2000 ACAT2000 - FNAL Ideas for Replica Management A case study with Objectivity/DB (thanks to C.Grandi Bologna,INFN) –can be extended to any kind of file

13. L.M.Barone – INFN Rome 16-20 October 2000 ACAT2000 - FNAL Cloning federations Cloned federations have a local catalog (boot file) –It is possible to manage each of them in an independent way. Some databases may be attached (or exist) only in one site –“Manual work” is needed to keep the schemas synchronized (this is not the key point today...)

14. L.M.Barone – INFN Rome 16-20 October 2000 ACAT2000 - FNAL CERN CERNFD DB1 DB2 DB3 DBn CERN Boot RC1FD RC1 Boot RC2FD RC2 Boot Cloning federations Clone FD DB_a DB_b

15. L.M.Barone – INFN Rome 16-20 October 2000 ACAT2000 - FNAL Productions Using a DB-id pre-allocation system it is possible to produce databases at RCs which can then be exported to other sites –A notification system is needed to inform other sites when a database is completed –This is today accomplished by GDMP using a publish-subscribe mechanism

16. L.M.Barone – INFN Rome 16-20 October 2000 ACAT2000 - FNAL Productions When a site receives notification, it can: – ooattachdb to the remote site DB –copy the DB and ooattachdb it locally –ignore it

17. L.M.Barone – INFN Rome 16-20 October 2000 ACAT2000 - FNAL Productions CERN CERNFD DB1 DB2 DB3 DBn CERN Boot RC1FD RC1 Boot RC2FD RC2 Boot DBn+1 DBn+m DBn+m+1 DBn+m+k

18. L.M.Barone – INFN Rome 16-20 October 2000 ACAT2000 - FNAL Analysis In each site a complete catalog with the location of all the datasets is needed. Some DBs are local and some are remote In case more copies of a DB are available it would be nice to have in the local catalog the closest one (NWS)

19. L.M.Barone – INFN Rome 16-20 October 2000 ACAT2000 - FNAL Information service Create an Information Service with information about all the replicas of the databases (GIS ?) In each RC there is a reference catalog which is updated taking into account the available replicas It is even possible to have a catalog created on-the-fly only for the datasets needed by a job

20. L.M.Barone – INFN Rome 16-20 October 2000 ACAT2000 - FNAL Analysis CERN CERNFD DB1 DB2 DB3 DBn CERN Boot RC1FD RC1 Boot RC2FD RC2 Boot DBn+1 DBn+m DBn+m+1 DBn+m+k DBn+m+k DBn+m+1 DBn+m DBn+1

21. L.M.Barone – INFN Rome 16-20 October 2000 ACAT2000 - FNAL Logical vs Physical Datasets Each dataset is composed by one or more databases –datasets are managed by application-sw Each DB is uniquely identified by a DBid –DBid assignment is a logical-db creation The physical-db is the file –zero, one or more instancies The IS manages the link between a dataset, its logical-dbs and its physical- dbs

22. L.M.Barone – INFN Rome 16-20 October 2000 ACAT2000 - FNAL Logical vs Physical Datasets Dataset: H  2  Dataset: H  2e Hmm.1.hits.DB Hmm.2.hits.DB Hmm.3.hits.DB Hee.1.hits.DB id=12345 id=12346 id=12347 id=5678 Hee.2.hits.DB id=5679 Hee.3.hits.DB id=5680 pccms1.bo.infn.it::/data1/Hmm1.hits.DB shift23.cern.ch::/db45/Hmm1.hits.DB pccms1.bo.infn.it::/data1/Hmm2.hits.DB shift23.cern.ch::/db45/Hmm2.hits.DB shift23.cern.ch::/db45/Hmm3.hits.DB pccms5.roma1.infn.it::/data/Hee1.hits.DB shift49.cern.ch::/db123/Hee1.hits.DB pccms5.roma1.infn.it::/data/Hee2.hits.DB shift49.cern.ch::/db123/Hee2.hits.DB shift49.cern.ch::/db123/Hee3.hits.DB pccms5.roma1.infn.it::/data/Hee3.hits.DB pccms3.pd.infn.it::/data3/Hmm2.hits.DB

23. L.M.Barone – INFN Rome 16-20 October 2000 ACAT2000 - FNAL Database creation In each production site we have: –a production federation including incomplete databases –a reference federation with only complete databases (both local and remote ones) When a DB is completed it is attached to the site reference federation The IS monitors the reference federations of all the sites and updates the database list

24. L.M.Barone – INFN Rome 16-20 October 2000 ACAT2000 - FNAL RC1Prod DB4 DB4 RC1Ref pc.rc1.net CERN CERNFD DB1 DB2 DB3 DB4 0001 DB1.DB shift.cern.ch::/shift/data 0002 DB2.DB shift.cern.ch::/shift/data 0003 DB3.DB shift.cern.ch::/shift/data 0004 DB4.DB pc.rc1.net::/pc/data shift.cern.ch::/shift/data 0005 Database creation DB5 DB5 DB5 DB5 0001 DB1.DB shift.cern.ch::/shift/data 0002 DB2.DB shift.cern.ch::/shift/data 0003 DB3.DB shift.cern.ch::/shift/data 0004 DB4.DB pc.rc1.net::/pc/data shift.cern.ch::/shift/data 0005 DB5.DB pc.rc1.net::/pc/data 0001 DB1.DB shift.cern.ch::/shift/data 0002 DB2.DB shift.cern.ch::/shift/data 0003 DB3.DB shift.cern.ch::/shift/data 0004 DB4.DB pc.rc1.net::/pc/data shift.cern.ch::/shift/data 0005 DB5.db pc.rc1.net::/ps.data shift.cern.ch::/shift/data shift.cern.ch

25. L.M.Barone – INFN Rome 16-20 October 2000 ACAT2000 - FNAL Replica Management In case of multiple copies of the same DB each site may choose which copy to use: –it should be possible to update the reference federation at given times –it should be possible to create on-the-fly a mini-catalog only with information about the datasets requested by a job this kind of operation is managed by application- sw (e.g. ORCA)

26. L.M.Barone – INFN Rome 16-20 October 2000 ACAT2000 - FNAL 0001 DB1.DB shift.cern.ch::/shift/data pc1.bo.infn.it::/data 0002 DB2.DB shift.cern.ch::/shift/data 0003 DB3.DB shift.cern.ch::/shift/data 0001 DB1.DB shift.cern.ch::/shift/data pc1.bo.infn.it::/data 0002 DB2.DB shift.cern.ch::/shift/data pc1.bo.infn.it::/data 0003 DB3.DB shift.cern.ch::/shift/data Replica Management CERN CERNFD DB1 DB2 DB3 BORef shift.cern.ch pc1.bo.infn.it PDRef pc1.pd.infn.it DB1 DB2

27. L.M.Barone – INFN Rome 16-20 October 2000 ACAT2000 - FNAL Summary of the Case Study Basic functionalities of a Replica Manager for production are already implemented in GDMP The use of an Information Server would allow easy synchronization of federations and optimized data access during analysis The same functionalities offered by the Objectivity/DB catalog may be implemented for other kind of files

28. L.M.Barone – INFN Rome 16-20 October 2000 ACAT2000 - FNAL Conclusions (?) Globus and the various GRID projects try to address the issue of Large Scale distributed data access Their effectiveness is still to be proven The problem again is not the software, it is the organization