1 The DataGrid WorkPackage 8 F.Carminati 28 June 2001

2FOCUSJune 28, 2001 Distributed computing Basic principle Physicists must have in principle equal access to data and resources The system will be extremely complex Number of sites and components in each site Different tasks performed in parallel: simulation, reconstruction, scheduled and unscheduled analysis Bad news is that the basic tools are missing Distributed authentication & resource management Distributed namespace for files and objects Local resource management of large clusters Data replication and caching WAN/LAN monitoring and logging Good news is that we are not alone These issues are central to the developments in the US and in Europe under the collective name of GRID

3FOCUSJune 28, 2001 The GRID “Dependable, consistent, pervasive access to [high-end] resources” Dependable: Can provide performance and functionality guarantees Consistent: Uniform interfaces to a wide variety of resources Pervasive: Ability to “plug in” from anywhere

4FOCUSJune 28, 2001 GLOBUS hourglass Focus on architecture issues Low participation cost, local control and support for adaptation Use to construct high-level, domain-specific solutions A set of toolkit services Security (GSI) Resource management (GRAM) Information services (MDS) Remote file management (GASS) Communication (I/O, Nexus) Process monitoring (HBM) Diverse global services Core Globus services Local OS A p p l i c a t i o n s

5FOCUSJune 28, 2001 DataGRID project Huge project 21 partners, 10MEuro, 3 years, 12 WorkPackages Global design still evolving An ATF has been mandated to design the architecture of the system to be delivered at PM6 (=now!) Continuous feed-back from users needed Users are in three workpackages HEP in WP8 Earth Observation in WP9 Biology in WP10

6FOCUSJune 28, 2001 DataGRID Project WP1 Grid Workload Management 11/18.6 WP2 Grid Data Management 6/17.6 WP3 Grid Monitoring Services 7/10 WP4 Fabric Management5.7/15.6 WP5 Mass Storage Management 1/5.75 WP6 Integration Testbed 6/27 WP7 Network Services0.5/9.4 WP8 High Energy Physics Applications1.7/23.2 WP9 Earth Observation Science Applications3/9.4 WP10 Biology Science Applications1.5/6 WP11 Dissemination and Exploitation1.2/1.7 WP12 Project Management2.5/2.5

7FOCUSJune 28, 2001 WP8 mandate Partners CNRS, INFN, NIKHEF, PPARC, CERN Coordinate exploitation of testbed by the HEP experiments Installation kits have been developed by all experiments Coordinate interaction of the experiments with the other WP’s and the ATF Monthly meetings of the WP8 TWG Common feedback to WP’s and to ATF Identify common components that could be integrated in an HEP “upper middleware” layer Large unfunded effort (776 m/m ~ 22 people) Very small funded effort (60 m/m ~ 1.5 people)

8FOCUSJune 28, 2001 The multi-layer hourglass OS & Net services Bag of Services (GLOBUS) DataGRID middleware PPDG, GriPhyn, EuroGRID HEP VO common application layer Earth Obs.Biology ALICEATLASCMSLHCb Specific application layer WP9WP 10 GLOBU S team DataGRI D ATF WP TWG

9FOCUSJune 28, 2001 CPU(SI95) DSK(GB) TP(TB) NWK(Mb/s) People Bari Birming (RAL?) ? 1 Bologna Cagliari Catania CERN (CASTOR) 2 Dubna GSI (D) ? (robot) 34 2 IRB (Kr) Lyon (HPSS) Merida (MX) NIKHEF (NL) ? (robot) OSU (US) (HPSS) Padova Saclay ? ? - ? 1 Torino UNAM (MX) ALICE Site list & Resources

10FOCUSJune 28, 2001 ALICE – Testbed Preparation ALICE Installation KIT (M. Sitta) List of ALICE Users (& CERTIFICATES) Distributed production test (R. Barbera) HPSS/CASTOR Storage (Y. Schutz) Logging/Bookkeeping (Y. Schutz) Input from stdout & stderr Store in MySQL/ORACLE Integration in DataGRID WP6 6/2001: L ist of ALICE users (& Certificates) to WP6 7/2001: Distributed test with bookkeeping system 8/2001: Distributed test with PROOF (reconstruction) 9/2001: Test with DataGRID WP6 resources 12/2001: Test M9 GRID Services release (preliminary)

11FOCUSJune 28, 2001 ALICE distributed analysis model Local Remote Selection Parameters Procedure Proc.C PROOF CPU TagD B RD B DB 1 DB 4 DB 5 DB 6 DB 3 DB 2 Bring the KB to the PB and not the PB to the KB

12FOCUSJune 28, 2001 AliRoot - DataGRID Services Reconstruction & Analysis for the ALICE Physics Performance Report will be the short term use case ALICE Data Catalog being developed data selection (files, to become objects soon TGlobus class almost completed interface to access GLOBUS security from ROOT TLDAP class being developed interface to access IMS from ROOT Parallel Socket Classes (TPServer, TPServerSocket) TFTP Class, root deamon modified parallel data transfer

13FOCUSJune 28, 2001 ATLAS Grid Activities - 1 ATLAS physicists are involved, and play relevant roles in the EU and US grid projects started in the last year EU DataGrid, Japan also active, US GriPhyN and PPDG The Computing Model is based on Tier-1, Tier-2 and local Tier-3 connected using grid Data Challenges planned for testing and tuning of the grid and the Model Till now work has been done for requirements and use case collection first tests of the available grid services interaction with the different projects for cooperation and at least interoperability

14FOCUSJune 28, 2001 ATLAS Grid Activities - 2 The ATLAS Grid Working Group (EU, US, Japan, etc.) meets regularly during the s/w weeks and 2 ad hoc workshop have been held Jobs have been run between different ATLAS sites using Globus + additions A s/w kit for Atlas Geant3 simulation + reconstruction on any Linux 6.2 or 7.1 machine is available and installed in INFN sites +CERN+ Glasgow and Grenoble. Inclusion of Athena-Atlfast being studied Production style jobs have been run between Milan- CERN-Rome. In July planned with also Glasgow, Grenoble and possibly LUND using the kit

15FOCUSJune 28, 2001 ATLAS Grid Activities - 3 Grid test using also Objectivity federations are planned in summer-autumn ATLAS is involved in the proposed projects EU DataTag and US iVDGL Transatlantic Testbed are between the deliverables Very important for testing EU-US interoperability and getting stricter cooperation between grid projects Links needed at level of projects: at experiment level links in ATLAS are excellent, but this is not enough Tests already going on between US and EU, but a lot of details (authorization, support) to be sorted out. More CERN involvement requested.

16FOCUSJune 28, 2001 CMS Issues CMS is a “single” collaboration  a coordination required among DataGrid, GriPhyN, PPDG and also INFN-Grid “Production” (Data Challenges) and CPT projects are the driving efforts Computing & Core Software=CCS + Physics Reconstruction & Selection= PRS + Trigger and Data Acquisition Systems= TriDAS Being Grid one of the CMS Computing activities, not all the CMS sites and people are committed to Grid Therefore (only) some of the CMS sites will be involved in first testbed activities The “grid” of sites naturally include EU and US sites (+ others countries/continents)

17FOCUSJune 28, 2001 Current CMS Grid Activities Definition of the CMS Applications Requirements to the Grid Planning and implementation of the first tests on “proto-Grid” tools Use of GDMP in some “real production” distributed environment [done] Make use of CA authorization in a distributed sites’ scenario [tested] Use of some Condor distributed (remote) submission schemes [done] Evaluate the PM9 deliverables of WP1 and WP2 (Grid Scheduler and enhanced Replica Management) Test dedicated resources to be provided by the participating sites To provide a reasonable complex and powerful trial To allow for “quasi-real” environment testing for the Applications CERN has a central role in these tests, and adeguate resource support has to be provided aside of the “Experiments’ Data Challenges” needed support.

18FOCUSJune 28, 2001 Current LHCb Datagrid activities Distributed MC production at CERN, RAL, Liverpool and Lyon Extending to Nikhef, Bologna, Edinburgh/Glasgow by end 2001 Current Testbed-0 tests using CERN and RAL (Globus problems encountered) Will extend to other sites in next few months Using parts of MC production system for ‘Short term use case’ Active participation in WP8 from UK, France, Italy, Nikhef, CERN (15 people part-time) Need more dedicated effort

19FOCUSJune 28, Production started by filling out a Web form:  Version of software  Acceptance cuts  Database  Channel  Number of events to generate  Number of jobs to run  Centre where the jobs should run 2. Web form calls a java servlet that:  Creates a job script (one per job)  Creates a cards file (one-three per job) with random number seeds and job options (The cards files need to be accessible by the running job)  Issues a job submit command to run script in batch-> WP1 3. Script does the following:  Copies executable, detector database and cards files  Executes executable  Executable creates output dataset  Output copied to local mass store -> WP5  Log file copied to web browsable area  Script calls java program (see 4) 4. Java program calls servlet at CERN to:  Transfer data back to CERN -> WP2  Update meta-database at CERN -> WP2 LHCb Short term Use Case for GRID

20FOCUSJune 28, 2001 LHCb – Medium to long term planning Would like to work with middleware WPs from now, helping test early prototype software with use case (parts of MC production system) More generally would like to be involved in the development of the architecture for the long term Writing planning document based on ‘use scenarios’- will develop library of use scenarios for testing Datagrid software Starting major project interfacing LHCb OO software framework GAUDI to GRID services Awaiting recruitment of ’new blood’

21FOCUSJune 28, 2001 GAUDI and external services (some GRID based) Converter Algorithm Event Data Service Persistency Service Algorithm Transie nt Event Store Detec. Data Service Persistency Service Transien t Detecto r Store Message Service JobOptions Service Particle Prop. Service Other Services Histogram Service Persistency Service Transien t Histogra m Store Application Manager Converter Event Selector Analysis Program OS Mass Storage Event Database PDG Database DataSet DB Other Monitoring Service Histo Presenter Other Job Service Config. Service GAUDI framework

22FOCUSJune 28, 2001 Issues and concers After a very difficult start the project seems now taking up speed Interaction with the WP’s and the ATF was difficult to establish Experiments are very active a large unfunded effort activity is ongoing Approximately 5-6 people per experiment Funded effort still very slow in coming (2 out of 5!) CERN testbed still insufficiently staffed & equipped DataGrid should have a central role in the LHC computing project – it is not the case now The GRID testbed and the LHC testbed should merge soon Interoperability with US Grid’s is very important We have to keep talking with them and open the testbed to US colleagues