DataGrid Applications Federico Carminati WP6 WorkShop December 11, 2000

DataGrid WP6 Workshop 2 11 December 2000 The distributed computing model Assumptions Raw data will be kept at CERN – (backup if affordable) Tier 1 will have ~10% of raw data Reconstruction will be done at CERN (2 passes) ESD (aka DST) and TAG will be shipped to Tier1-2 Simulation will all be done in the Tier1-2, data will be shipped to CERN (in which form? Raw, ESD, AOD) The Tier1-2 will process ESD to produce AOD (formerly aka n- tuple) as many time as necessary, producing their own TAG Users will access the ESD/AOD/TAG remotely

DataGrid WP6 Workshop 3 11 December 2000 The distributed computing model Basic principle Every physicist should have in principle equal access to the data and to the resources The system will be extremely complex Number of components in each site Number of sites Different tasks performed in parallel: simulation, reconstruction, scheduled and unscheduled analysis

DataGrid WP6 Workshop 4 11 December 2000 WP 8 philosophy Define a common upper middle-layer of GRID services common to the 4 experiments Common API for common tasks File replica Job submission and monitoring …. Collaborate to define a common set of requirements and milestones, also with WP 9/10 Share the same testbeds and facilities for data challenges Introduce a user view in the project

DataGrid WP6 Workshop 5 11 December 2000 WP8 plans and requirements Testbed Release 0 (1Q 2001) A working, standard, installation of Globus, at CERN and at other labs. Standard recipes for file transfer and job submission (at CERN, this installation will probably be interfaced to LSF). A contact point for GLOBUS and later DataGRID software in each lab incase of problems A clear policy for "experiment-wide" authorisation to allow testing across national boundaries. This will only work if enough support is provided to users WP8 activities in general require to have a substantial part of the GRID services available since as soon as possible

DataGrid WP6 Workshop 6 11 December 2000 WP8 plans and requirements Testbed release 1 (3Q 2001) Distributed user autentication and resource allocation or pre- allocation, as dynamic allocation can come later. Distributed data dictionary (location of files on different servers). Basic network configuration, monitoring and diagnostic tools. Basic monitoring and diagnostic of a cluster of PC's. Distributed scheduling for the jobs that are submitted in a coordinated way (this does not include the "chaotic job activity" coming from isolated users which should be addressed in the subsequent release). Access to the basic information about job status and errors. Guidelines for configurating farms of PC's with fast disk access.

DataGrid WP6 Workshop 7 11 December 2000 WP8 plans and requirements Testbed release 2 (3Q 2002) Replica management and network optimised trasfer of data from different file systems. Tools for configurating farms of PC's with fast disk access, monitoring their status and for automathized s/w installation and management. A prototype of scheduling and load balancing for chaotic analysis jobs Basic functionality for dynamic resource allocation Basic functionality for job partitioning

DataGrid WP6 Workshop 8 11 December 2000 WP8 plans and requirements Testbed release 3 (3Q 2003) Scheduling and balancing of chaotic analysis Tools to ensure "robustness" and error recovery of the system

DataGrid WP6 Workshop 9 11 December 2000 WP8 Wkshop Nov 16 Conclusions WP8 non-experiment-specific personpower Needed to implement WP8 common policies CERN person identified and hired (I.Augustin) Should come from the funded contribution of each partner (CNRS, CERN, PPARC, NIKHEF and INFN) as a share of the 60 funded person-months A message sent to the PMB for each partner to identify this personpower

DataGrid WP6 Workshop December 2000 WP8 Wkshop Nov 16 Conclusions Two kinds of Test-bed participation. Formal participation defined by WP6 Informal commitment to install Datagrid provided tools, and participate in tests. Kick-start and Update kits Experiments to provide WP8 with installation and upgrade kits WP8 personpower will install them in the WP6 test-beds locations Kits have to coexist on the same machines without interference Locations participating but not in WP6 will provide their own personpower for the installation This activity will be coordinated by the CERN WP8 person (I.Augustin)

DataGrid WP6 Workshop December 2000 WP8 Wkshop Nov 16 Conclusions Collection of requirements Present WP8 requirements judged vague by other WP's (rightly!) Other WP's should have asked WP8/9/10 questions – they didn’t Meeting of December 1 st of the ATF was rather inconclusive New strategy: WP8-10 to produce a three tiered document Short term use cases Long term use cases General requirements WP8-10 also to produce pilot applications December 15 ATF should consolidate user requirements DataGRID Workshop on January 15 US experts invited to discuss user requirements and first proposition of architecture

DataGrid WP6 Workshop December 2000 WP8 Wkshop Nov 16 Conclusions Set up a technical WP8 technical WG one application software expert from each experiment and from ESA (WP9) and biology (WP10) the WP8 experiment-neutral personpower in the different partners (5 people) Chaired by the CERN WP8 person acting as WP8 architect)

DataGrid WP6 Workshop December 2000 WP8 Wkshop Nov 16 Conclusions Main tasks Help WP8 architect to collect the requirements from the experiment and ESA and Biology for the ATF Liaise with the Middleware WP's (1-5) regarding the services required by the applications, and the definition of the appropriate interfaces. Discuss WP8 architectural questions Do WP8-10 require a common 'upper middleware' layer over middleware services? Do they want to interface to generic middleware services directly? Definition of the 'sample' applications CMS and LHCb have existing running distributed applications ALICE is following and ATLAS will see whether it can add its Define requirements for the Test-bed and Networking

DataGrid WP6 Workshop December 2000 WP8 Wkshop Nov 16 Conclusions Main tasks (cont) Provide technical liaison with the Test-bed and Networking workpackages (e.g. attend their meetings) Special priority points are, for instance 1. Provision of Standard, Supported, Documented Globus installation kit 2. Definition of Test-bed sites and contact people 3. Definition of Certification system to be employed by the project Compact sub-group Will meet, virtually or in person, in its own right, preparing our point of view for the global Test-bed and Architecture meetings Experiment TB reps, unless the same person, would normally not attend, unless required for a forthcoming TB meeting The frequency of meetings decided by WP8 architect and coordinator.

DataGrid WP6 Workshop December 2000 Conclusions Activity of WP8 well started Main concerns are Architecture design seems to start with difficulty We do not have an architect yet Coordination with WP9-10 not yet very effective Need better communication with WP1-5 Need to start interacting more effectively with WP6-7