1. David Adams ATLAS ADA, ARDA and PPDG David Adams BNL June 28, 2004 PPDG Collaboration Meeting Williams Bay, Wisconsin

2. David Adams ATLAS ATLAS and ARDA ARDA WorkshopJune 21, 20042 Contents Definitions ADA/DIAL model Current status of ADA Connections to PPDG Conclusions

3. David Adams ATLAS ATLAS and ARDA ARDA WorkshopJune 21, 20043 Definitions ADA – ATLAS Distributed Analysis ATLAS project to deliver an end-to-end analysis system ARDA – A Realisation of Distributed Analysis for LHC CERN project to coordinate the development of prototype distributed analysis systems for the LHC experiments Prototypes are based on the EGEE middleware (gLite) DIAL – Distributed Interactive Analysis of Large Datasets BNL project to study interactive analysis of large amounts of data Delivered framework is also suitable for non-interactive jobs AJDL – Abstract Job Definition Language Generic and extensible definition of jobs intended for use with any batch or workload management system Components: transformation (application and task), dataset and job

4. David Adams ATLAS ATLAS and ARDA ARDA WorkshopJune 21, 20044 ADA/DIAL model User interaction scenario: Select a dataset –AJDL, not HEPCAL Examine dataset –Display content (event ID’s, type-keys) –Location of data (logical files, physical files, DB, …) >Including means to localize data Define transformation –Application that processes the data –Task carrying data to configure application –Select these from catalog and modify Create job to apply transformation to dataset –Submit app, task and dataset to scheduler (e.g. analysis service) –Result will be a new dataset

5. David Adams ATLAS ATLAS and ARDA ARDA WorkshopJune 21, 20045 ADA/DIAL model (cont) User interactions (cont) Monitor job –Status (running, done, failed, …) –Status of sub-jobs –Examine partial result Examine final result (output dataset) Repeat Following block diagram shows major components Client High-level services Middleware services

6. David Adams ATLAS ATLAS and ARDA ARDA WorkshopJune 21, 20046 ADA/DIAL model (cont) Release 1.0 Started

7. David Adams ATLAS ATLAS and ARDA ARDA WorkshopJune 21, 20047 Current status of ADA Framework based on DIAL 0.90 Clients ROOT from DIAL (using rootcint) Python from GANGA (using lcgdict with DIAL classes) GUI being developed by GANGA

8. David Adams ATLAS ATLAS and ARDA ARDA WorkshopJune 21, 20048 Current status of ADA (cont) Analysis services DIAL provides the starting point for all the following Interactive service (i.e. low latency for short jobs) –Currently uses a fast LSF queue –Condor COD being tested –Running at BNL –Will be deployed at other sites –May also be deployed for long-running jobs Service to insert jobs in ATLAS production queue is being developed –Jobs may be run on any of the ATLAS grids ARDA service based on gLite being developed –Jobs sent to gLite WMS

9. David Adams ATLAS ATLAS and ARDA ARDA WorkshopJune 21, 20049 Current status of ADA (cont) Applications DC1 CBNT  histograms in place DC2 reconstruction for release 8.0.1 in place Working on DC2 event data  histograms Capability to reuse production xforms being developed Datasets Generic classes in DIAL DC1: CbntDataset and HbookDataset DC2: AtlasPoolEventDataset and RootHistogramDataset Catalogs AMI adding required tables and user interface –Repositories, selection catalogs, … Programmatic access via AMI web service

10. David Adams ATLAS ATLAS and ARDA ARDA WorkshopJune 21, 200410 Connections to PPDG DIAL Provides many parts of ADA –The overall framework –The ROOT client –Infrastructure for all analysis services delivered and under development –Connection to LSF, Condor (including COD) Depends on PPDG for support and exchange of ideas

11. David Adams ATLAS ATLAS and ARDA ARDA WorkshopJune 21, 200411 Connections to PPDG (cont) AJDL Facilitates the development of clients or services without DIAL –E.g. plan for GANGA python interface is to drop dependence on DIAL C++ libraries >Lightweight client for any platform Goal to standardize within PPDG –Send non-ATLAS jobs to ATLAS analysis service –And vice versa

12. David Adams ATLAS ATLAS and ARDA ARDA WorkshopJune 21, 200412 Connections to PPDG (cont) U.S. grid infrastructure U.S. sites important to ATLAS –And especially to USATLAS Assume continuing support from PPDG –Grid3, grid3+, OSG, … Most sites continue taking jobs from ATPROD queues A few sites to also host interactive analysis service

13. David Adams ATLAS ATLAS and ARDA ARDA WorkshopJune 21, 200413 Connections to PPDG (cont) Monitoring and modeling US monitoring data fed into ATLAS computing model –Job data from ADA catalogs –Site data from grid monitoring ACM is interested in constructing models to understand analysis patterns –Collaboration with CS-11?

14. David Adams ATLAS ATLAS and ARDA ARDA WorkshopJune 21, 200414 Connections to PPDG (cont) EGEE middleware Install gLite on one or more U.S. sites –At least services required to support WMS >CE, SE, … Submit jobs from the ATLAS-ARDA prototype Common project –Also submit jobs from CMS and Alice prototypes Will U.S. sites deploy EGEE middleware? –Important issue for evolution of ADA

15. David Adams ATLAS ATLAS and ARDA ARDA WorkshopJune 21, 200415 Connections to PPDG (cont) File cataloging and movement SRM likely to become standard –Licensing issues? Many replica catalogs relevant to ATLAS –Magda (phased out for DC2) –EDG RLS (LCG catalog) –ISI RLS (separate catalogs for U.S. and NorduGrid) –EGEE/AliEn file catalog (required for ARDA prototype?) Also many options for high-level interfaces –ATLAS internal system Don Quijote –CMS file transfer system –EGEE file transfer system (same as CMS?) –LBL RMS/RRS PPDG can help sort this out

16. David Adams ATLAS ATLAS and ARDA ARDA WorkshopJune 21, 200416 Conclusions ATLAS has benefited in may ways from PPDG Magda, DIAL, AJDL, SRM, Grid3, … Ideas in HEPCAL and ARDA Continue to look to PPDG Forum to exchange ideas Provider or identifier of useful tools and services Support for U.S. grid infrastructure