1. US ATLAS Distributed IT Infrastructure Rob Gardner Indiana University October 26, 2000 http://atlassw1.phy.bnl.gov/rwg-Planning/dit.html

2. 10/26/00US ATLAS Distributed IT Distributed IT Infrastructure Software –“Grid” toolkits: PPDG, GriPhyN, DataGrid –ATLAS extensions and adaptors Tier 2 regional centers Networks

3. 10/26/00US ATLAS Distributed IT GriPhyN 4 physics experiments + leading computer scientists in distributed computing ITR R&D project: 11.9M/5y ATLAS resources: –1 postdoc (physicist-computation) –2 grad students Significant matches from IU and BU –ITP2 (both); 0.5FTE of several IU IT personnel (IU)

4. 10/26/00US ATLAS Distributed IT Typical Tier 2 Regional Center (1 of 5) CPU: 50K SpecInt95 (t1: 209K) –Commodity Pentium/Linux –Estimated 144 Dual Processor Nodes (t1: 640) Online Storage: 70 TB Disk (t1: 365) –High Performance Storage Area Network –Baseline: Fiber Channel Raid Array

5. 10/26/00US ATLAS Distributed IT Tertiary Storage Capability Exploit existing mass store infrastructure at 2 of the 5 Tier 2 centers –Assume existing HPSS or equivalent liscense, tape silo, robot –Augment with drives, media, mover nodes, and disk cache –Each site contributes 0.3-0.5 PB store Reprocessed ESDs, User AODs,

6. 10/26/00US ATLAS Distributed IT Timeline (follows Tier 1) R&D Tier 2’s – FY ‘01 & FY ‘02 –Initial Development & Test, 1% to 2% scale –Start Grid testbeds: ATLAS-GriPhyN, PPDG, DataGrid Data Challenges – FY ‘03 & FY ‘04 Production Tier 2’s – FY ‘04 & FY ‘05 Operation – FY ‘05, FY ‘06 & beyond –Full Scale System Operation, 20% (‘05) to 100% (‘06)

7. 10/26/00US ATLAS Distributed IT

8. 10/26/00US ATLAS Distributed IT

9. 10/26/00US ATLAS Distributed IT

10. 10/26/00US ATLAS Distributed IT Tier 2 Costs Facilities and labor $K

11. ATLAS Grid Related Activities

12. 10/26/00US ATLAS Distributed IT ATLAS Grid Workshops June 14-15 at Indiana University –ATLAS-GriPhyN Testbed –ATLAS requirements for grid software. –Identify APIs between ATLAS and grid services. –Specify grid related milestones. –Identify deliverables for MOU, WBS documents. July 17-18 at CERN –First ATLAS-wide grid workshop –Talks from most major grid efforts in HEP – September 29-30 at CERN –Focus attention on ATLAS requirements http://atlasinfo.cern.ch/Atlas/GROUPS/SOFTWARE/OO/grid/

13. 10/26/00US ATLAS Distributed IT Participants at June Testbed Workshop Lab: – Larry Price (ANL) – Ed May (ANL) – Rich Baker (BNL) – Stu Loken (LBL) – David Malon (ANL) – Craig Tull (LBL) – Bruce Gibbard (BNL) – Torre Weanus (BNL) – David Quarrie (LBL) – Bill Allcock (ANL) University: – John Huth (HU) – Rob Gardner (IU) – Fred Luehring (IU) – Shawn McKee (UM) – Jim Shank (BU) – Steve Wallace (IU) – Leigh Grundhoefer (IU) – Thom Sulanke (IU) – Mary Papakian (IU) – Jane Liu (IU)

14. 10/26/00US ATLAS Distributed IT ATLAS GriPhyN Goals Provide linkage between Athena, database, simulation framework, and the grid toolkits –Feedback to software developers in both communities (ATLAS core developers and grid toolkit developers) Develop an ATLAS-GriPhyN Test-bed –Validate distributed computing model for LHC computing –Provide input to new models by testing tools and distributed functionality of ATLAS software –Provide input to planning for facilities development (at each Tier) and networks

15. 10/26/00US ATLAS Distributed IT Who’s doing what in the US? Participants and projects –Argonne: PPDG activities, Database-grid GriPhyN-DataGrid Integration –Berkeley Lab Athena grid interfaces, DOE sciences grid –Brookhaven Lab Tier 1 development, file replication, grid requirements document –Boston U Globus evaluations, applns –Harvard U US ATLAS computing management, planning –U of Michigan Globus interfaces, QoS authentication –Indiana U Athena grid interfaces, Testbed coordination, GIS UTA: D0 cluster, SAM SUNY Albany - test bed List is growing

16. 10/26/00US ATLAS Distributed IT Gaudi (Athena) Control Framework Converter Algorithm Event Data Service Persistency Service Data Files Algorithm Transient Event Store Detec. Data Service Persistency Service Data Files Transient Detector Store Message Service JobOptions Service Particle Prop. Service Other Services Histogram Service Persistency Service Data Files Transient Histogram Store Application Manager Converter LHC-B

17. 10/26/00US ATLAS Distributed IT Grid vs. Gaudi Services Craig Tull NERSC/LBL Athena and Grids: http://documents.cern.ch/age?a00380

18. 10/26/00US ATLAS Distributed IT ATLAS-GriPhyN Testbed Platform for testing grid concepts, computing models Provide input back to grid developers Expose weaknesses to better plan for infrastructure upgrades Identify and specify application-grid services interfaces Developers, administrators, users need grid experience Perform realistic test cases and make available as a test-suite Prepare infrastructure for ATLAS data challenges Distributed Monte Carlo production for TDRs

19. 10/26/00US ATLAS Distributed IT Calren Esnet, Abilene, Nton Abilene Esnet, Mren UC Berkeley LBNL-NERSC Esnet NPACI, Abilene Brookhaven National Laboratory Indiana University Boston University Argonne National Laboratory Initial Participants HPSS sites U Michigan University of Texas at Arlington SUNY Albany

20. 10/26/00US ATLAS Distributed IT ATLAS Applications Tile Cal testbeam data (expand on Ed May’s PPDG work) TRT System Test Data (Module Production –Fred Luehring) Monte Carlo Production Evaluation of Data Distribution Models Data Cataloging, Metadata, Bookkeeping

21. ATLAS Grid Schedules

22. 10/26/00US ATLAS Distributed IT ATLAS Data Challenges “DC” DC 1 first half 2002 –first project releases: of DATAGRID (PM12) is January 2002 of GriPhyN VDT-1 October 2001 ATLAS DC Milestones –were written on the safe side, but we will try to be more ambitious involve more sites and more complexity further define DC1 in next months getting involved GRID people and sites Computing TDR ready by end 2002: all experimentation needed before TDR to be planned for DC1

23. 10/26/00US ATLAS Distributed IT ATLAS Schedule Summary Significant milestones 11/01 – 12/01 – 11/01 – 12/01 Data Challenge 0 01/02 – 07/02 – 01/02 – 07/02 Data Challenge 1 » Provide input to computing model 05/02 – 11/02 – 05/02 – 11/02 Computing TDR 01/03 – 09/03 – 01/03 – 09/03 Data Challenge 2 »Major tests of grid enabled software 01/04 – 06/06 – 01/04 – 06/06 Physics Readiness Report

24. 10/26/00US ATLAS Distributed IT ATLAS GriPhyN Milestones (1) Year 1 –Perform tests using Globus and Particle Physics Data Grid tool kits providing basic file replication, transport, and security infrastructure. Data samples of order (1 TB) from test beam data recorded at CERN will be used in these tests. –Participate in development of information models for data types and elements. Develop metadata file catalogs, and tools to manipulate them to organize existing test beam and Monte Carlo data sets. –Begin implementing an ATLAS-GriPhyN test bed involving CERN and several US ATLAS institutions, including, Brookhaven, Argonne, and Berkeley Labs; Boston University, Indiana University, University of Michigan, UT Arlington, and SUNY-Albany –Begin identification and specification of APIs for the ATLAS analysis framework (Athena) and Monte Carlo programs. Evaluate and adopt basic grid services from the VDT-1 toolkit as they become available. –Provide feedback to CS researchers designing and planning VDT-2 services. ’00-’01

25. 10/26/00US ATLAS Distributed IT ATLAS GriPhyN Milestones (2) Year 2 –Deploy and evaluate VDT-2 (Centralized Virtual Data Services) on existing ATLAS-GriPhyN testbed focusing on support of the Athena analysis framework and Monte Carlo simulation programs. –Provide support to physicists located at many US sites and CERN requiring access to distributed data sets using metadata file catalogs, high speed multi-site file replication and network caching services. Testbeam data and Monte Carlo data data will be used, ranging in size of 2 to 10 TB. –Incorporate request planning and execution services into Athena framework. Gather policy and resource information for participating nodes on the grid. –Continue to expand US ATLAS testbed. –Tests of distributed object databases across several sites in the ATLAS-GriPhyN testbed. ’01-’02 DC1 DC0

26. 10/26/00US ATLAS Distributed IT ATLAS GriPhyN Milestones (3) Year 3 –Athena framework becomes fully distributed: incorporate VDT-3 toolkit (distributed virtual data services). –Perform tests of the hierarchy in a production mode. This will involve the CERN (as the Tier 0 site), Brookhaven Lab (as the US ATLAS Tier 1 site), and Tier 2 regional data centers –Begin multi-site distributed simulations to generate O(100) TB Monte Carlo data samples for physics studies and Data Challenges. –Continue to provide feedback especially with regard to fault tolerance in practical, realistic planning and request executions. ’02-’03 DC2 TDR

27. 10/26/00US ATLAS Distributed IT ATLAS GriPhyN Milestones (4) Year 4 –Scaled versions of previous Mock Data Challenges using VDT-4 (Scalable Virtual Data Services) using existing testbed between CERN, the Tier 1 center at BNL, Tier 2 regional centers, and of order 20 Tier 4 university groups with physicists doing Monte Carlo studies and analysis of testbeam data. –The goal is to approach realistic data samples >O(500 TB) and to involve thousands of computers. –Full integration of ATLAS analysis tools such as Athena and the Monte Carlo production control framework with virtual data services. ’03-’04 DC3

28. 10/26/00US ATLAS Distributed IT ATLAS GriPhyN Milestones (5) Year 5 –Continue to build on experience from data challengs in previous years. –Build a production quality offline distributed data analysis for the ATLAS GRID using GriPhyN tools. ’04-’05 Physics Readiness Report

29. 10/26/00US ATLAS Distributed IT ATLAS Grid Requirements EU DataGrid –To produce a grid requirements document June 1, 2001 ATLAS: –A group has been formed led by Rich Baker (BNL) and Larry Price (ANL) to produce an ATLAS grid requirements document by November 1, 2000 (draft)

30. ATLAS – CS Collaborative Projects

31. 10/26/00US ATLAS Distributed IT Possible CS-Physics Projects Athena grid interface Toolkit installation & distribution kit (a la INFN) Collaboration w/ DataGrid WP8 through grid forum, etc. Grid information services: user registration interface

32. 10/26/00US ATLAS Distributed IT Summary of Grid Activities ATLAS fully supports grid efforts and is redesigning its computing model in this new context Significant involvement of ATLAS personnel in PPDG, DataGrid, GriPhyN –This involvement will grow as more US ATLAS institutions become involved –GriPhyN needs to develop a clear and open policy for their contributions