ATLAS-Specific Activity in GridPP EDG Integration LCG Integration Metadata

22 September 2003GridPP8 meeting, Bristol 2 Personalia Alvin Tan (Birmingham) (3.1.4,3.1.7) Fully funded by GridPP to work on MC and production Clear link with GANGA activity, to which he contributes Alvin Tan (Birmingham) (3.1.4,3.1.7) Fully funded by GridPP to work on MC and production Clear link with GANGA activity, to which he contributes Frederic Brochu: (3.1.7, 3.1.9) Half-funded by GridPP Integration of EDG and LCG into ATLAS Framework Data Challenges Frederic Brochu: (3.1.7, 3.1.9) Half-funded by GridPP Integration of EDG and LCG into ATLAS Framework Data Challenges

22 September 2003GridPP8 meeting, Bristol 3 Mike Gardner (-2003) Fully funded post at RHUL (3.1.9) Installation and packaging tools Analysis architectures Fully funded post at RHUL (3.1.9) Installation and packaging tools Analysis architectures As alluded to in last report, Mike was ill and off work from the end of Q2 Despite encouraging signs, Mike died of liver cancer on Sunday 14 th September Last contribution a very nice paper for All Hands 03; we intend to publish a fuller version, with a dedication We will discuss the continuation of the post over the next week or so As alluded to in last report, Mike was ill and off work from the end of Q2 Despite encouraging signs, Mike died of liver cancer on Sunday 14 th September Last contribution a very nice paper for All Hands 03; we intend to publish a fuller version, with a dedication We will discuss the continuation of the post over the next week or so

22 September 2003GridPP8 meeting, Bristol 4 EDG Integration (3.1.4, 3.1.9) Recent UK mini production used EDG 1.4 input data stored on RAL’s tape server, the requirements in JDL the IC Resource Broker a boxed set executables. SiteRALCambridgeI CBirmingham Jobs Allocated Success The test took only 1 week, 1 operator and 3000 SpecInt95 days. The success rate was higher than 90 %. Not suitable for production but encouraging step towards a brokered production system. UK integrating validating EDG grid middleware for ATLAS EDG1.2 on the core sites revealed many problems with Resource Broker saturation and the use of a single Replica Catalogue (solved with RLS) The job success rate was only 70%.

22 September 2003GridPP8 meeting, Bristol 5 LCG-1 Integration & Validation (3.1.4) Partially functional LCG-1 release in July: Full functionality by October Pool integration underway, but bugs being fixed LCG-1 is only just ready for integration and validation: Deliverable will be missed because of late delivery (3 months) ATLAS-LCG team (FB a key member) now becomes ATLAS-EDG team EDG-2 is also just ready for integration and validation: We will also integrate this We will have another mini-production Partially functional LCG-1 release in July: Full functionality by October Pool integration underway, but bugs being fixed LCG-1 is only just ready for integration and validation: Deliverable will be missed because of late delivery (3 months) ATLAS-LCG team (FB a key member) now becomes ATLAS-EDG team EDG-2 is also just ready for integration and validation: We will also integrate this We will have another mini-production

22 September 2003GridPP8 meeting, Bristol 6 Installation & Packaging (3.1.9) Despite the loss of Mike, I&P activity continues: Incorporation into ATLAS standard release (internal deliverable by Dec) Extension to analysis code deployment SCRAM/DAR: ATLAS has committed to remaining with CMT, pacman, so this remains an active line of development Deliverables: production of the development kit (started) expected end 12/ deployment of the development kit on test site start: 08/03 end : 10/ integration with Grid infrastructure start: 09/03 end : 12/ systematic deployment of the development kit start: 10/03 end : 12/ production of the full source kit start: 09/03 end : 12/03 Looking at ways to make use of pacman from Ganga Concentration on deployment for analysis Despite the loss of Mike, I&P activity continues: Incorporation into ATLAS standard release (internal deliverable by Dec) Extension to analysis code deployment SCRAM/DAR: ATLAS has committed to remaining with CMT, pacman, so this remains an active line of development Deliverables: production of the development kit (started) expected end 12/ deployment of the development kit on test site start: 08/03 end : 10/ integration with Grid infrastructure start: 09/03 end : 12/ systematic deployment of the development kit start: 10/03 end : 12/ production of the full source kit start: 09/03 end : 12/03 Looking at ways to make use of pacman from Ganga Concentration on deployment for analysis

22 September 2003GridPP8 meeting, Bristol 7 Data Challenges (3.1.9) Data Challenge 2 is the key test of the Grid Computing Model Tools must be ready and tested for intensive tests in April-June (1 week bursts) Unlike DC1, data produced is not the main aim, and the computing is the client Frederic Brochu directs the DC in the UK DC Tools Task Force now created GANGA acknowledged as key input tool to the DC Alvin Tan is on the DCTF Integrating AMI, MAGDA etc Defining required bookkeeping and Metadata services Job analysis skeleton exists (SG, MG); this needs a lot more work for a production version Data Challenge 2 is the key test of the Grid Computing Model Tools must be ready and tested for intensive tests in April-June (1 week bursts) Unlike DC1, data produced is not the main aim, and the computing is the client Frederic Brochu directs the DC in the UK DC Tools Task Force now created GANGA acknowledged as key input tool to the DC Alvin Tan is on the DCTF Integrating AMI, MAGDA etc Defining required bookkeeping and Metadata services Job analysis skeleton exists (SG, MG); this needs a lot more work for a production version

22 September 2003GridPP8 meeting, Bristol 8 Test Bench –Data Challenges DC 1Jul 2002-May 2003 Showed the many resources available (hardware, willing people) Made clear the need for integrated system Some tests of Grid software Mainly driven by HLT and Physics Workshop needs One external driver is sustainable, two is not! DC2April-July 04 Real test of computing model for computing TDR Must use Grid systems Analysis and calibration + reconstruction and simulation Pre-production period (Nov03…) then 1-week intensive tests DC3 05/06 Physics readiness TDR. Big increase in scale

22 September 2003GridPP8 meeting, Bristol 9 At this stage the goal includes:  Full use of Geant4; POOL; LCG applications  Pile-up and digitization in Athena  Deployment of the complete Event Data Model and the Detector Description  Simulation of full ATLAS and 2004 combined Testbeam  Test the calibration and alignment procedures  Use widely the GRID middleware and tools  Large scale physics analysis  Computing model studies (document end 2004)  Run as much as possible the production on LCG-1  Combined Test beam operation foreseen as concurrent with DC2 and using same tools DC2: April – July 2004

 September 03: Release7  Mid-November 03: pre- production release  February 27 th 04: Release 8 (production)  April 1 st 04:  June 1 st 04: “DC2”  July 15th  Put in place, understand & validate:  Geant4; POOL; LCG applications  Event Data Model  Digitization; pile-up; byte-stream  Conversion of DC1 data to POOL; large scale persistency tests and reconstruction  Testing and validation  Run test-production  Start final validation  Start simulation; Pile-up & digitization  Event mixing  Transfer data to CERN  Intensive Reconstruction on “Tier0”  Distribution of ESD & AOD  Calibration; alignment  Start Physics analysis  Reprocessing DC2:Scenario & Time scale Test beam runs in parallel

22 September 2003GridPP8 meeting, Bristol 11 Metadata (3.1.9, GridPP2) Metadata definition and structure is a crucial issue for ATLAS DC2 Metadata must not only describe (collections of) files but (collections of events) and even at a sub-event level POOL provides one layer of metadata – how does this interact with RC etc? We require metadata services (middleware issue) but also an instantiation and design for our own Metadata workshop in Oxford, th July 2003 (RWLJ, AS, AT in attendance) Initial designs evolving GANGA must allow metadata queries, dynamic collection definition Metadata service integration, query services, metadata design will be key elements of the ATLAS GridPP2 activity.

22 September 2003GridPP8 meeting, Bristol 12 Papers/Presentations General ATLAS-UK Grid poster at Lepton-Photon 03 Installation & packaging paper and talk at All Hands GANGA paper & talk at all hands MC production system paper & poster at All Hands 03 Integration and validation paper & poster at All Hands 03

22 September 2003GridPP8 meeting, Bristol 13 Conclusions The GridPP effort for ATLAS has rightly focussed on GANGA The associated effort tries to realise the services that are needed for ATLAS production and analysis Installation and Packaging EDG and LCG integration and validation The DCs continue to test and validate the Grid tools Good progress, but Mike Gardner is a sad loss to the project The metadata services/schema/query tools/bookkeeping etc are moving onto the critical path and will be a major focus in GridPP2