1. David Adams ATLAS DIAL/ADA JDL and catalogs David Adams BNL December 4, 2003 ATLAS software workshop Production session CERN

2. David Adams ATLAS DIAL/ADA JDL ATLAS SW – Prod sessionDecember 4, 20032 Contents DAC mandate Scope Strategy Scenario for first release Plans for the first release GANGA status DIAL status Deliverables for the first release Conclusions

3. David Adams ATLAS DIAL/ADA JDL ATLAS SW – Prod sessionDecember 4, 20033 ADA Strategy Implement DA as a collection of grid services As described in ARDA document Use ARDA components where possible Add missing and ATLAS-specific pieces Provide clients for ATLAS analysis environments Python, ROOT, command line Regular releases Perhaps for each SW week and ATLAS X.0 Provide useful tool Demonstrate functionality Expand functionality with each release

4. David Adams ATLAS DIAL/ADA JDL ATLAS SW – Prod sessionDecember 4, 20034 ADA Strategy (cont) Look to common projects for most of the pieces ARDA, GANGA, DIAL, … Share as much as possible with ATLAS production –Also distributed –Similar interfaces and code for bulk and user-level production ADA (ATLAS distributed analysis) must identify these pieces and tie them together Deployment ADA services must be deployed at relevant sites Provide testing and monitoring of these services Work with facilities to deploy and maintain –Also to develop facility-specific features

5. David Adams ATLAS DIAL/ADA JDL ATLAS SW – Prod sessionDecember 4, 20035 DIAL JDL High-level JDL DIAL envisions a hierarchy of schedulers Interface to these schedulers constitutes a high-level JDL (job definition language) –Job submission, monitoring and gathering of results –See figure Would like to standardize this JDL so schedulers can be shared between projects and experiments –See figure Exchanged objects have XML representations

6. David Adams ATLAS DIAL/ADA JDL ATLAS SW – Prod sessionDecember 4, 20036 User Analysis Job 1 Job 2 ApplicationTask Dataset 1 Scheduler 1. Create or locate 2. select3. Create or select 4. select 5. submit(app,tsk,ds) 6. split Dataset Dataset 2 7. create e.g. ROOT e.g. athena Result 9. fill 10. gather Result 9. fill ResultCode Components of DIAL high-level JDL

7. David Adams ATLAS DIAL/ADA JDL ATLAS SW – Prod sessionDecember 4, 20037 DIAL status: sharing via JDL

8. David Adams ATLAS DIAL/ADA JDL ATLAS SW – Prod sessionDecember 4, 20038 Scheduler Web service interface Scheduler class has similar interface See DIAL JDL page for complete interface & WSDL Partial list (last argument is return) –has_application(XML app, bool stat) –add_task(XML app, XML tsk, bool stat) –submit(XML app, XML tsk, XML dst, XML jobid) –job(XML jobid, XML job) Provide clients with similar interface Command line C++ (imported in ROOT) Python (future from GANGA)

9. David Adams ATLAS DIAL/ADA JDL ATLAS SW – Prod sessionDecember 4, 20039 Application Contents Name Version Corresponds to a software package Same name and version Provides two entry points –Build task –Process a dataset and generate a result Specifies dependencies (other packages) Eventually install with package management system –Now applications must be preinstalled

10. David Adams ATLAS DIAL/ADA JDL ATLAS SW – Prod sessionDecember 4, 200310 Task Contents Collection of named files –Embedded text, PFN or LFN Usage Input to task build –Used only by application

11. David Adams ATLAS DIAL/ADA JDL ATLAS SW – Prod sessionDecember 4, 200311 Dataset Contents Depend on type –See following class diagram for existing types Usage User –select from catalog (DSC) –query for content, # events, … System –locate accessible replica (DRC) –split –determine logical files for staging –extract application view (e.g. event collection) And more…

12. David Adams ATLAS DIAL/ADA JDL ATLAS SW – Prod sessionDecember 4, 200312 Dataset classes Single combined ntuple file (input to application) User selects Used for splitting

13. David Adams ATLAS DIAL/ADA JDL ATLAS SW – Prod sessionDecember 4, 200313 Result Content Depends on type Perhaps should be fixed as list of files –as for task Also need collection of results Usage Communicate results –From application to scheduler –From scheduler to user Provides code to merge with another result –This should probably move to a SW package >Then carry package name in result?

14. David Adams ATLAS DIAL/ADA JDL ATLAS SW – Prod sessionDecember 4, 200314 Job Content Application, task, dataset –provenance Status (running, done, failed, …) Time of start, stop and last update Result –May be empty or partial Usage User –Check status –Access partial results

15. David Adams ATLAS DIAL/ADA JDL ATLAS SW – Prod sessionDecember 4, 200315 Catalogs Two categories Repository –Provides access to XML description indexed by ID –Might be XML DB Selection –Enables user to select with query on catalog contents –Might be relational table Grid service interface User access only through grid service interface Same for access from other services? What granularity? –1 svc, 1 svc/catalog, …

16. David Adams ATLAS DIAL/ADA JDL ATLAS SW – Prod sessionDecember 4, 200316 Catalogs (cont) Application Which applications are available Task Repository –Task XML accessible by ID –Part of provenance system Selection catalog? Application-task consistency?

17. David Adams ATLAS DIAL/ADA JDL ATLAS SW – Prod sessionDecember 4, 200317 Catalogs (cont) Dataset Repository Selection catalog –Very important user interface –Only virtual datasets? Replica catalog –Virtual to non-virtual mapping Single file catalog File Replica catalog Result Repository

18. David Adams ATLAS DIAL/ADA JDL ATLAS SW – Prod sessionDecember 4, 200318 Catalogs (cont) Job Repository –Maybe use selection catalog as repository Selection –Contents >Job ID >Application name and version >Task ID >Dataset ID (and type?) >Result ID (and type?) >Parent job ID >Status –Is this the same catalog that feeds the production supervisor?

19. David Adams ATLAS DIAL/ADA JDL ATLAS SW – Prod sessionDecember 4, 200319 Connection to ATLAS production

20. David Adams ATLAS DIAL/ADA JDL ATLAS SW – Prod sessionDecember 4, 200320 Conclusions