1. Breaking the frontiers of the Grid R. Graciani EGI TF 2012

2. Outline  Introduction  The DIRAC Interware  DIRAC as a Framework/Service  Summary Prague, Sep 20th 20122EGI TF 2012

3. Distributed Computing  Distributed Computing is about splitting up a Computing Task into parts that run on multiple computers communicating over a network.  The main goal of a Distributed Computing system is to connect users to resources in a transparent, open, and scalable way. Prague, Sep 20th 20123EGI TF 2012

4. Grid proven to work  Discovery of the Higgs boson (July 2012).  A collaborative work of thousands of physicists, sharing resources on hundreds of resource centers.  Results available shortly after the experiments collected the data.  Key role of Grid on this success. Prague, Sep 20th 20124EGI TF 2012

5. Known issues  End user:  Complicated interfaces for non-experts  Large communities:  Many heterogeneous resources  Intra-community policies  Other communities:  Lack of support for collective work  Resource Providers:  Large: avoid specific configurations and services  Small: reduce overhead of a full grid installation Prague, Sep 20th 20125EGI TF 2012

6. The DIRAC Interware Prague, Sep 20th 20126EGI TF 2012

7. DIRAC Solution Prague, Sep 20th 2012EGI TF 20127  LHC experiments developed their own middleware to address the above problems  DIRAC was developed originally for the LHCb experiment  DIRAC is providing a complete interware solution with the goal:  Integrate all the heterogeneous computing resources available  Provides both WMS and DMS  Make the grid convenient for the users:  Single interface  Allowing interoperability  Enabling Community policies  Minimize human intervention at sites Grid A Grid B (WLCG) (NDG) Grid A Cloud B

8. Workload Management Prague, Sep 20th 2012EGI TF 2012 8  Jobs are submitted to the DIRAC Central Task Queue with credentials of their owner  Pilot Jobs are submitted by specific Directors to computing resources using appropriated credentials  The Pilot Job fetches the user job and the job owner’s proxy  The User Job is executed with its owner’s proxy used to access SE, catalogs, etc

9. Physicist User EGEE Pilot Director EGEE Grid NDG Pilot Director NDG Grid Cloud VM Director Cloud CREAM Pilot Director CREAM CE Matcher Service Production Manager Prague, Sep 20th 2012EGI TF 20129

10. Data Management Components Prague, Sep 20th 2012EGI TF 201210  Storage Elements  gLite/EGI Storage Elements (SRM/gsiftp)  DIRAC Storage Elements  More Storage Elements can be included  (F,SF,HT,BBF)TP servers  File Catalogs  LCG File Catalog (LFC)  DIRAC File Catalog  Support for the User Metadata  Support for data provenance  More Catalogs can be included  LHCb has developed several specific catalogs in the same framework

11. Data Management Prague, Sep 20th 2012EGI TF 201211  Asynchronous data operations  transfers, registration, removal  Two complementary replication mechanisms  Transfer Agent  user data  public network  FTS service  Production data  Private FTS OPN network  Smart pluggable replication strategies

12. User Interfaces  Unix command line interface  Historically the first one  Suitable for unix addicts  Easy to use in shell scripts  Too many commands  Even more switches  Python API  Originally for DIRAC developers  More users are starting to use it  The most versatile  Build your own DIRAC applications Prague, Sep 20th 201212EGI TF 2012

13. And Web Portal Prague, Sep 20th 2012EGI TF 2012 13

14. Integration with other tools  DIRAC uses a private API for the secure connection of all its components  A prototype RESTful API has been implemented  Tested to integrate with Gateways or mobile applications  Currently working on the production version  Based on OAuth v2 Prague, Sep 20th 201214EGI TF 2012

15. It is a Framework  Other necessary components are integrated :  Information system  User / Resource management  Accounting  Monitoring  Can be easily extended  Not all use cases will be covered out of the box  Develop new services, agents, portal pages, scripts,…  May be incorporated to Core functionality  DIRAC is now an open source project  http://diracgrid.org/  https://github.com/DIRACGrid Prague, Sep 20th 201215EGI TF 2012

16. DIRAC as a Framework/Service Prague, Sep 20th 201216EGI TF 2012

17. Large communities  Dedicated installations  Several large communities are using DIRAC to implement their computing model  LHCb (CERN)  ILC (CERN)  Belle II (KEK, Japan)  BES III (IHEP, China)  CTA  …  Use exiting functionality, develop ad-hoc extensions, and contribute new functionality Prague, Sep 20th 201217EGI TF 2012

18. LHCb, the main user Prague, Sep 20th 201218EGI TF 2012

19. ILC (CERN) Prague, Sep 20th 201219EGI TF 2012

20. Belle II, combines EGI-OSG Prague, Sep 20th 201220EGI TF 2012

21. CTA, beyond HEP Prague, Sep 20th 201221EGI TF 2012

22. BES, building their “Grid” Prague, Sep 20th 201222EGI TF 2012

23. Why?  Make extensive usage of tools for automated manipulation of large numbers of similar jobs  Simulations  Data processing (workflows / pipelines)  Deep interest on integrated DIRAC File Catalog solution  Replica catalog  Metadata catalog  Provenance catalog  Push development of new interfaces  Interfaces to different batch systems  Combination of different resources Prague, Sep 20th 201223EGI TF 2012

24. Usage by small communities  Several communities can share a single DIRAC installation operated by a service provider  Currently organized at NGI or Grid level  France-Grilles @ IN2P3  IberGrid @ U. Barcelona  GISELA Latin America Grid  DIRAC provided as a service  Little customization on the installation  Emphasis on dissemination and training Prague, Sep 20th 201224EGI TF 2012

25. France-Grilles Prague, Sep 20th 201225EGI TF 2012

26. IberGrid Prague, Sep 20th 201226EGI TF 2012

27. Summary Prague, Sep 20th 2012EGI TF 201227  DIRAC is an open source software project providing a production- ready solution for distributed computing.  Integrates Grids, Clouds and Clusters, different storage solutions, providing your users with a single interface to all of them.  Ten year HEP experience using the Grid is now available to any interested community.  New functionality and integration of new resource types is added following user needs.  Working to provide a RESTful interface to connect DIRAC with your preferred workflow or application portal solution. http://diracgrid.org

28. Questions? Prague, Sep 20th 2012EGI TF 201228

29. Scalability test Prague, Sep 20th 201229EGI TF 2012