Breaking the frontiers of the Grid R. Graciani EGI TF 2012

Outline  Introduction  The DIRAC Interware  DIRAC as a Framework/Service  Summary Prague, Sep 20th 20122EGI TF 2012

Distributed Computing  Distributed Computing is about splitting up a Computing Task into parts that run on multiple computers communicating over a network.  The main goal of a Distributed Computing system is to connect users to resources in a transparent, open, and scalable way. Prague, Sep 20th 20123EGI TF 2012

Grid proven to work  Discovery of the Higgs boson (July 2012).  A collaborative work of thousands of physicists, sharing resources on hundreds of resource centers.  Results available shortly after the experiments collected the data.  Key role of Grid on this success. Prague, Sep 20th 20124EGI TF 2012

Known issues  End user:  Complicated interfaces for non-experts  Large communities:  Many heterogeneous resources  Intra-community policies  Other communities:  Lack of support for collective work  Resource Providers:  Large: avoid specific configurations and services  Small: reduce overhead of a full grid installation Prague, Sep 20th 20125EGI TF 2012

The DIRAC Interware Prague, Sep 20th 20126EGI TF 2012

DIRAC Solution Prague, Sep 20th 2012EGI TF  LHC experiments developed their own middleware to address the above problems  DIRAC was developed originally for the LHCb experiment  DIRAC is providing a complete interware solution with the goal:  Integrate all the heterogeneous computing resources available  Provides both WMS and DMS  Make the grid convenient for the users:  Single interface  Allowing interoperability  Enabling Community policies  Minimize human intervention at sites Grid A Grid B (WLCG) (NDG) Grid A Cloud B

Workload Management Prague, Sep 20th 2012EGI TF  Jobs are submitted to the DIRAC Central Task Queue with credentials of their owner  Pilot Jobs are submitted by specific Directors to computing resources using appropriated credentials  The Pilot Job fetches the user job and the job owner’s proxy  The User Job is executed with its owner’s proxy used to access SE, catalogs, etc

Physicist User EGEE Pilot Director EGEE Grid NDG Pilot Director NDG Grid Cloud VM Director Cloud CREAM Pilot Director CREAM CE Matcher Service Production Manager Prague, Sep 20th 2012EGI TF 20129

Data Management Components Prague, Sep 20th 2012EGI TF  Storage Elements  gLite/EGI Storage Elements (SRM/gsiftp)  DIRAC Storage Elements  More Storage Elements can be included  (F,SF,HT,BBF)TP servers  File Catalogs  LCG File Catalog (LFC)  DIRAC File Catalog  Support for the User Metadata  Support for data provenance  More Catalogs can be included  LHCb has developed several specific catalogs in the same framework

Data Management Prague, Sep 20th 2012EGI TF  Asynchronous data operations  transfers, registration, removal  Two complementary replication mechanisms  Transfer Agent  user data  public network  FTS service  Production data  Private FTS OPN network  Smart pluggable replication strategies

User Interfaces  Unix command line interface  Historically the first one  Suitable for unix addicts  Easy to use in shell scripts  Too many commands  Even more switches  Python API  Originally for DIRAC developers  More users are starting to use it  The most versatile  Build your own DIRAC applications Prague, Sep 20th EGI TF 2012

And Web Portal Prague, Sep 20th 2012EGI TF

Integration with other tools  DIRAC uses a private API for the secure connection of all its components  A prototype RESTful API has been implemented  Tested to integrate with Gateways or mobile applications  Currently working on the production version  Based on OAuth v2 Prague, Sep 20th EGI TF 2012

It is a Framework  Other necessary components are integrated :  Information system  User / Resource management  Accounting  Monitoring  Can be easily extended  Not all use cases will be covered out of the box  Develop new services, agents, portal pages, scripts,…  May be incorporated to Core functionality  DIRAC is now an open source project   Prague, Sep 20th EGI TF 2012

DIRAC as a Framework/Service Prague, Sep 20th EGI TF 2012

Large communities  Dedicated installations  Several large communities are using DIRAC to implement their computing model  LHCb (CERN)  ILC (CERN)  Belle II (KEK, Japan)  BES III (IHEP, China)  CTA  …  Use exiting functionality, develop ad-hoc extensions, and contribute new functionality Prague, Sep 20th EGI TF 2012

LHCb, the main user Prague, Sep 20th EGI TF 2012

ILC (CERN) Prague, Sep 20th EGI TF 2012

Belle II, combines EGI-OSG Prague, Sep 20th EGI TF 2012

CTA, beyond HEP Prague, Sep 20th EGI TF 2012

BES, building their “Grid” Prague, Sep 20th EGI TF 2012

Why?  Make extensive usage of tools for automated manipulation of large numbers of similar jobs  Simulations  Data processing (workflows / pipelines)  Deep interest on integrated DIRAC File Catalog solution  Replica catalog  Metadata catalog  Provenance catalog  Push development of new interfaces  Interfaces to different batch systems  Combination of different resources Prague, Sep 20th EGI TF 2012

Usage by small communities  Several communities can share a single DIRAC installation operated by a service provider  Currently organized at NGI or Grid level  IN2P3  U. Barcelona  GISELA Latin America Grid  DIRAC provided as a service  Little customization on the installation  Emphasis on dissemination and training Prague, Sep 20th EGI TF 2012

France-Grilles Prague, Sep 20th EGI TF 2012

IberGrid Prague, Sep 20th EGI TF 2012

Summary Prague, Sep 20th 2012EGI TF  DIRAC is an open source software project providing a production- ready solution for distributed computing.  Integrates Grids, Clouds and Clusters, different storage solutions, providing your users with a single interface to all of them.  Ten year HEP experience using the Grid is now available to any interested community.  New functionality and integration of new resource types is added following user needs.  Working to provide a RESTful interface to connect DIRAC with your preferred workflow or application portal solution.

Questions? Prague, Sep 20th 2012EGI TF

Scalability test Prague, Sep 20th EGI TF 2012