1. DIRAC for Grid and Cloud Dr. Víctor Méndez Muñoz (for DIRAC Project) LHCb Tier 1 Liaison at PIC EGI User Community Board, October 31st, 2013

2. Introduction 2  DIRAC Project provides all necessary components to build ad-hoc computing systems  integrating distributed resources  simplifying interfaces  ensuring interoperability 10/31/13DIRAC for Grid and Cloud

3. Introduction 310/31/13DIRAC for Grid and Cloud

4. Technology Delivery: Grid 4  Aggregation of different resource providers  gLite: EGI  VDT: OSG  ARC: NDGF sites, RAL, etc. 10/31/13DIRAC for Grid and Cloud

5. Technology Delivery: Cloud 5  Resource allocation policies  Elastic VM launching driven by workload  Static VM launching for dedicated resources  VMs are stopped when no longer needed  Aggregation of resource providers  Amazon, OCCI, OpenNebula, OpenStack, CloudStack  Golden images with dynamic context  Preconfigured ad-hoc images  Fully interoperable with Grid resources 10/31/13DIRAC for Grid and Cloud

6. Clusters Volunteer Computing 6  Torque  Condor  OAR  SLURM  LSF  EDGI  BOINC Technology Delivery 10/31/13DIRAC for Grid and Cloud

7. Technology Delivery: Data 7  Aggregation of storage resources  Access to standard grid storage elements  Vanilla DIRAC storage element  Integration of multiple data catalogs  LFC (replica)  DIRAC File Catalog (replica & metadata) 10/31/13DIRAC for Grid and Cloud

8. Web Portal 8 Launchpad Proxy Upload Certificate Authentication Job&Pilot Browsering Job&Pilot Monitor Configuration Browsering Accounting … or Admin 10/31/13DIRAC for Grid and Cloud

9. Available Services 9  Security infrastructure  Digital certificate authentication  Single and Multi-Community portal  Intra-Community groups and users policies  Resource usage Accounting for providers, communities, groups and users  CPU, Jobs, WallTime  Storage  Transfers 10/31/13DIRAC for Grid and Cloud

10. Available Services 10  Workload Management with Pilot Jobs  Technology proven in the HEP experiments  Bulk submissions with parametric jobs  MPI Jobs support  Automated data driven workflows 10/31/13DIRAC for Grid and Cloud

11. Interfaces 11  For users  Web portal  Command line tools  For advanced users  Python API  RESTful interface  Science Gateways  Mobile Apps 10/31/13DIRAC for Grid and Cloud

12. The Current Actors LHCb/WLCG Clouds EGI Fedcloud FG Clouds 1210/31/13DIRAC for Grid and Cloud

13. Human Teams User Communities Scientific Director & Liaisons DIRAC Admins DIRAC Developers Resource Providers 13 User Communities Resource Providers 10/31/13DIRAC for Grid and Cloud

14. Proven Scalability 14  500 10/31/13DIRAC for Grid and Cloud

15. Proven Scalability 15  1.500 10/31/13DIRAC for Grid and Cloud

16. Proven Scalability 16  1.500 10/31/13DIRAC for Grid and Cloud

17. Proven Scalability 17  2.500 10/31/13DIRAC for Grid and Cloud

18. Proven Scalability 18  20.000 10/31/13DIRAC for Grid and Cloud

19. France-Grilles 19  DIRAC services are provided by several National Grid Initiatives: France, Spain, Italy, UK …  Example: France-Grilles DIRAC service  Hosted by the CC/IN2P3  Distributed administrator team  5 participating universities  15 VOs, ~100 registered users  In production since May 2012 10/31/13DIRAC for Grid and Cloud

20. 20 DIRAC & Clouds VM scheduler with pull model job brokering WMS Task Queue Matcher Sandboxes Job Manager VM Scheduler Web API VM Cloud B Job Agent VM Cloud A Job Agent VM Cloud C Job Agent EC2 ONE OCCI 2010: Belle with Amazon 2013: LHCb with PIC and CERN 2013: Belle II with Cracow and Zurich Use cases in EGI Fedcloud 10/31/13DIRAC for Grid and Cloud

21. 21 DIRAC & Clouds VM scheduler with third party job broker EC2 ONE OCCI WMS Topos server Sandboxes VM Scheduler Web VM Cloud B Payload VM Cloud A Payload VM Cloud C Payload Validated in EGI Fedcloud 10/31/13DIRAC for Grid and Cloud

22. 22 DIRAC & Clouds VM Scheduler, job broker and federated services WMS Task Queue Matcher Sandboxes Job Manager VM Scheduler Web API VM Job Agent Cloud B VM Gmond slave VM Gmond slave VM Job Agent Cloud A Community VMs Monitoring Resource Provider VMs Monitoring VM Gmond master VM Gmond master OCCI EC2 VM Gmond slave VM Gmond slave VM Gmond master VM Gmond master Web 2013: LHCb with PIC and CERN 10/31/13DIRAC for Grid and Cloud

23. EGI Multi-Community DIRAC Service 23  What?  Deploy a EGI-wide multi-Community DIRAC service providing integrated access to Grid and Cloud  Requirements  Manpower: Scientific Director, Liaisons with User Communities, Liaisons with Resource Providers, DIRAC Admins  DIRAC Hosting Services: adequate sysadmin and service with high availability and redundancy  Who?  EGI User Support & EGI Operation Teams  DIRAC project  Hosting Service Providers (Main server + redundancy servers)  When?  Soon (weeks) after requirements are met 10/31/13DIRAC for Grid and Cloud

24. Conclusions 24  DIRAC interware provides a robust framework for building ad-hoc computing systems  DIRAC interware incorporates more than 10 years of know-how  DIRAC interware helps users to get started in the world of distributed computing and to discover its full potential  Are you interested on a EGI DIRAC installation for Grid and Cloud? 10/31/13DIRAC for Grid and Cloud