1. The EDGI project receives Community research funding 1 EDGI Brings Desktop Grids To Distributed Computing Interoperability Etienne URBAH urbah@lal.in2p3.fr OGF29 Chicagourbah@lal.in2p3.fr LAL, Univ Paris-Sud, IN2P3/CNRS, Orsay, France

2. EDGI Bringing Desktop Grids To Distributed Computing Interoperability Author : E. Urbah v1.0 2 EDGI project : Context –EGEE = Service Grid focused on Secure Data Sharing –DG = Desktop Grid = Loose opportunistic grid using idle resources –EDGeS project : Presentation, 3G Bridge, Infrastructure, Gflop/s EDGI project : Goal = Grid + Cloud –Bridging Security and Info –Useful Official and De facto Standards –Interfaces : Diagrams – Scope of JSDL + BES Summary

3. EDGI Bringing Desktop Grids To Distributed Computing Interoperability Author : E. Urbah v1.0 3 EDGI project : Context Academic Clouds OGF PGI EDGeS EDGI Cluster based service grids (SGs) (EGEE, NDGF,...) Supercomputer based SGs (DEISA,...) Desktop grids (DGs) (Volunteer DGs – Home computers, Organizational DGs – Institutional desktops) Achievements of EDGeS Modular design of 3G Bridge with Application Repository, permitting integration between various DGs and SGs 3G Bridges in full production for SGs implementing gLite : –BOINC and XWHEP EGEE, –OurGrid EELA2. ATTIC = Proof of Concept of Data Distribution

4. EDGI Bringing Desktop Grids To Distributed Computing Interoperability Author : E. Urbah v1.0 4 EGEE = Service Grid focused on Secure Data Sharing Grid User  X509 proxy  VOMS Extensions VOMS Admin Manages VO Site Admin Manages Site  Accesses Data with VOMS proxy VOMS Server Site Storage Resource VOMS Proxy = X509 proxy with VOMS extensions Trust Anchor = IGTF Publishes the Root CA Certificates Interoperability with another Grid Infrastructure is possible only if it uses IGTF as Trust Anchor

5. EDGI Bringing Desktop Grids To Distributed Computing Interoperability Author : E. Urbah v1.0 5 EGEE = Service Grid focused on Secure Data Sharing Grid User  X509 proxy  VOMS Extensions  Submits Job with VOMS proxy Publishes available Resources  Pushes Job   Log  Log VOMS Admin Manages VO Site Admin Manages Site   Accesses Data with VOMS proxy  Accesses Data with VOMS proxy   Gives Job Status Gives Accounting and Auditing VOMS Server Accounting Logging & Bookkeeping Meta-scheduler (WMS) Site Computing Resource Site Storage Resource Grid Admin  Sends back Output Sandbox  Sends back Output Sandbox VOMS Proxy = X509 proxy with VOMS extensions

6. EDGI Bringing Desktop Grids To Distributed Computing Interoperability Author : E. Urbah v1.0 6 DG = Desktop Grid = Loose opportunistic grid using idle resources WU = Work Unit = Application + Input Data Grid User  Submits input data for an application  Requests WU  Sends WU Application Manager  Certifies Application Resource Owner (often volunteer)  Owns Resource  Sends back results  Accepts or Refuses an application on his resource Grid Server with Application Repository Computing Resource (often Desktop Computer)  Sends back results Currently, for BOINC, both roles of ‘Application Manager’ and ‘Grid User’ are fulfilled by ‘BOINC Project Owners’.

7. EDGI Bringing Desktop Grids To Distributed Computing Interoperability Author : E. Urbah v1.0 7 DG = Desktop Grid = Loose opportunistic grid using idle resources Computing and Storage Resources are owned by various Owners (it is often volunteer computing), but they are NOT managed and NOT authenticated. Grid Servers are authenticated by a X509 certificate. Users are authenticated by the Grid Servers, but NOT by the Computing and Storage Resources. Executables are certified by managers of the Grid Servers. So :– Resource Owners have to trust the Grid Servers, – BOINC sends each Work Unit to several Resource Owners, because BOINC does NOT fully trust them. Order of magnitude can be 1 000 000 CPUs. Starving Computing Resources pull Work Units from Grid Servers. Examples : BOINC, XtremWeb (XWHEP and XWCH), OurGrid, xGrid

8. EDGI Bringing Desktop Grids To Distributed Computing Interoperability Author : E. Urbah v1.0 8 EDGeS project : Presentation FP7 project started in 2008 and just ended Integrate Service Grids and Desktop Grids Enable very large number of computing resources (100K-1M processors) Attract new scientific communities Provide a Grid application development environment Provide application repository and bridges for the execution in the SG-DG system WLCG (CERN) EDGeS gLite (EGEE, EELA, GridPP) ARC (NDGF) Boinc (Berkeley) XWHEP (INRIA/IN2P3) OurGrid (Brazil) Unicore (DEISA) VDT (OSG) Operational Future Prototype with BES GT4 (NGS) Operational Future

9. EDGI Bringing Desktop Grids To Distributed Computing Interoperability Author : E. Urbah v1.0 9 EDGeS 3G Bridge : Modular Design EDGeS 3G Bridge EGEE Source Grid Handler Interface Job Database Queue Manager Target Grid Plugin Interface BOINC Plugins (DC-API) EGEE Plugins Scheduler Control path WU  Job BOINC Handlers EGEE Handler LCG-CE for EDGeS gLite WMS BOINC Server User Job with VOMS proxy Work Unit BOINC Server gLite WMS WU Job Control path Job  WU Job WU XWHEP Plugins XWHEP Server BES Plugins NDGF DEISA

10. EDGI Bringing Desktop Grids To Distributed Computing Interoperability Author : E. Urbah v1.0 10 EDGeS Infrastructure : Principle of Operation

11. EDGI Bringing Desktop Grids To Distributed Computing Interoperability Author : E. Urbah v1.0 11 EDGeS operational EGEE  DG infrastructure EDGeS VO of EGEE SZTAKI CE 16 cpus CNRS / IN2P3 CE 1.600 cpus CIEMAT CE 20 cpus BDIIVOMS MyProxy WMSLB EGEE Users EDGeS BOINC  EGEE bridge EGEE  XWHEP bridge BOINC-based Desktop Grids SZDG (public) 80.000 PCs IberCivis (public) 28.000 PCs AlmereGrid (public) 1.800 PCs UoW (local) 3.100 PCs EDGeS@ home (public) 3.300 PCs BOINC Project Owners XWHEP-based Desktop Grids LAL (local) 80 PCs IN2P3 (public) 30 PCs AlmereGrid (local) 10 PCs XWHEP Users EGEE  BOINC bridge Application Repository XWHEP Agent

12. EDGI Bringing Desktop Grids To Distributed Computing Interoperability Author : E. Urbah v1.0 12 EDGeS@Home – GFLOP/s (7 days)

13. EDGI Bringing Desktop Grids To Distributed Computing Interoperability Author : E. Urbah v1.0 13 EDGI project : Goals Academic Clouds OGF PGI EDGeS EDGI Cluster based service grids (SGs) (EGEE, NDGF,...) Supercomputer based SGs (DEISA,...) Desktop grids (DGs) (Volunteer DGs – Home computers, Organizational DGs – Institutional desktops) On the model of IberCivis, creation of a self- sustainable EuroCivis Integration between DGs and SGs implementing BES (ARC, Unicore) Bridge to Academic Clouds for Jobs needing QoS Long term support of the EDGI bridges by the NGIs Secure Data Distribution using ATTIC

14. EDGI Bringing Desktop Grids To Distributed Computing Interoperability Author : E. Urbah v1.0 14 BOINC  EGEE Bridging Security EGEE WMS EDGeS 3G bridge EGEE Plugin 1 for each (BOINC Project Owner, EGEE VO) pair Queue Manager & Job DB BOINC Handler 1 for each (BOINC server, BOINC Project Owner, EGEE VO) triple WU i+1 WU i+2 WU i+3 Job Job i+1 Job i+2 BOINC Server Work Unit BOINC Project Owner Submission MyProxy trusting EDGeS 3G bridge Medium term X509 proxy Config. file DN of X509 proxy Short term X509 proxy VOMS Server VOMS extensions Source Grid Handler InterfaceTarget Grid Plugin Interface BOINC jobwrapper client (simulating a large BOINC computing resource) 3G job- wrapper VOMS proxy Retriever

15. EDGI Bringing Desktop Grids To Distributed Computing Interoperability Author : E. Urbah v1.0 15 BOINC Server EGEE LCG-CE for EDGeS EDGeS 3G bridge EGEE  BOINC Bridging Info Gets EXE Reports resources and performance Checks EXE Adds job Watches job Gets VOMS proxy BOINC Computing Resource BOINC plugin (DC-API) EDGeS Application Repository Information provider GRAM Job Manager for EDGeS EGEE VOMS EGEE User Queue Manager & Job DB Generic Job WS Handler BOINC Service Watches Pushes job Submits Job Logs events EGEE BDII gLite WMS EGEE LB Sends output Gets output Gets Infos

16. EDGI Bringing Desktop Grids To Distributed Computing Interoperability Author : E. Urbah v1.0 16 Grid + Cloud : Useful Official and De facto Standards

17. EDGI Bringing Desktop Grids To Distributed Computing Interoperability Author : E. Urbah v1.0 17 Interfaces for Grid + Cloud : Abstract Diagram

18. EDGI Bringing Desktop Grids To Distributed Computing Interoperability Author : E. Urbah v1.0 18 Interfaces for Grid + Cloud : Detailed Diagram – Scope of JSDL + BES