11 The European Desktop Grid Federation: status of the infrastructure and integration plans Peter Kacsuk, Jozsef Kovacs, and Robert Lovas (MTA SZTAKI) Start date: Duration: 24 months EDGI is supported by the FP7 Capacities Programme under contract nr RI

2 History of EDGF: EDGeS – EDGI – DEGISCO – EDGF relationship EDGI ARC, Unicore, Clouds ARC, Unicore, Clouds QoS with Clouds QoS with Clouds Data intensive apps Data intensive apps SG  DG direction support SG  DG direction support EDGeS gLite  BOINC, XtremWeb gLite  BOINC, XtremWeb BOINC, XtremWeb  gLite BOINC, XtremWeb  gLite Compute intensive apps Compute intensive apps further d eveloped by supported by EDGI – DEGISCO common interest: To create and extend DG community in Europe and worldwide. Establishment of the International Desktop Grid Federation with European Chapter DEGISCO Disseminate EDGeS results worldwide (ICPC partners) Disseminate EDGeS results worldwide (ICPC partners) DG  SG direction support DG  SG direction support

33 EDGeS Production DG  EGEE: Infrastructure maintained by DEGISCO VOMS LBWMSBDII gLite based EDGeS desktopgrid VO BOINC-EGEE bridge (ui1) XtremWeb-EGEE bridge (xw2) BOINC-based DGs UoW, Correlation Systems local DGs SZDG, Almere, Ibercivis public DGs XtremWeb- based DGs IN2P3, Almere local DGs IN2P3 public DGs SZTAKI-16 CPUs AR UI/ Bridge Core Service Resource CIEMAT-20 CPUs CE CNRS-1800 CPUs CE LFC SE myproxy ASGC- 6 CPUs CE LIP-1080 CPUs CE BIFI-36 CPUs CE IPB-672 CPUs CE UFCG- 6 CPUs CE KFKI- 410CPUs CE

4 Focus of EDGI 4 Academic Cloud Supercomputer based SGs (DEISA, TeraGrid) Cluster based service grids (SGs) (EGEE, OSG, etc.) Desktop grids (DGs) (volunteer DGs – home PCs, organisational DGs – institutional desktops) OGF PGI EDGeS/EDGI EDGI EDGeS EDGI Academic cloud

5 Main objectives of EDGI 1. To develop bridge middleware to extend ARC-, gLite and Unicore-based Service Grids (SGs) with Desktop Grids (DGs) 2. To develop DG  Cloud bridge middleware in order to get instantly available additional resources on demand when QoS requirements need it 3. To improve Desktop Grid middleware in order to handle QoS requirements 4. To improve the SG  DG bridge middleware in order to support data-intensive applications 5. To consolidate the existing EDGeS production infrastructure and further develop it towards the more comprehensive EDGI production infrastructure 6. To create a European Desktop Grid Federation

6 Overview of the planned EDGI Infrastructure EDGeS scope only for compute intensive applications for EGEE (gLite) EDGI scope for both compute and data intensive applications for EMI/EGI (gLite, ARC, Unicore) Extends Desktop Grids with Clouds for QoS

7 Support to Extend NGI Infrastructures with Institutional DGs Institutional DGs are organized from the desktop resources of a single organization (e.g. university, research institute) EDGeS/EDGI technology will enable to extend gLite, ARC, Unicore VOs with such institutional DGs Consequence: the number and capacity of computing resources can be significantly increased by limited investment

8 University DGs in practice – University of Westminster as an example New Cavendish Street576 nodes 2.Marylebone Campus559 nodes 3.Regent Street395 nodes 4.Wells Street31 nodes 5.Little Tichfield Street66 nodes 6.Harrow Campus254 nodes Total:1881 nodes Lifecycle of a node: 1.PCs basically used by students/staff 2.If unused, switch to Desktop Grid mode 3.No more work from DG server -> shutdown (green solution)

9 EDGI resources are available for all EGI user communities who intend to utilize Desktop Grid resources EDGI applications should run on both EGI and DG resources EGI supports a much wider scale of applications than DGs We should assure that the application runs on DGs Virtual Research Communities using EGI and EDGI resources

10 Which EGI applications are suitable for EDGI? Requirements towards an EGI application to be executable on DGs: Parallelization: Only master/worker, parameter sweep and similar parallelisation No MPI or direct communication between worker nodes Nodes can only use the results of other nodes through the server Data handling: Small or medium-sized (max. 100 MB per worker) inputs and outputs No shared data storage No confidential data (sent down to potentially un-trusted worker) Operating systems Depends on the DGs where the application will run windows version may be required to utilise larger number of resources

11 EGI user communities using EDGI ISDEP - Integrator of Stochastic Differential Equations in Plasmas Represented by BIFI as sub-contractor Used in EGEE before EDGeS/EDGI The EDGeS work: production quality BOINC implementation deployed on the Ibercivis and Desktop Grids. Application runs through the BOINC  EGEE bridge allowing Ibercivis users to utilise resources from the EGEE EDGeS VO. Application runs through the EGEE  BOINC bridge allowing EGEE fusion VO users to utilise resources of

12 EGI user communities using EDGI VisIVO - Visualisation Interface to the Virtual Observatory Application porting Desktop Grid version is deployed and running on UoW Local DG Application runs in DG to EGEE direction from UoW Local DG to EGEE EDGeS VO Application is validated and published in EDGeS Production Application Repository Application runs from EGEE to DG (UoW Local DG and Data distribution –Medium sized input files (up to 100 Mbytes) are currently feasible –Division of input file (potentially GBytes) and better data distribution using ADICS will be investigated in EDGI

13 EGI user communities using EDGI WISDOM Production Environment Meta middleware to submit pull (pilot) jobs to EGEE Millions of completed jobs (Wisdom II 142 millions jobs) Wide range of applications: Docking, Profiling etc.. The EDGeS extension: Submit push (direct) jobs to the DG when EGEE resources are overloaded Successfully prototyped with AutoDock jobs – collaboration will continue in EDGI

14 EGI user communities using EDGI Search for new user communities Invitation based call for user community in EDGI To join EDGI as sub-contractor and port application to joint EGI-EDGI platform Support for all EGI user communities who intend to port to and run their applications on the EDGI platform The EDGI Application Support Service provides help and expertise in porting applications to DG platforms and to EDGI

15 European Desktop Grid Federation Main goals Main goals promoting Desktop Grid technologies promoting Desktop Grid technologies advancing Desktop Grid technologies advancing Desktop Grid technologies advancing the usage of Desktop Grid technologies advancing the usage of Desktop Grid technologies The European Desktop Grid Federation will bring together The European Desktop Grid Federation will bring together organisations (institutes, companies, universities) that operate Desktop Grids or infrastructures that incorporate Desktop Grids organisations (institutes, companies, universities) that operate Desktop Grids or infrastructures that incorporate Desktop Grids organisations or groups that run and develop applications on these infrastructures organisations or groups that run and develop applications on these infrastructures EDGF is setup to be long-lived, i.e. after the project lifetime EDGF is setup to be long-lived, i.e. after the project lifetime EDGF will provide EDGF will provide Member administration, News services, Discussion fora Member administration, News services, Discussion fora Support services Support services Dissemination and Organising meetings/events Dissemination and Organising meetings/events EDGF is planned to be established in the first half year of the project EDGF is planned to be established in the first half year of the project

16 Main planned collaborations EGI EGI – organizing grid activities in Europe – dissemination desktop grid extension know-how for grid systems desktop grid extension know-how for grid systems available EDGI applications available EDGI applications – publishing EDGI applications into the EGI application repository EDGI applications into the EGI application repository publishing EDGI middleware into the EGI repository (UMD) publishing EDGI middleware into the EGI repository (UMD) – support for the EGI user communities: application porting for the EGI user communities: application porting NGIs NGIs – extending NGI infrastructures with EDGI services/middleware – supporting NGI operators using EDGI services/middleware – supporting NGI user communities EMI EMI – middleware development for maintaining the EDGI bridge for ARC, gLite, Unicore

17 Other possible collaborations IGE IGE – explore the possibility of developing a Globus DG bridge – potential user: UK NGS StratusLab StratusLab – Cloud API – QoS (scheduler) VENUS-C VENUS-C – Cloud API More collaborations are under investigation…

18 Thank you for your attention… Contacts: Peter Kacsuk (coordinator of EDGI, EDGeS) Jozsef Kovacs (deputy coordinator, technical leader) Websites: Acknowledgements:  EDGI EU FP7 project (RI )  EDGeS EU FP7 project (INFSO-RI ) 18