1. 1 Virtual EZ Grid AAA/Switch project Nabil Abdennadher HES-SO, hepia (Geneva)

2. 2 Swiss Grid Day, Bern, 26th Nov. 2009 Virtual EZ Grid at a glance AAA/Switch funding Partners: UniGE, USI, UniNE, HES-SO Start/End dates : February 2009 - July 2010 Objectives : Build a sustainable desktop Grid platform based on a Volunteer Computing (VC) middleware Evaluate it in a real world setting with two medical applications

3. 3 Swiss Grid Day, Bern, 26th Nov. 2009 Virtual EZ Grid objectives Infrastructure: build a grid platform with more than 1000 PCs Non dedicated nodes Belong to several institutions. Reliability: manage the volatility of the “worker” nodes Check pointing, migration, restarting, etc. Security: guarantee the security of resources providers Negotiation Model: provide a resource-credit system Applications: two medical packages NeuroWeb: build neural maps extended from brain measurements MedGift: medical image analysis application

4. 4 Swiss Grid Day, Bern, 26th Nov. 2009 PLAN Virtual EZ Grid ingredients JOpera XtremWeb-CH EZ Grid Applications

5. 5 Swiss Grid Day, Bern, 26th Nov. 2009 Virtual EZ Grid ingredients JOpera (developed at USI) A grid workflow management system: http://www.jopera.orghttp://www.jopera.org XtremWeb-CH (developed at HES-SO) A volunteer computing platform : www.xtremwebch.netwww.xtremwebch.net EZ grid (developed at UniNE and UniGE) Based on virtualization Job check-pointing Migration and restarting ARC

6. 6 Swiss Grid Day, Bern, 26th Nov. 2009 Virtual EZ Grid architecture Infrastructure EZ-Grid XtremWeb-CH (XWCH) JOpera ARC

7. 7 Swiss Grid Day, Bern, 26th Nov. 2009 JOpera “ Provide the scientist with a platform that takes care of all data handling and record keeping chores so that the user can concentrate on the science and not computer science ” Make it easy to build Grid and VC applications composed of multiple jobs Based on slides by Cesare Pautasso

8. 8 Swiss Grid Day, Bern, 26th Nov. 2009 Drag, Drop and Connect Based on slides by Cesare Pautasso

9. 9 Swiss Grid Day, Bern, 26th Nov. 2009 Run, Monitor, Steer and Debug Based on slides by Cesare Pautasso

10. 10 Swiss Grid Day, Bern, 26th Nov. 2009 XtremWeb-CH coordinator Service Request result PC Work Request Work Result Consumer (Users’ applications) Providers (Workers) PC XtremWeb-CH at a glance Job0 Job2 Job1 Job3 Application

11. 11 Swiss Grid Day, Bern, 26th Nov. 2009 XtremWeb-CH coordinator Communication protocol Work Request Worker Work Result Work Alive

12. 12 Swiss Grid Day, Bern, 26th Nov. 2009 User’ servicesJava API C/C++ API Python API Scheduler XWCH DB Work request Work Result Warehouse Worker Work Alive Worker and Warehouse servicesUser application XtremWeb-CH Coordinator Admin. Service

13. 13 Swiss Grid Day, Bern, 26th Nov. 2009 User application (jobs generator) int main () { int Job 0, Job 1, Job 2, Job 3 … Job 0 = addjob (…); while (getJobStatus (Job 0,…)) ≠ COMPLETE); Job0 Job1 = addjob (…); Job2 = addjob (…); while ((getJobStatus (Job1,…)) ≠ COMPLETE) && (getJobStatus (Job2,…)) ≠ COMPLETE)); Job2 Job1 Job3 = addjob (…); while (getJobStatus (Job3,…)) ≠ COMPLETE); getJobFileOut (Job3,…); } Job3

14. 14 Swiss Grid Day, Bern, 26th Nov. 2009 User application: waves generator

15. 15 Swiss Grid Day, Bern, 26th Nov. 2009 EZ Grid Isolates XWCH jobs from “local” jobs Why?: guarantee the privacy of the providers How?: use virtualization technology Supports check-point, and migration Why?: manage the volatility of nodes How?: remotely monitor the XWCH virtual machine

16. 16 Swiss Grid Day, Bern, 26th Nov. 2009 Work request Work Result Warehouse Worker Work Alive XWCH coordinator OS (Windows) VM Manager OS (Linux) XWCH Application EZ Grid module OS (Linux) XWCH Application Local application XWCH worker XWCH VM1XWCH VM2 EZ Grid coordinator

17. 17 Swiss Grid Day, Bern, 26th Nov. 2009 XWCH nodes Coordinator node Scheduling : tasks pre-assignment Tasks management Workers management Worker node Receives data from warehouses/workers Execute jobs Save data on warehouses Warehouse node Data repository Client node User application

18. 18 Swiss Grid Day, Bern, 26th Nov. 2009 PLAN Virtual EZ Grid ingredients JOpera XtremWeb-CH EZ Grid Applications

19. 19 Swiss Grid Day, Bern, 26th Nov. 2009 Applications NeuroWeb : build neuronal maps extracted from brain measurements MedGift : medical image analysis and retrieval application

20. 20 Swiss Grid Day, Bern, 26th Nov. 2009 NeuroWeb Objective Reconstruction of the electromagnetic brain map Which neuron is responsible of what? — Epileptic crisis (*) — Parkinson — Alzheimer — Etc. Why? Avoid invasive surgeries

21. 21 Swiss Grid Day, Bern, 26th Nov. 2009 Neuronal activity MEG Scanner # Sensors = 256 dt = 1 millisecond MEG scanner Magneto-EncephaloGraphy (MEG) scanner: provides temporal information (functional data) Based on slides by Cédric Bilat

22. 22 Swiss Grid Day, Bern, 26th Nov. 2009 ? NeuroWeb # of sensors: 256 A large scale optimization problem # of neurons: 60’000 Based on slides by Cédric Bilat

23. 23 Swiss Grid Day, Bern, 26th Nov. 2009 Mapping of the MEG signals on voxels (IRM): IRM MEG How does the system work? Based on slides by Cédric Bilat

24. 24 Swiss Grid Day, Bern, 26th Nov. 2009 Start with a random matrix A 0 In each step, calculate A i. A i = F (A i-1, functional data, anatomic data) Stop when A i = A i-1 ~16 Mb ~360 Mb The Algorithm

25. 25 Swiss Grid Day, Bern, 26th Nov. 2009.................. … A0A0 A1A1 A i-1 Gridification AiAi

26. 26 Swiss Grid Day, Bern, 26th Nov. 2009 HD XWCH Worker node Main Memory Data persistence Persistent server XWCH Job Why ? Avoid loading/storing data from/to the HD How ? Data remain in main memory even after the end of the task

27. 27 Swiss Grid Day, Bern, 26th Nov. 2009 Virtual EZ Grid today Infrastructure: Around 500 workers — HES-SO (Geneva + Yverdon): ~250 — UNiGe: ~ 200 — UniNe: ~ 20 — Univ. Franche Comté (France): ~30 Reliability : The implementation of EZ Grid is in progress (50%) Security : Fully operational. based on SWITCHaai: Shibboleth-based AAI Negotiation Model: In progress (30%) Applications NeuroWeb: a prototype (proof-of-concept) is already deployed MedGift: gridification will start soon (January 2010)

28. 28 Swiss Grid Day, Bern, 26th Nov. 2009 Swiss Grid Portal Virtual EZ-Grid Swiss Multi-Science Computing Grid Infrastructure Virtual EZ Grid : Links with other projects

29. 29 Swiss Grid Day, Bern, 26th Nov. 2009 Model 1 Prices are fixed by a “central” agency. All the “institutions” receives initial credit which: depends on the quantity of resources provided by the institution can be used against the usage. The client (from a given institution) chooses the priority of jobs. The goal is to: Minimize the cost of execution Execute the application ASAP The model determines the “best” prices which optimize the use of the platform

30. 30 Swiss Grid Day, Bern, 26th Nov. 2009 Model 2 XWCH coordinator Worker (1) Proposed price (2) feedback (3) Price updating