1. 1 IDGF International Desktop Grid Federation NEW TRENDS IN DESKTOP GRID COMPUTING Mikhail Posypkin Centre for Distributed Computing IITP RAS

2. 2 DEGISCO WP4 24/06/2016 2 PRESENTATION PLAN  Intro to Desktop Grid Computing  Tools and technologies in DG computing  Russian Chapter of IDGF

3. 3 WHAT “DESKTOP GRIDS” ARE  Use free CPU cycles  Volunteer computing  Use master-slave model of distributed computation  Major technologies:  BOINC  XTREMWEB-HEP  OURGRID  CONDOR  X-COM  SARD

4. 4 DESKTOP GRIDS BENEFITS  Low-cost: DGs are leaders in terms of Flops/$  Ease of setup and maintenance  Reuse of existing resources  DGs are “Green”  Restricted area of application: coarse grain master-slave  Complex technology  Low w.r.t. supercomputers efficieincy STRONG POINTSWEAK POINTS NOT a REPLACEMENT for SUPECOMPUTERS/SERVICE GRIDS/ CLOUDS

5. 5 DEGISCO WP4 24/06/2016 5 BOINC: Berkeley Open Infrastructure for Network Computing  BOINC is an open source middleware system for deploying a desktop “grid” based on volunteer computing.  Originally created from the SETI@home project by David Anderson from University of California, Berkeley.  Now has over 22 full scale production projects and dozens of alpha projects exist  Projects are based on different scientific disciplines  BOINC has over 538,354 active computers (hosts) worldwide processing on average 7,777.280 TeraFLOPS TFLOPS (June 28, 2014)

6. 6 BOINC

7. 7 BOINC CREDIT SYSTEM The project's server keeps track of how much work your computer has done. This is called credit. To ensure that credit is granted fairly, most BOINC projects work as follows: Each task may be sent to two computers. When a computer reports a result, it claims a certain amount of credit, based on how much CPU time was used. When at least two results have been returned, the server compares them. If the results agree, then users are granted the smaller of the claimed credits.

8. 8 BOINC Client

9. 9 BOINC Server

10. 10 BOINC App Specific Daemons work generator (creates stream or batches of jobs) assimilator (handles correct result) validator (compares replicas, selects “correct” result) BOINC

11. 11 DEGISCO WP4 24/06/2016 11 BOINC DEVELOPMENT TOOLS  C++ API – BOINC native API – DC-API  WRAPPERS – BOINC Wrapper – GenWrapper  VIRTUALIZED WRAPPERS – BOINC VBox WRAPPER – GBAC

12. 12 BOINC Native API (BOINC team) WU Generator: - work unit generation, submission - work unit generation, submission - input and output file handling - input and output file handlingValidator: - validator frameworks (simple, low-level) - validator frameworks (simple, low-level)Assimilator: - handler function linked with a special program - handler function linked with a special program Initialisation Input and output file identification I/O wrappers Checkpointing Progress, credit reporting Critical sections Atomic file update Time handlers Graphics API (screen savers, icons) Other APIs (status, diagnostics, etc.) Grid System Client Application BOINC API Client Application BOINC API WU GeneratorValidatorAssimilator BOINC API

13. 13 DC-API (SZTAKI team, Hungary) Grid System Client Application DC-API Client Application DC-API Initialisation Input and output file identification Event processing (e.g. checkpointing) Progress reporting Configuration, logging, Common features: Master Application DC-API Initialisation Work unit generation Input and output file handling Work unit submission Event processing (e.g. results) Result assimilation Work unit deletion

14. 14 BOINC Wrapper (BOINC team) BOINC core client BOINC application compiled with the BOINC libraries Wrapper Legacy Application Without the WrapperWith the Wrapper Executes the legacy application(s) in a specific order

15. GenWrapper technology (SZTAKI team) BOINC core client ApplicationApplication Work unit A ZIP file (optional) Launcher GenWrapper binaries Legacy application Profile script Application script Input files 3. Extracts 5. Executes 1. Downloads 2. Downloads Acts as the BOINC client application 4. Launches GitBox Can do almost anything that a shell script can (e.g. can start the legacy application) Output files 7. Produces GBAC

16. 16 GenWrapper technology (SZTAKI team) BOINC core client ApplicationApplication Work unit A ZIP file (optional) Launcher GenWrapper binaries Legacy application Profile script Application script Input files 3. Extracts 5. Executes 1. Downloads 2. Downloads Acts as the BOINC client application 4. Launches GitBox Can do almost anything that a shell script can (e.g. can start the legacy application) Output files 7. Produces GBAC BINARY COMPATIBILITY ISSUE

17. 17 DEGISCO WP4 24/06/2016 17 VBoxWrapper (BOINC team)  Developers work in environment of choice  App consists of – compressed minimal VM image (same across multiple apps) – executable – BOINC “Vbox wrapper”  Volunteers need to install VirtualBox

18. 18 VBoxWrapper BOINC client Vbox wrapper VM instance shared directory: executable input, output files

19. 19 GBAC : Generic BOINC Application Client Developed by SZTAKI ( Hungary )

20. 20 DEGISCO WP4 24/06/2016 20 VBoxWrapper advantages  Advantages – Portability issues solved (no “mininal” linux is needed) – Safety is almost absolute – No special checkpointing is needed  Drawbacks – Virtual Box should be installed by volunteers – Virtual appliances load network traffic

21. BOINC on mobile devices

22. 22 AndroBOINC: Remote control tool for BOINC clients This light-weighted manager for BOINC clients: - can connect to BOINC core client via WiFi or mobile network - displays status of client: projects, tasks, transfers and messages -provides basic controls for client Pavol Michalec https://code.google.com/p/androboinc/

23. 23 NativeBOINC for Android NativeBOINC is Android version of the BOINC client. This software moves BOINC to mobile device's world. Main application is fully-fledged BOINC manager (based on AndroBOINC), which can controls BOINC client on the Android OS. This version provides manager, BOINC client and especially optimized applications for ARM processors. Mateusz Szpakowski (Poland) http://nativeboinc.org/site/uncat/starthttp://nativeboinc.org/site/uncat/start

24. 24 BOINC on Android (by Berkeley team) BOINC harnesses the unused computing power of your Android device and runs jobs for scientific research projects. BOINC computes only when your device is plugged in and charged, so it won't run down your battery. It transfers data over WiFi, so it won't use up your cell phone plan's data limit.

25. 25 DEGISCO WP4 24/06/2016 25 Interoperability among Desktop Grids and other HPC resources  System-level interoperability – 3G-Bridge – CluboRUN  Application-level interoperability – gUSE/WS-PGRADE

26. 26 3G-BRIDGE Service grid or Cloud BOINC

27. 27 DEGISCO WP4 24/06/2016 27 CluboRUN: use idle supercomputer cycles  Supercomputers also have ‘idle cycles’ – free rooms in a queue  Periodic script queries the queue and submits BOINC jobs to cluster if there is a room for it  MPI wrapper for BOINC jobs is used to run multiple clients on the cluster IDSTU RAS, IITP RAS, volunteers

28. WS-PGRADE ARCHITECTURE (SZTAKI) Graphical User Interface: WS-PGRADE Workflow Engine Workflow storage File storage Application repository Logging gUSE information system Submitters Local resources, service grid VOs, Desktop Grid resources, Web services, Databases gUSE Meta-broker Submitters File storage Submitters

29. Desktop Grid infrastructure on Demand

30. 30 The activity of Russian Chapter of International Desktop Grid Federation

31. 31 DEGISCO WP4 24/06/2016 31 What is IDGF? http://desktopgridfederation.org

32. 32 What is Russian Chapter of IDGF? http://desktopgrid.ru

33. 33 RUSSIAN CHAPTER’S BOARD

34. 34 DEGISCO WP4 24/06/2016 34 GOALS  Communication, Dissemination, Community management  Joint Research activity  Collaboration with volunteers’ community  International collaboration

35. 35 DEGISCO WP4 24/06/2016 35 Communication, dissemination, and Community management  Tutorials: GRID’2010, IT-EDU’2010,DICR’2010,PAVT’2011, GRID’2012, GRID’2014  Special tracks: GRID’2010, GRID’2012, GRID’2014, NSCF’2013  Scientific publications and presentations

36. 36 DEGISCO WP4 24/06/2016 36 PRESS RELEASES  Андреев А.Л., Манзюк М.М., Ватутин Э.И. Весь мир как суперкомпьютер // Троицкий вариант наука. № 16 (110) от 14 августа 2012.  Ивашко Е.Е. Выскопроизводительные вычисления в DesktopGrid // Суперкомпьютеры 2013  Посыпкин М.А. Как стать компьютерным донором? // Научная Россия 2013

37. 37 BOINC-FAST in Karelia 2013 BOINC-FAST in Karelia 2015 is coming!

38. 38 DEGISCO WP4 24/06/2016 38 Joint Research  IITP RAS provides support for SAT@home, for new team from Crimea  CluboRUN (joint project of IDSTU RAS, IITP RAS and volunteers)  IITP RAS donates unused computing power for volunteer computing

39. 39 DEGISCO WP4 24/06/2016 39 Collaboration with Volunteers’ Community  Promoting and getting resources for scientific projects  Testing scientific projects  Feedback  Joint events  Joint publications

40. 40 COMMUNITY STUDY Details: Dr. Ilya Kurochkin, Prof. Vladimir Yakimets RESEARCH OF PREFERENCES OF PARTICIPANTS OF THE VOLUNTARY DISTRIBUTED COMPUTING, today, 18.00 at Desktop Grid section

42. 42 DEGISCO WP4 24/06/2016 42 International Collaboration  Invited talks to BOINC-FAST (Robert Lovasz, David Anderson)  Invited talks to GRID’2010, GRID’2012, GRID’2014 by Robert Lovasz, Gabor Terstzyanski, Attila Csaba Marosi  Joint projects – FP-7 DEGISCO project (completed) – RSCF project under supervision of Miklos Kozlovzsky submitted

43. 43 DEGISCO WP4 24/06/2016 43 DEGISCO FP7 PROJECT Main objective: Expand European DCIs into ICPC countries by supporting the creation of new Desktop grids for e-Science in those countries and in Europe and by connecting them using the EDGeS bridge technology. Support applications on this expanded infrastructure, disseminate, promote and provide training about this expanded infrastructure and its usage.

44. 44 DEGISCO WP4 24/06/2016 44 RSCF Project  Combined distributed computing infrastructure, mathematical models, methods and tools for fundamental and applied resource-consuming research problems  Partners: SZTAKI (Hungary), IITP RAS, KAR RC RAS, IDSTU RAS

45. 45 DEGISCO WP4 24/06/2016 45 Welcome to our section! Today, 14:30, room 407

46. 46 http:// desktopgridfederation.eu http://desktopgrid.ru Globe adapted from http://upload.wikimedia.org/ wikipedia/commons/f/fa/ Globe.svg IDGF International Desktop Grid Federation

47. 47 DESKTOP GRID ACTIVITY IN RUSSIA  ISA RAS: OPTIMA@home, DG technology, tools for performance monitoring, dissemination and user support  ISDCT SB RAS: SAT@home  KRC RAS: Useful tools (BOINC portal), DG based data mining, dissemination  BOINC.RU – Russian team of BOINC Volunteers  Other teams not mentioned here…

48. 48 DEGISCO WP4 24/06/2016 48 ISA RAS Desktop Grid Infrastructure BOINC Educational Infrastructure boinc-edu.isa.ru BOINC Test Infrastructure boinc-test.isa.ru BOINC Production Infrastructure boinc.isa.ru, sat.isa.ru Based on SZTAKI Desktop Grid Package

49. 49 DEGISCO WP4 24/06/2016 49 The Typical Pass NetmaxOPTIMA@home SAT@home Lots of (student) projects

50. 50 DEGISCO WP4 24/06/2016 50 OPTIMA@home  Project OPTIMA@home - a research project that uses Internet-connected computers to solve challenging large-scale optimization problems http://boinc.isa.ru/dcsdg/ http://boinc.isa.ru/dcsdg/  Institute for Systems Analysis of RAS, M. Posypkin, N. Khrapov

51. 51 More than 800 active hosts connected

52. 52 DEGISCO WP4 24/06/2016 52  Exhibitions – Softool’2010 – Softool’2011 – SIMO Network(Spain)’2011  Talks 1 st year: IT-EDU’2010,DICR’2010,PAVT’2011 2 nd year: – IT-EDU’2011 (regular talk) – PAVT’2012 (regular talk) – GRID’2012 (regular and plenary + Desktop Grid Track) Communication, dissemination, and Community management

53. 53 DEGISCO WP4 24/06/2016 53 Communication, dissemination, and Community management  Courses – Moscow State University – Moscow Institute for Physics and Technology  Localized material – Posters – Flyers – Russian Chapter of IDGF site: desktopgrid.ru

54. 54 DEGISCO WP4 24/06/2016 54 SAT@home: Most Successful BOINC Project in Russia Project SAT@home – research project aimed at solving hard combinatorial problems reduced to SAT problems http://sat.isa.ru/pdsat/ http://sat.isa.ru/pdsat/ Institute for System Dynamics and Control Theory of Siberian Branch of Russian Academy of Sciences (ISDCT SB RAS) A. Semenov, O. Zaikin

55. 55 SAT@home

56. 56 SAT@home: More than 5500 hosts connected

57. 57 SAT@home: Finds Useful Solutions

58. 58 KRC RAS: BOINC BASED DATA MINING Based on Partition algorithm, the special BOINC- based software has been developed in High- performance Data Center of Karelian Research Centre, RAS (HpDC) for discovering associative rules in large data sets. The software performs three stages of data processing with two points of synchronization. The software includes a client's software and the following developed BOINC's services: work generator, scheduler and assimilator. Also it uses the single MySQL database to store workunits.

60. OPTIMA@home

61. 61 DEGISCO WP4 24/06/2016 61 DEGISCO FP7 PROJECT Main objective: Expand European DCIs into ICPC countries by supporting the creation of new Desktop grids for e-Science in those countries and in Europe and by connecting them using the EDGeS bridge technology. Support applications on this expanded infrastructure, disseminate, promote and provide training about this expanded infrastructure and its usage.

62. 62 DEGISCO WP4 24/06/2016 62 International Desktop Grid Federation  The International Desktop Grid Federation (IDGF) brings together organisations and persons interested in Desktop grid computing and related technologies.  Focuses on application developers and Desktop grid operators.  IDGF is a member organisation that provides services for its members (legally: participants).  IDGF brings them together, so they can get better information, documentation and training, a platform to exchange experiences.  IDGF serves as a marketing channel for its members

63. 63 DEGISCO WP4 24/06/2016 63 The Goals of Russian Chapter  Help Russian research and industry teams to develop, deploy and maintain their desktop-grid distributed applications  Maintain DCI infrastructures for our projects and for other organizations  Provide courses and tutorials on desktop grids and volunteer computing for Russian academic and industrial community  Promote Russian BOINC projects  Explain the mission of IDGF and benefits of the membership

64. 64 DEGISCO WP4 24/06/2016 64 The Future of Russian Chapter  Maintain and improve Web-site desktopgrid.ru  Dissemination activity: Tracks Tutorials Courses Papers  Create “Umbrella” project for Russian BOINC applications  International collaboration

65. 65 DEGISCO WP4 24/06/2016 65 The Future of Russian Chapter  Joint Activity (development) BOINC Web-portal (easy application deployment and job submission) Scientific Gateways Supporting interoperability: Bridges Messaging service for BOINC Alternative DG systems ….

66. 66

67. 67 WP2: Service grid infrastructure EGI Grid site: RU-ISA-CGTDC http://grid.isa.ruhttp://grid.isa.ru

68. 68 DEGISCO WP4 24/06/2016 68 WP3: Application and User Support  ISA RAS support local developers from – Institute for System Dynamics and Control Theory of Siberian Branch of Russian Academy of Sciences – Moscow Institute for Electronic Technology (MIET) – Joint Institute for Nuclear Research

69. 69 WP2: 3G-Bridge Infrastructure 3gbridge.isa.ru