1. 11 SZTAKI & International Desktop Grid Federation: status report József Kovács (MTA SZTAKI) jozsef.kovacs@sztaki.mta.hu jozsef.kovacs@sztaki.mta.hu http://idgf-sp.eu IDGF-SP is supported by the FP7 Capacities Programme under contract nr RI-312297

2. 2 International Desktop Grid Federation: short introduction Legal form: Dutch foundation Main vehicle for dissemination during the lifetime of the EC funded projects and for continuation after the projects Federation is open for organisations and individuals in research and industry A member organisation with chapters and working groups Focal point for DCI/Desktop operators and application developers Webportal: discussion fora, events and workshops, support tools, etc. Provide assistance and facilities to launch new joint actions, initiatives, projects

3. 3 Involve and engage in long-term significantly more citizens and new communities in the volunteer andvolunteer and private (campus-wide or enterprise) Distributed Computing Infrastructuresprivate (campus-wide or enterprise) Distributed Computing Infrastructures by supporting the (1) rapid creation, (2) efficient operation, and (3) dynamic expansion of this type of DCIs for e- Science. Coordinate and synchronise the dissemination and support activities of major European stakeholders of volunteer and Desktop Grids with focus on the International Desktop Grid Federation. IDGF-SP International Desktop Grid Federation - Support Project Supported by: EC in the FP7 Project acronym: IDGF-SP Contract n°: RI-312297 Project type: CSA-SA Start date: 01/11/2012 Duration: 24 months Funding from the EC: 860.000 € Total funded effort in person-month: 136 Web site: http://idgf-sp.euhttp://idgf-sp.eu Coordinator: Dr. Robert Lovas email: rlovas@sztaki.hu

4. 4 SUMMARY: software components of maintained by SZTAKI SZDG boinc package: debian variant of BOINC supported by our toolsSZDG boinc package: debian variant of BOINC supported by our tools 3G Bridge: remote job submission, converting incoming jobs to WUs3G Bridge: remote job submission, converting incoming jobs to WUs DC-API: c library for creating master-worker applicationsDC-API: c library for creating master-worker applications Genwrapper: porting legacy application to BOINC, shell interpreterGenwrapper: porting legacy application to BOINC, shell interpreter GBAC: virtualisation under BOINC client, based on virtualboxGBAC: virtualisation under BOINC client, based on virtualbox Metajob: submission of large number of jobs to 3Gbridge in one turnMetajob: submission of large number of jobs to 3Gbridge in one turn Cloud handling by 3G Bridge: dynamic on-demand cloud burstingCloud handling by 3G Bridge: dynamic on-demand cloud bursting Monitoring by 3G Bridge: information collection on job throughputMonitoring by 3G Bridge: information collection on job throughput EGI: gLite CREAM mCE: automatic job transfer from gLite to BOINCEGI: gLite CREAM mCE: automatic job transfer from gLite to BOINC EGI: Munin and Nagios: EGI compliant monitoringEGI: Munin and Nagios: EGI compliant monitoring gUSE: web-based graphical portal for executing jobs and workflowsgUSE: web-based graphical portal for executing jobs and workflows 4

5. 5 Documentation site: doc.desktopgrid.hu 5

6. 6 Recent (ongoing) activities and plans (I.) Supporting the latest BOINC release by our tools (DC-API, 3GBridge, …)Supporting the latest BOINC release by our tools (DC-API, 3GBridge, …) plan is to put DC-API into the BOINC trunkplan is to put DC-API into the BOINC trunk testing has been successfully donetesting has been successfully done need David to integrate our staff into the BOINC code reponeed David to integrate our staff into the BOINC code repo Improving GBACImproving GBAC handling of stdout & stderr of application has been improvedhandling of stdout & stderr of application has been improved returning the log of the internal GBAC component for debugging purposesreturning the log of the internal GBAC component for debugging purposes automatic image extraction on the client side (image size 700MB -> 100MB)automatic image extraction on the client side (image size 700MB -> 100MB) short term plan is to support image selection (discussing what is the easiest/most efficient way to implement it…)short term plan is to support image selection (discussing what is the easiest/most efficient way to implement it…) Improving 3GBridgeImproving 3GBridge 3GBridge supports REST interface, implementation finished3GBridge supports REST interface, implementation finished http://doc.desktopgrid.hu/doku.php?id=manual:restifhttp://doc.desktopgrid.hu/doku.php?id=manual:restifhttp://doc.desktopgrid.hu/doku.php?id=manual:restif consumed CPU hours, credits can now be queried (required for EGI accounting & ELTE)consumed CPU hours, credits can now be queried (required for EGI accounting & ELTE) 6

7. 7 Recent (ongoing) activities and plans (II.) Volunteer StorageVolunteer Storage Using random linear coding in distributed storageUsing random linear coding in distributed storage See presentation from Adam VisegradiSee presentation from Adam Visegradi New applications on our public, volunteer projectsNew applications on our public, volunteer projects BIOVEL community applications on EDGeS@homeBIOVEL community applications on EDGeS@home Biome-BGC ecosystem modeling is the estimation of behavior of terrestrial ecological systems (forests, grasslands, agricultural areas) as accurately as possible. Biome-BGC ecosystem modeling is the estimation of behavior of terrestrial ecological systems (forests, grasslands, agricultural areas) as accurately as possible. New mathematical application will appear on SZTAKI Desktop Grid these daysNew mathematical application will appear on SZTAKI Desktop Grid these days Riemann zeta research project: http://www.riemann-siegel.com/ Riemann zeta research project: http://www.riemann-siegel.com/http://www.riemann-siegel.com/ Locating large values of zeta functions on the critical lines Locating large values of zeta functions on the critical lines New user community: DIRACNew user community: DIRAC The DIRAC system has been recently integrated with BOINC through GBACThe DIRAC system has been recently integrated with BOINC through GBAC The GBAC VM image has been adjusted to the needs of DIRACThe GBAC VM image has been adjusted to the needs of DIRAC DIRAC jobs are currently arriving through gLiteDIRAC jobs are currently arriving through gLite Testing finished, deployment on EDGeS@home will happen these daysTesting finished, deployment on EDGeS@home will happen these days 7

8. 8 Recent (ongoing) activities and plans (III.) Cloud scenariosCloud scenarios We created cloud images to support automatic setup of BOINC server with clientsWe created cloud images to support automatic setup of BOINC server with clients To ease the deployment To ease the deployment To perform dynamic, on-demand computing infrastructure for scientific users To perform dynamic, on-demand computing infrastructure for scientific users To encourage scientific users to utilise BOINC To encourage scientific users to utilise BOINC We are creating images for each scenario defined at doc.desktopgrid.huWe are creating images for each scenario defined at doc.desktopgrid.hu Image with BOINC clientImage with BOINC client Needs a proper configuration to force the client to continuously execute WUs in the cloudNeeds a proper configuration to force the client to continuously execute WUs in the cloud Not so easy in the new scheduling algorithm, recompilation is neededNot so easy in the new scheduling algorithm, recompilation is needed Adjusting related parameters hardwired in the client_state.h would help…Adjusting related parameters hardwired in the client_state.h would help… #define WORK_FETCH_PERIOD 60 #define WORK_FETCH_PERIOD 60 #define WF_MIN_BACKOFF_INTERVAL 600 #define WF_MIN_BACKOFF_INTERVAL 600 #define WF_MAX_BACKOFF_INTERVAL 86400 #define WF_MAX_BACKOFF_INTERVAL 86400 #define WF_DEFER_INTERVAL 300 #define WF_DEFER_INTERVAL 300 8

9. 9 European Grid Infrastructure (short overview) EuropeanEuropean Over 35 countriesOver 35 countries GridGrid Secure sharing of IT resourcesSecure sharing of IT resources InfrastructureInfrastructure Computers (clusters): 372,612 CPUsComputers (clusters): 372,612 CPUs Data: 180 PB diskData: 180 PB disk Applications: 1.67 million job/day (thousands of scientists)Applications: 1.67 million job/day (thousands of scientists) …. and beyond!!…. and beyond!! Technologies toTechnologies to Federate resourcesFederate resources Access resourcesAccess resources 9 Coordinator: EGI.eu, Amsterdam

10. 10 Proposed EGI DG-VT Project General project infoGeneral project info Start (planned) :September-October 2013Start (planned) :September-October 2013 Duration : 6 monthsDuration : 6 months Leader: MTA SZTAKI from the Hungarian NGILeader: MTA SZTAKI from the Hungarian NGI “EGI-InSPIRE is ideally placed to join together the new Distributed Computing Infrastructures (DCIs) such as clouds, supercomputing networks and desktop grids, for the benefit of user communities within the European Research Area.” --- EGI-InSPIRE Project fact sheet“EGI-InSPIRE is ideally placed to join together the new Distributed Computing Infrastructures (DCIs) such as clouds, supercomputing networks and desktop grids, for the benefit of user communities within the European Research Area.” --- EGI-InSPIRE Project fact sheet MotivationMotivation Despite the on-going work and the benefits of the integrated Desktop Grid DCI resources available for EGI users, only a part of the EGI community is aware of the latest Desktop Grid related achievements, and a fraction of the EGI users and infrastructure operators take their advantages in everyday practice.Despite the on-going work and the benefits of the integrated Desktop Grid DCI resources available for EGI users, only a part of the EGI community is aware of the latest Desktop Grid related achievements, and a fraction of the EGI users and infrastructure operators take their advantages in everyday practice. In order to fill this gap; a part of the targeted objectives are technical ones but there is stronger focus on the human aspects – training, networking and support activities.In order to fill this gap; a part of the targeted objectives are technical ones but there is stronger focus on the human aspects – training, networking and support activities. See list of Virtual Teams at https://wiki.egi.eu/wiki/Overview_of_Virtual_Team_projectsSee list of Virtual Teams at https://wiki.egi.eu/wiki/Overview_of_Virtual_Team_projects https://wiki.egi.eu/wiki/Overview_of_Virtual_Team_projects 10

11. 11 Proposed EGI DG-VT Project Promote and train the Desktop Grid related technologies in the EGI communitiesPromote and train the Desktop Grid related technologies in the EGI communities Utilize the available bridged Desktop Grid resources by more VOsUtilize the available bridged Desktop Grid resources by more VOs Increase the number of heavily used EGI applications on the integrated (Desktop Grid) infrastructure with focus on widespread tools/solutions/approachesIncrease the number of heavily used EGI applications on the integrated (Desktop Grid) infrastructure with focus on widespread tools/solutions/approaches Improve documentation, e.g. Road maps, training materials, manualsImprove documentation, e.g. Road maps, training materials, manuals Complete the final remaining steps for full integration concerning e.g. dashboardComplete the final remaining steps for full integration concerning e.g. dashboard Find joint EGI champion(s) - IDGF ambassador(s)Find joint EGI champion(s) - IDGF ambassador(s) 11

12. 12 Current status of integration into the EGI infrastructure https://wiki.egi.eu/wiki/Desktop_Grids_integration 12

13. 13 EGI related activities and plans EGI activated its efforts in full integration of the Desktop Grid systems/projects to the EGI infrastructure and in promoting Desktop Grid among the EGI scientific usersEGI activated its efforts in full integration of the Desktop Grid systems/projects to the EGI infrastructure and in promoting Desktop Grid among the EGI scientific users EGI plans to put manpower behind as wellEGI plans to put manpower behind as well Plans:Plans: EDGeS@home will be renamed to EGI@homeEDGeS@home will be renamed to EGI@home Technical discussion is needed, how to do that? Share your experiences if any… Technical discussion is needed, how to do that? Share your experiences if any… Screensavers for the EGI applicationsScreensavers for the EGI applications Use the cooperation with BIOVEL and DIRAC as a positive exampleUse the cooperation with BIOVEL and DIRAC as a positive example More dissemination (newsletters, leaflets) to notify scientific users about DG resourcesMore dissemination (newsletters, leaflets) to notify scientific users about DG resources 13

14. 14 Thank you for your attention… Presenter: Jozsef Kovacs (jozsef.kovacs@sztaki.mta.hu) jozsef.kovacs@sztaki.mta.hujozsef.kovacs@sztaki.mta.hu Project website: http://idgf-sp.eu http://desktopgridfederation.eu http://www.lpds.sztaki.hu Acknowledgements:  The presented work is supported by the FP7 Capacities Programme under contract nr RI-312297 (IDGF-SP). 14