1 SCI-BUS use case Peter Kacsuk, Sandor Acs, Mark Gergely MTA SZTAKI Start date: Duration: 36 months SCI-BUS is supported by the FP7 Capacities Programme under contract nr RI

2 Main objectives of SCI-BUS Create a generic-purpose gateway technology Elaborate an application-specific gateway building technology and customisation methodology Establish production gateway services both for NGIs (horizontal user communities) and various science communities (vertical user communities) Provide seamless access to major computing, data and networking DCIs and services including supercomputers, clusters, grids and clouds Provide gateway development and application development support Develop business models to guarantee the sustainability of the gateway services and to enable the commercial exploitation

3 Project partners NoParticipant organisation namePart. short nameCountryExpertise & Experience 1Magyar Tudomanyos Akademia Szamitastechnikai es Automatizalasi Kutato Intezet MTA SZTAKIHungaryCoordinator in EDGeS, EDGI, SHIWA, developer of WS-PGRADE 2Academisch Medisch Centrum bij de Universiteit van Amsterdam AMCThe NetherlandsGateway developer for the medical user community 3Simsoft Bilgisayar Teknolojileri Ltd. Sti.SIMSOFTTurkeyGateway developer for business process modeling 4E-GROUP Ict Software Informatikai ZrtEGHungaryGateway developer for Web 2.0 5Eidgenössische Technische Hochschule ZürichETH ZurichSwitzerlandGateway developer for bioscience 6Middle East Technical UniversityMETUTurkeyGateway developer for seizmology 7Scaletools Ltd.STSwitzerlandGateway developer for clouds 8Eberhard Karls Universität TübingenEKUTGermanyGateway developer for computational chemistry 9University of WestminsterUoWUKDeveloper of UK NGS P-GRADE 10Universidad de ZaragozaUnizarSpainGateway developer for PireGrid SMEs 11CloudBroker GmBhCBSwitzerlandGateway developer for clouds 124D SOFT SZAMITASTECHNIKAI KFT4D SOFTHungaryGateway developer for ETICS-2 13Istituto Nazionale di AstrofisicaINAFItalyGateway developer for astrophysics 14Laurea-ammattikorkeakoulu oyLaureaFinnlandGateway developer for Blender 15The Provost Fellows & Scholars of the College of the Holy and Undivided Trinity of Queen Elizabeth near Dublin TCDIrelandGateway developer for helio- physics

4 Re-usability and adaptability to different communities and needs Create 11 gateways in the 1st project year for the following user communities: –International seismology community –Helio-physics communityc –Swiss systems biology community of the SystemsX.ch project –German MoSGrid computational chemistry and bioinformatics community –Biomedical researchers community of the Academic Medical Centre of the University of Amsterdam –Astrophysics community –PireGrid SMEs community –Business process modelling community –Blender rendering community –Citizen web-2 community –Public application developer community (based on ETICS-2 results) Create another 16 user community gateways (including 6 subcontractors)

5 The SCI-BUS Infrastructure Create 11 gateways in the 1 st project year

SCI-BUS and EGI Cloud Federation (ECF) 6 What SCI-BUS will provide for ECF users? SCI-BUS gateway to access ECF IaaS cloud resources on- demand for parameter sweep (high-throughput computation) applications (like many simulation applications) Access the ECF IaaS cloud resources for SaaS services using the CloudBroker platform Images of the SCI-BUS gateways to enable their easy deployment in the ECF (and other) clouds

SCI-BUS and EGI Cloud Federation (ECF) 7 What SCI-BUS requires from ECF? DCI images based on which DCIs can be easily and quickly deployed in the Clouds Cloud resources where these images can be deployed Use scenario To quickly test new gateway versions with various DCIs (use case for ECF) SCI-BUS partners are not DCI experts so they would need the already existing DCI service images based on which they could deploy easily and quickly a certain DCI in the cloud for the time of testing if the gateway runs correctly with this DCI Why? DCI services are frequently updated or changed It is difficult to find existing DCIs where gateway developers can test the gateway with different versions of DCIs

Use case objectives 8 SCI-BUS uses the ETICS-2 systematic build and test technology for quality assurance of the gateways. The objectives of using the clouds of ECF are: To quickly test new generic purpose gateway versions (WS- PGRADE/gUSE) with various DCIs by SZTAKI developers –Required DCI images: ARC, gLite, UNICORE, GT2, GT4, GT5, PBS, LSF, BOINC To quickly test new science gateways customized from WS- PGRADE/gUSE with various DCIs by all 20 project partners and subcontractors –Required DCI images: ARC, gLite, UNICORE, GT2, GT4, GT5, PBS, LSF, BOINC

9 SZTAKI Cloud experiences Production laboratory-level Cloud –Opennebula –Size: 112 cores, 3 Tbyte disk –Used by the 25 lab members in daily practice for example developing SCI-BUS gateways Based on the positive experiences of the laboratory-level Cloud, SZTAKI decided to set up an institute-level cloud to be used as production cloud by all 300 staff members in daily practice –Opennebula –Size: 462 cores, 72 Tbyte disk

SZTAKI CLOUD Architecture Node 64Core 256GB RAM Node 64Core 256GB RAM Node 64Core 256GB RAM Node 64Core 256GB RAM Frontend 8 Core, 16GB RAM Switch 48port,4X10G Switch 48port,4X10G Storage 36TB Storage 36TB Frontend 8 Core, 16GB RAM Node 64Core 256GB RAM Node 64Core 256GB RAM Node 64Core 256GB RAM

Use case architecture 11 SCI-BUS gateway ECF Cloud 1 ECF Cloud 2 Glite DCI ARC DCI Test access SZTAKI Cloud (Opennebula)

First experiment 12 SCI-BUS gateway SZTAKI Cloud (Opennebula) EDGI (gLite+BOINC) DCI Test access Component images of EDGI DCI: 4 gLite images: UI, VOMS, WMS, CREAM CE 1BOINC image: BOINC server and client All these images have been developed by EDGI and accessible at: SZTAKI Cloud (Opennebula) Successful in-house experiment in SZTAKI.

Second experiment 13 SCI-BUS gateway CESNET Cloud (Opennebula) EDGI (gLite+BOINC) DCI Test access Many thanks to CESNET and particularly Boris Parak Component images of EDGI DCI: 4 gLite images: UI, VOMS, WMS, CREAM CE 1BOINC image: BOINC server and client All these images have been developed by EDGI and accessible at: SZTAKI Cloud (Opennebula)

14 Problems we have found Access to the sites –Access to the sites first should be asked from the cloud admins (some of them helpful, some of them not) –Slow and unpredictable procedure –No unified access mechanism: Some service should be accessed by certificate (e.g. CESNET OCCI interface X509 certificate is needed, Sunstone access with login user/pass)

15 Problems we have found Use of the sites –The different clouds use different hipervisors but this crucial information is missing in the site descriptions –Network access is different site by site (e.g. CESNET gives public network access but others might not), this information is also missing in the site descriptions –Some short documentation for experts how to deploy the virtual appliances would be useful: What kind of contextualization is used What kind of hipervisor to be used

16 Problems we have found Access to DCI images –We would need a cloud marketplace from where the DCI bridge images could be accessed –First we thought StratusLab’s marketplace to use. However: There is no complete gLite set: APEL, SE, CREAM CE, WE are available but for eample, there is no WMS and UI (even that is available is the old gLite 3.2) ARC and UNICORE are completely missing Images are containing lot of unnecessary packages StratusLab contextualization is different from Opennebula and not so widespread, just 14% in the testbeds of ECF

17 Suggested solution ECF (or someone else) should provide a marketplace with useful DCI images Who create the images? Three options: 1.The middleware projects like EMI and EDGI since they are the experts of the middleware services. (These projects soon will be finished) 2.A specialized project like StratusLab 3.A community effort where members of the various communities create the images (no guarantee for the quality of the images) We recommend Option 2. EGI could organize a new project with this purpose. SZTAKI is happy to contribute based on the current experience of creating and using images.

18 Sustainability of the marketplace It is not enough to create the marketplace, it should be maintained in a sustainable way. Who would maintain the images? The three main options are the same as before for creation. We recommend Option 3. This could be a community effort with some additional services organized by EGI: –Basically we expect that the EGI/NGI community maintains the images (since it is their interest) –However, if an image is not updated after a certain time EGI should find a volunteer (or paid) community member to do the job (maybe some financial resources of EGI could be allocated for this purpose)