DEGISCO Desktop Grids For International Scientific Collaboration Support for application porting Budapest, Robert Lovas, MTA SZTAKI DEGISCO is supported by the FP7 Capacities Programme under contract nr RI

DEGISCO WP3 14/07/ Presentation outline 1.‘Application support’ workpackage objectives 2.Current applications 3.IDGF Application Development Methodology 4.Repositories

DEGISCO WP3 14/07/ Work package objectives  Actively participate in the search for new user communities associated with ICPC countries that could benefit from using the operated SG/DG infrastructure. (Together with WP4)  Provide support for application developers who want to port their applications to the above distributed computing infrastructure.  Provide application validation service and support based on the EDGeS Application Validation Procedure extended and supported by ETICS tools and practices.  Support the deployment and operation of ported applications on the production infrastructure by establishing and operating regional helpdesks and a 2nd tier expert group.

DEGISCO WP3 14/07/ Applications  BNB Grid, a generic framework for implementing optimization algorithms on distributed systems – ISA RAS  CPDynSG, City Population Dynamics and Sustainable Growth – IMP  MultiscaleIVideoP, Multiscale image and Video Processing – IMP  SLinCA, Scaling Laws in Cluster Aggregation - IMP  VAST, VisuAl and SemanTic image search - HUST  AutoDock via GVSS, virtual screening for potential drugs - ASGC  HAP, Hydro-Aerodynamic Problem - KBTU

DEGISCO WP3 14/07/ Applications II.  PD-SAT, Parallel and Distributed SAT - Institute for System Dynamics and Control Theory of Siberian Branch of Russian Academy of Sciences and ISA RAS  LAMMPS, Large-scale Atomic/Molecular Massively Parallel Simulator - IMP  SemreX - Semantic based large scale literature retrieval and browsing - HUST  Reservoir - predicts the behaviour of a reservoir - UFCG  Cisterns - simulates several scenarios of rain water collecting systems - UFCG  MARBS - simulating and helping to predict the water demanded by the soil - UFCG  EPANET - plan and improve a system's hydraulic performance - UFCG

DEGISCO WP3 14/07/ Application Example I LAMMPS: Large-scale Atomic/Molecular Massively Parallel Simulator LAMMPS: molecular dynamics simulations of nanofabrication processes with physical parameter decomposition for parameter sweeping

DEGISCO WP3 14/07/ Application Example I LAMMPS and SLinCA running at the public DG SLinCA: Scaling Laws in Cluster Aggregation LAMMPS: Large-scale Atomic/Molecular Massively Parallel Simulator Supported by over 2000 volunteers Over 1200 GFLOP/s performance

DEGISCO WP3 14/07/ More information

DEGISCO WP3 14/07/ Application Example II GVSS – Grid Virtual Screening Service using AutoDock3.05 Virtual screening of molecules supporting drug discovery Using AutoDock3.05 Web portal interface Runs on both SG and DG resources

DEGISCO WP3 14/07/ Application Example II GVSS – Grid Virtual Screening Service using AutoDock3.05

DEGISCO WP3 14/07/ The recommended application support framework

DEGISCO WP3 14/07/ DEGISCO partners

13 Why do we need a methodology? Motivations Grid application development is very often ad-hoc –Developers do not follow any methodology –Poorly documented systems –User expectations not fulfilled Systems design and development methodologies are too generic –Special focus is required when porting/developing an application to a SG/DG platform EADM: IDGF Application Development Methodology

14 EADM EADM IDGF Application Development Methodology Aims and objectives: –develop a generic methodology that addresses the problem of application porting and defines how the recommended software tools, supported by DEGISCO, can aid this process. EADM – an iterative approach –EADM identifies well defined stages that have a suggested logical order. However, the overall process in most cases is non-linear allowing revisiting and revising the results of previous phases at any point.

15 EADM Roles and participants End–users: those who utilize the application in their scientific research or industrial/business conduct Developers/system administrators of original application: application programmers who developed the current version of the application / system administrators who are responsible for the installation and administration of the software DEGISCO systems analysts: capturing user requirements and making conceptual design specification DEGISCO application programmers: responsible for the implementation of the migration Grid operators: operating the Grid on which the ported applications are running

DEGISCO WP3 14/07/ IDGF Application Development Methodology

17 EADM Stages – an overview EADM Stages – an overview Earlier stages can be revisited at any time 1. Analysis of current application 2. Requirements analysis 3. Systems design EADM stagesParticipants Outcome Developers of original application, DEGISCO systems analysts, (end-users) Application Description Template (ADT) End-users, developers of original application, DEGISCO systems analysts User Requirement Specification (URS) Systems Design Specification (SDS) Developers of original application, DEGISCO systems analysts, application programmers, Grid operators

18 4. Detailed design 5. Implementation 6. Testing EADM stagesParticipants Outcome DEGISCO systems analysts, EDGeS application programmers Technical Design Specification (TDS) DEGISCO application programmers, DEGISCO systems analysts Incremental system prototypes Test Report Independent testers, end- users, DEGISCO application programmers EADM Stages – an overview EADM Stages – an overview Earlier stages can be revisited at any time

19 7. Validation 8.Deployment 9.User support, maintenance and feedback EADM stagesParticipants Outcome DEGISCO validation team Validation report DEGISCO Grid operators Application deployed on DEGISCO infrastructure Feedback, support and system modification End-users DEGISCO systems analysts EADM Stages – an overview EADM Stages – an overview Earlier stages can be revisited at any time

20 EADM EADM Stage 1: Analysis of current application Aims and objectives: –describe the currently existing application in detail Participants: –developers/system administrators of original application, DEGISCO systems analysts, (end-users) Questions to be answered (original application): –Identification of problem domain –Identification of target user community –Type of computing platform currently utilised –Type of parallelism (if any) –Current data access –Current functionalities –Other factors (licensing issues, programming language, operating system, memory usage, security, ethical or gender issues)

21 EADM EADM Stage 2: Requirements analysis Aims and objectives: –identify how the target user community will benefit from porting the application to the DEGISCO platform Participants: –end-users, developers/ system administrators of original application, DEGISCO systems analysts Questions to be answered (ported application): –User requirements –Desired functionalities –Desired efficiency of execution –Desired efficiency of data access –Target computing platform –Required user interface –Other factors (licensing, security, ethical issues etc.)

22 EADM EADM Stage 3: Systems design Aims and objectives: –Design the ported application at conceptual level taking user requirements and technical feasibility into consideration Participants:, –developers/system administrators of original application, DEGISCO systems analysts, DEGISCO application programmers, DEGISCO Grid operators Questions to be answered: –Identify target computing platform to be used as entry point - changes in application required? –Identify/design user interface –Define and design parallelisation principles –Design data access and transfer scenarios –Other factors

23 EADM EADM Stage 4: Detailed design Aims and objectives: –prepare a detailed technical design specification that forms the basis of implementing the required changes when porting the application to the DEGISCO platform Participants: –DEGISCO systems analysts, DEGISCO application programmers Questions to be answered: –How the required changes will be implemented to utilise the target computing platforms –How the user interface will be implemented –How to implement the required parallelism –How to implement the data access and transfer scenarios

24 EADM EADM Further stages 5.Implementation: implement the required changes in the application based on the detailed design specification 6.Testing: both functional and performance tests 7.Validation: in order to run on the DEGISCO platform every application has to be validated to assure they cause no harm to the underlying DG infrastructure (SG->DG direction) 8.Deployment: validated applications are published in the application repository (SG->DG direction) or deployed on the target DG system (DG->SG direction) 9.User support, maintenance and feedback: full user support is provided by the Application Support Service

25 Application Development to SG/DG platform Challenges For the Developer –DG version of the application has to be developed Only low level APIs (e.g. BOINC API) Specific expertise is required Platform dependent solutions (different BOINC/XtremWeb version) –Application composition Creation of workflows is not supported For the user –Transparency for the user User would require the same or similar user interface for DG and SG applications Total lack of UI for DG systems (e.g. only BOINC Admin can run the application) –Only validated applications can be run on a DG Requires an application repository Publish select, parameterize and execute applications

26 Tools supporting application development and execution in DEGISCO API level support – for the developer DC-API: Provides a uniform interface for different Grid systems (BOINC, Condor, XtremWeb) GenWrapper: Generic wrapper to port legacy applications to a BOINC platform without “Boincification ” High level graphical user interface – for the developer and the user WS-PGRADE portal: to support the transparent exploitation of SG/DG systems at workflow level Application Repository: Integrated with the P-GRADE portal Publish, select, parameterise and execute validated applications

27 application source code available only binary available Developing a DG application GenWrapper DC-API WS-PGRADE portal GUI BOINC enabling WS-PGRADE portalApplication (Super)repositories

DEGISCO WP3 14/07/ DG Application super-repository

DEGISCO WP3 14/07/ Ported applications EGI AppDB

DEGISCO WP3 14/07/ Applications from ICPC partners

DEGISCO WP3 14/07/ Support activity  Hierarchical  Global support  Expert groups  Ticketing & Fora

DEGISCO WP3 14/07/ Scientific gateways (workflows / jobs)

Globe adapted from wikipedia/commons/f/fa/ Globe.svg