SEE-GRID-SCI The SEE-GRID-SCI initiative is co-funded by the European Commission under the FP7 Research Infrastructures contract no Grid - korisnicki pristup i razvoj aplikacija Branko Marovic RCUB - UoB Meteo VO Training, Belgrade 24. June 2008

Application Gridification2/26 SEE-GRID-2 Applications Adaptation & Support Application lifecycle defined in “Gridification Guidelines” document. SG Gridification Guide (wiki pages) structured and contributed by developers. Policy: Applications initially supported at the country level. Application Support Group (ASG) created for gridification support. Application Review Committee(ARC) created to assess existing and new candidate applications. Application selection criteria defined by ARC. MoU Between Applications and SEEGRID-2.

Application Gridification3/26 SEE-GRID-2 Strategy for New Applications There is “Gridification Guideline” document to provide initial help to developers. (application life- cycle, policy and methodology to run and support an application on Grid) There is “SG Gridification Guide” on SEE-GRID-2 Wiki to provide comprehensive help to developers (porting an existing application to the Grid) ARC (Application Review Committee) recommendations push applications towards collaborations with other applications and projects. Applications are required to provide update on their established user community and collaborations within their periodic Application Progress Report.

Application Gridification4/26

Application Gridification5/26 SEE-GRID2 Applications

Application Gridification6/26 Support the deployed applications Application support group (ASG) created to provide fast and continuous gridification support to developers.  Experienced developers & admins  ASG team members assigned for each application.  National level application support  SEE-GRID - global level application support SG Gridification Wiki online. SEE-GRID helpdesk available and used for problems.  Useful Helpdesk Tickets for Developers.  SEE-GRID Helpdesk “Applications Support” knowledge base.  GOODs responsible to dump useful tickets of the week. Work in close collaboration with WP5 (training) and WP3 (software requirements, maintenance of performance)

Application Gridification7/26 Application Lifecycle

Application Gridification8/26 SEE-GRID-2 Gridification Guides An initial guide for the developer including the application life-cycle, VO and site policies, gridification methodology:  Gridification Guideline  grid.eu/content/modules/downloads/SEEGRID2- WP4-TR-008-Gridification_Guideline-n doc Detailed wiki pages with illustrations for gridification issues:  SG Gridification Guide  see.org/index.php/SG_Gridification_Guide

Application Gridification9/26 SEE-GRID Gridification Guide Relevant topics for application developers identified trough online questionnaire system Some investigation areas identified as well – candidates for future GG topics Gridification guide will provide information on these topics Wiki as collaboration medium

Application Gridification10/26 SEE-GRID Gridification Guide 1 Job Management 1.1 Running Jobs (T1) 1.2 Software Installation Management (T3) 2 File Management and Access 2.1 Data Management (T2) 2.2 Data Access (T4) 2.3 Logical Files (T6) 2.4 Advanced Replica Management (T2, T6 related investigation area) 3 Inter-Job Communication 3.1 Workflows (T7) 3.2 MPI (T9) 3.3 Interactive & Pilot Jobs (T1 related investigation area) 3.4 Java networking 4 Information Services 4.1 Application-Level Event Logging & Performance (T5) Applications and Systems monitoring tools Benchmarking 4.2 Metadata Catalogues 5 Development Environments and Portals 5.1 Web Authentication and Authorization (T8) 5.2 P-GRADE Portal (T1-T9 related investigation area) 5.3 P-GRADE (T1-T9 related investigation area)

Application Gridification11/26

Application Gridification12/26 VO Resource Allocation Policies Seegrid VO is used for all applications:  Utilization of groups and roles: /seegrid/COUNTRY_CODE/SHORT_APP_NAME/Role=production /seegrid/COUNTRY_CODE/SHORT_APP_NAME/Role=software For gridification testing phase,  no role or group is used, all sites support all applications. For deployed applications,  sites are encouraged to support as many applications as possible. MoU template between SEE-GRID-2 Project and the Application created.

Application Gridification13/26 SEE-GRID-SCI

Application Gridification14/26 Meteorology VO Members  National Observatory of Athens (coordinator)  South Environment and Weather Agency of Serbia  Hydrometeorological Institute of Montenegro  RBI of Zagreb  University of Zagreb  Federal Hydro-Meteorological Institute of Bosnia and Herzegovina  Republic Hydrometeorogical Institute, Banjaluka Two sets of applications for initial deployment:  Regional scale Multi-model, Multi-analysis ensemble forecasting system  Study of the interaction of airflow with complex terrain

Application Gridification15/26 Developer Resources Grid environment is constantly evolving, but  Useful features persist  New are constantly being added  Bugs are being fixed  Gained knowledge remains relevant, must be updated  Applications can be easily migrated to new/updated APIs gLite User Guide  WMS user guides  SEE-GRID Gridification Guide  SEEGRID Wiki  gLite documentation 

Application Gridification16/26 Grids: a foundation for e- Research Enabling a whole-system approach Collaborative research / engineering / public service … sensor nets Shared data archives computers software colleagues instruments Grid Diagram derived from Ian Foster’s slide

Application Gridification17/26 Načini upotrebe grida Komandna linija, JDL, input/output sandbox i/ili ručna manipulacija podacima Primena skriptova u generisanu poslova, obradi i manipulaciji podacima Korišćenje postojećih Grid portala Primena grid API-ja i servisa iz koda koji se izvršava na gridu Pravljenje aplikativno specifičnih pristupnih tačaka-servisa

Application Gridification18/26 Complexities of grid applications 1. Simple jobs – submitted to WMS to run in batch mode 2. Job invokes grid services To read & write files on SE Monitoring For outbound connectivity (interactive jobs) To manage metadata … 3. Complex jobs An environment controls multiple jobs on users’ behalf High-level services Portals with workflow Software written for the VO (or by the user) …

Application Gridification19/26 Invocation of applications From the UI  Command Line Interfaces / Scripts  APIs  Higher level tools From desktop Windows applications  Use Grids without awareness of them!  But gLite not (yet) fully supporting Windows (more info: From portals  For recurring tasks: “core grid services” as well as application layer  Accessible from any browser  Tailored to applications  Different portal solutions, and wide range of capabilities. Second part of this course: P-GRADE Portal

Application Gridification20/26 Characteristics of VOs What is being shared?  resources of storage and/or compute cycles  software and/or data Distinct groups of developers and of users?  Some VOs have distinct groups of developers and users…  Biomedical applications used by clinicians,….  …. Some don’t  Physics application developers who share data but write own analyses  Effect: need to  hide complexity from the 1 st type of VOs  expose functionality to 2 nd type of Vos  many security issues

Application Gridification21/26 Challenges Research software is often  Created for one user: the developer  Familiarity makes it useable  Short-term goals: Used until papers are written and then discarded Grid applications are often used  by a VO  Without support from developer  In new contexts and workflows Need expertise in: software engineering application domain grid computing Grid application developers are  In a research environment  Yet their s/w must have:  Stability  Documentation  Usability  Extendability i.e. Production quality

Application Gridification22/26 Consequences Team work! Engaged in world-wide initiatives – reuse, don’t make your own! Cross disciplines for solutions. From research to production software: ~5 times the effort.  “80% of the time for last 10% of the functionality & reliability” Standardisation is key  For re-use, for dynamic configuration of services,..  Both for middleware and domain specific Need to follow a deliberate development process  Waterfall? Rapid prototyping?  Requirements engineering, design, implementation, validation, deployment  Engaged with the user community

Application Gridification23/26 Basic tasks while Porting applications to the Grid 1. Developing a non-grid application (or inheriting and updating an ancient one); 2. Go/no-Go decision about gridification Is it suitable for the Grid environment? “Cost/profit” analysis / feasibility study Typical Questions Groups:  Current structure of the application  Dependencies of the application  Available resources (manpower, knowledge, etc.)  Requirements for the gridified application  Expected impact of gridification  Requirements for the grid infrastructure More info: Application Description Template

Application Gridification24/26 Basic tasks while Porting applications to the Grid (contd.) 3. Grid environment access Requesting Certificates / VO membership Accessing Grid environment Appropriate VO UI machine account for command line Portal GUI account; 4. Executing, Testing and Debugging the application; Testing the non-grid application (debugging in Grid environment is a hard task), creating use cases for single (non-grid) runs; 5. Constructing the job suite – JDL (Job Description Language) files, executables, auxiliary scripts and input/output data files; 6. Submitting the job to the Grid as small-scale pilot application;

Application Gridification25/26 Basic tasks while Porting applications to the Grid (contd.) 7. Executing, Testing and Debugging the pilot application; 8. IF something goes wrong THEN GOTO 4; 9. IF everything seems to work THEN increase the scale of the application (increase problem size, amount of used resources); 10. Optimizing the grid application;