Porting applications to EU-IndiaGrid: EGEE Marco Verlato EU-IndiaGrid Workshop April 2007 Bangalore, India

2 Outline Concepts Job categories Levels of Gridification Software remote installation Applications invocation

3 Grid Applications Development Where computer science meets the application communities! VO-specific developments built on higher-level tools and core services Makes Grid services useable by non-specialists Grids provide the compute and data storage resources Basic Grid services: AA, job submission, info, … Higher-level grid services (brokering,…) Application toolkits, ….. Application Production grids provide these core services.

4 Job concept gLite middleware follows the job submission concept for application execution and resource sharing A job is a self contained entity that packages and conveys all the required information and artifacts for the successful remote execution of an application −Executable files −Input/Output data −Parameters −Environment −Infrastructure Requirements −Workflows

5 Type of applications The work done to port the application depends on its characteristics, complexity and the level of interactions with external grid services Size of input/output data Metadata request Access to external software (database, libraries...)

6 Applications categories Simulation Bulk Processing Responsive Apps. Workflow Parallel Jobs

7 Simulation Examples –LHC Monte Carlo simulation –Fusion Characteristics –Jobs are CPU-intensive –Large number of independent jobs –Run by few (expert) users –Small input; large output Needs –Batch-system services –Minimal data management for storage of results ATLAS ITER

8 Bulk Processing Examples –HEP processing of raw data, analysis –Earth observation data processing Characteristics –Widely-distributed input data –Significant amount of input and output data Needs –Job management tools (workload management) –Meta-data services –More sophisticated data management

9 Responsive Apps. (I) Examples –Prototyping new applications –Monitoring grid operations Characteristics –Small amounts of input and output data –Not CPU-intensive –Short response time (few minutes) Needs –Configuration which allows “immediate” execution (QoS) –Services must treat jobs with minimum latency

10 Responsive Apps. (II) Grid as a backend infrastructure: –gPTM3D: interactive analysis of medical images bioinformatics via web portal –GATE: radiotherapy planning –DILIGENT: digital libraries Characteristics –Rapid response: a human waiting for the result! –Many small but CPU-intensive tasks –User is not aware of “grid”! Needs –Interfacing (data & computing) with non-grid application or portal –User and rights management between front-end and grid

11 Workflow Examples –“Bronze Standard”: medical images registration –Flood prediction Characteristics –Use of grid and non-grid services –Complex set of algorithms for the analysis –Complex dependencies between individual tasks Needs –Tools for managing the workflow itself –Standard interfaces for services (I.e. web- services)

12 Parallel Jobs Examples –Climate modeling –Earthquake analysis –Computational chemistry Characteristics –Many interdependent, communicating tasks –Many CPUs needed simultaneously –Use of MPI libraries Needs –Configuration of resources for flexible use of MPI –Pre-installation of optimized MPI libraries

13 Universal Needs Security –Ability to control access to services and to data Fine-grained access control lists Encryption & logging for more demanding disciplines Access control consistently implemented over all services VO Management –Management of users, groups, and roles –Changing the priority of jobs for different users, groups, roles –Quota management for users, groups, roles –Definition and access to special resources Application-level services Responsive queues (guaranteed, low-latency execution)

14 Levels of “Gridification” No development: wrap existing applications as jobs. No source code modification is required Minor modifications: the application exposes minimal interaction with the grid services (e.g. Data Managements) Major modifications: a wide portion of the code is rewritten to adopt to the new environment (e.g. parallelization, metadata, information) Pure grid applications: developed from scratch. Extensively exploit existing grid services to provide new capabilities customized for a specific domain (e.g. metadata, job management, credential management )

15 No development It is the simplest case : the application has not external dependencies, just needs computational power Being sure that the application runs on the Worker Node, the executable, and the input files it may need are carried in the Input Sandbox and therefore executed Although the application can be directly run, it’s recommended to invoke it from a supplied script which prepare the environment before the execution

16 Minor modifications The application has minimal interactions with grid service : it is self consistent but... – e.g. takes input files from an SE – output files are stored on a SE – Perform I/O from a metadata catalog – the application itself can be downloaded from a SE User provides an adapter script which executes needed tasks, set the environment and run the application

17 Major modifications A part of the application software is rewritten taking care of the new environment : e.g. because of parallelization, interaction at run time with grid services... – Use of API – Web Service invoked – CLI wrap Source code recompiled taking into account this

18 Pure grid application Entirely developed from scratch taking into account grid capabilities Better exploits grid facilities Needs a deep knowledge of grid environment and possibilities in order to engineer entirely the application logic over a grid environment Very rare to meet...

19 Additional software installed Independently from the integration level, applications may need external software (libraries) that, as application itself, can be provided in several ways – at run time carrying out it within InputSandbox downloading it from a Storage Element – permanent install on the CE as tar.gz installed by the VO software manager as rpm being part of the gLite release

20 The Experiment Software Manager (ESM) is the member of the experiment VO entitled to install Application Software in the different sites. The ESM is generally acknowledged through a VOMS role (SoftwareManager) The ESM can manage (install, validate, remove...) Experiment Software on a site at any time through a normal Grid job, without previous communication to the site administrators. Such job has in general no scheduling priorities and will be treated as any other job of the same VO in the same queue. lcg-asis is a good generator for such installation jobs Software installation on CE

21 Software installation on CE The site provides a dedicated space where each supported VO can install or remove software. The amount of available space must be negotiated between the VO and the site An environmental variable holds the path for the location of a such space. Its format is the following: VO_ _SW_DIR

22 Software validation Once the software is installed, the ESM can perform the validation The validation is a process verifying that the software was properly installed and works as expected It can be performed in the same job, after installation (validation on the fly) or later on, via a dedicated job.

23 Software validation If the ESM judges that the software was properly installed he publishes in the Information System (IS) the tag which identifies unequivocally the software, e.g.: VO-cms-CMSSW_1_2_2 Such a tag is added to the GlueHostApplicationSoftwareRunTimeEnvironment attribute of the IS, using the GRIS running in the CE Jobs requesting for a particular piece of software can be directed to the appropriate CE just by setting special requirements on the JDL, e.g.: Requirements=Member("VO-cms-CMSSW_1_2_2", other.GlueHostApplicationSoftwareRunTimeEnvironment);

24 Invoke Application From the UI −Command Line Interfaces / Scripts −APIs −Higher level tools From portals −Accessible from any browser −Tailored to applications

25 References (I)

26 References (II)

27 Acknowledgements Thanks to Mike Mineter (NeSC), Valeria Ardizzone and Emidio Giorgio of the INFN GILDA team