EGEE-II INFSO-RI Enabling Grids for E-sciencE Application specific portlet other portal features and next steps Miklos Kozlovszky MTA SZTAKI

Enabling Grids for E-sciencE EGEE-II INFSO-RI Develop an application specific portlet: –To hide application logic from end users: Separate grid experts from grid users Complete P-GRADE development cycle Develop your jobs locally Compile your jobs for the grid –Compiler portlet Compose workflows from your jobs and execute it –Workflow editor, workflow manager, Information system portlets Scale up the workflow to a parameter study –Define input files (or add Input file generator) –Add output file collector

Enabling Grids for E-sciencE EGEE-II INFSO-RI P-GRADE and user roles Portal Administrator: installs P-GRADE Portal 2.7 Grid Application Developer develops a grid application with P-GRADE Portal 2.7 sends the application to the grid portal developer End Users executes the published application with custom input parameters by creating application instances using the published application as a template Grid Portal Developer Creates new classes from the Application Specific Module for P-GRADE by changing the names of the classes develops a Gridsphere portlet that fits to the application I/O pattern and the end users’ needs connects the GUI to P-GRADE Portal using the programming API of P-GRADE Application Specific Module using the Application Specific Module he publishes the grid application and its GUI for end users

Enabling Grids for E-sciencE EGEE-II INFSO-RI Basic idea A new module called “Application specific module” has been developed for P-GRADE Portal that ease the creation of Application Specified Portlets –The module provides programming API to control P-GRADE portal applications –The module provides programming API to access P-GRADE Portal services Any kind of Graphical interface can be developed on top of this module –GUI written in JSP, JSTL (Java Servlet Tag Library ) and Java –GUI can be customized for the grid application –GUI can be customized for end users –GUI can benefit from all the P-GRADE Portal services

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application specific module 1.3 Concept Custom user interface (Written in Java, JSP, JSTL) Web browser EGEE and Globus Grid services (gLite WMS, LFC,…; Globus GRAM, …) Client P-GRADE Portal server Grid Services of P-GRADE Portal (workflow management, parameter study management, fault tolerance, …) Additional module for P-GRADE Core P-GRADE Portal You develop it

Enabling Grids for E-sciencE EGEE-II INFSO-RI P-GRADE PORTAL 2.7 server Services of P-GRADE Portal (workflow management, parameter study management, fault tolerance, …) Detailed Scenario Publisher Part of Application specific module Portlet to publish applications Registry of published applications EGEE and Globus Grid services (gLite WMS, LFC, …; Globus GRAM, GridFTP, …) Portal Administrator installs the P-GRADE portal 2.7. Then adjusts the portlet to publish applications

Enabling Grids for E-sciencE EGEE-II INFSO-RI P-GRADE PORTAL 2.7 server Services of P-GRADE Portal (workflow management, parameter study management, fault tolerance, …) Detailed Scenario Publisher Part of Application specific module Portlet to publish applications Registry of published applications Grid Application EGEE and Globus Grid services (gLite WMS, LFC, …; Globus GRAM, GridFTP, …) Grid Application Developer creates own Grid Application.

Enabling Grids for E-sciencE EGEE-II INFSO-RI P-GRADE PORTAL 2.7 server Services of P-GRADE Portal (workflow management, parameter study management, fault tolerance, …) Detailed Scenario Publisher Part of Application specific module Portlet to publish applications Registry of published applications Application specific portlet Grid Application EGEE and Globus Grid services (gLite WMS, LFC, …; Globus GRAM, GridFTP, …) Grid Portal Developer develops an Application specific portlet using Java Server Pages and Gridsphere Tag Library Updates the application specific package: connects Grid application to new portlet

Enabling Grids for E-sciencE EGEE-II INFSO-RI P-GRADE PORTAL 2.7 server Services of P-GRADE Portal (workflow management, parameter study management, fault tolerance, …) Detailed Scenario Publisher Part of Application specific module Portlet to publish applications Registry of published applications Application specific portlet Grid Application Grid Application (Template) EGEE and Globus Grid services (gLite WMS, LFC, …; Globus GRAM, GridFTP, …) Grid Portal Developer develops an Application specific portlet using Java Server Pages and Gridsphere Tag Library Updates the the application specific package: connects Grid application to new portlet

Enabling Grids for E-sciencE EGEE-II INFSO-RI P-GRADE PORTAL 2.7 server Services of P-GRADE Portal (workflow management, parameter study management, fault tolerance, …) Detailed Scenario Publisher Part of Application specific module Portlet to publish applications Registry of published applications Application specific portlet Grid Application Grid Application (Template) Grid Application EGEE and Globus Grid services (gLite WMS, LFC, …; Globus GRAM, GridFTP, …) Grid Application End users can create projects from a Published Grid Application Template. Then they can adjust parameters, and run in Grid via Application specific portlet!

Enabling Grids for E-sciencE EGEE-II INFSO-RI I nterface to publish a P-GRADE Application Grid Application Developer can publish the application by a pre-developed portlet Button to revoke an already published application Button to publish the application Select an application

EGEE-II INFSO-RI Enabling Grids for E-sciencE Example: Matrix multiplication on EGEE end users’ point of view Do you see the grid somewhere?

Enabling Grids for E-sciencE EGEE-II INFSO-RI Manage your computations Button to load a multiplication project Button to delete a multiplication project Button to create new multiplication project

Enabling Grids for E-sciencE EGEE-II INFSO-RI Define a new multiplication task Name of the current project Matrix A (you can define values for Y) Matrix B is fixed

Enabling Grids for E-sciencE EGEE-II INFSO-RI Define values for Y Parameters can be adjusted as Set, Range and Random.

Enabling Grids for E-sciencE EGEE-II INFSO-RI Monitor progress Click to start computation Click to get status of computation Status bar of computation

Enabling Grids for E-sciencE EGEE-II INFSO-RI Get result Click to Download Results

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Specific Portals The Rendering Portal Create a new rendering job by uploading file to be rendered Submit workflow Download rendered images Contact: Mr. Tamas Kiss,

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application name: E-marketplace Model Integrated with Logistics Description of application: –EMMIL facilitates three sided negotiation between buyers, sellers and third party logistics providers aimed at optimising the total costs that has never been offered before. The reason for the lack of integration in the present practice can be the algorithmic and computational complexity involved in the three-sided negotiation mechanism. Scientific impact of application: –Grid parallelism can help to perform the lengthy computations within a time period that is acceptable for the business environment. The example of EMMIL may encourage researchers to apply more sophisticated algorithms in their business models. Social impact of application: –Spreading the new model helps to optimise logistics in order to decrease waste of energy and consumption of natural resources and decrease the environmental pollution. Current user community: –The EMMIL P-GRADE Procurement Portal is used at International Business School for teaching purposes in the frame of “E-business” course (~100 students). Get result EMMIL P-GRADE Procurement Portal

Enabling Grids for E-sciencE EGEE-II INFSO-RI Download rendered images Set up the most optimal supplier chain Log in as buyer supplier, or logistic service provider. Create auctions or take your bids Parameter study optimization workflow is automatically generated and executed Best buyer-seller-logistic service provider pairs discovered, result presented in an Excel file Application Specific Portals (contd.) Supplier chain portal: EMMIL P-GRADE Procurement Portal

Enabling Grids for E-sciencE EGEE-II INFSO-RI Part II Support of legacy codes, code reusability and grid interoperability

Enabling Grids for E-sciencE EGEE-II INFSO-RI GEMLCA P-GRADE Portal P-GRADE Portal extended with GEMLCA Grid service back-end –To share jobs and legacy codes as application components with others –A step towards collaborative e-Science Developed by the University of Westminster (London) Support for Globus 4 grids (besides GT2 and EGEE) Available on the UK NGS and OGF GIN Portals P-GRADE Portal GEMLCA Globus 4 VOs Globus 2 VOs LCG / gLite VOs job Legacy code

Enabling Grids for E-sciencE EGEE-II INFSO-RI Combine legacy codes with new codes inside the same workflow! Code invocation Job submission Combining legacy and non-legacy (job) components Publish successfully executed new code as legacy code

Enabling Grids for E-sciencE EGEE-II INFSO-RI NGS P-GRADE Portal NGS P-GRADE portal is a production service of the UK NGS It has several advanced extensions to the original P-GRADE portal: –Integration with the GEMLCA legacy code service –Integration with SRB –Integration with OGSA-DAI The next part of the lecture shows these features of the NGS P-GRADE portal

Enabling Grids for E-sciencE EGEE-II INFSO-RI Sharing jobs with other GEMLCA portal users New button for sucessfully executed jobs: create a legacy code service from this jobs

Enabling Grids for E-sciencE EGEE-II INFSO-RI GEMLCA server Sharing jobs with other GEMLCA portal users Job Repository Put job into repository

Enabling Grids for E-sciencE EGEE-II INFSO-RI Workflow Level Data Access Grid Interoperation Problem Different grids use different distributed file systems: –EGEE:  SRM  Grid FTP –UK NGS, OSG:  Grid FTP  SRB (SDSC (San Diego SuperComputing) Storage Resource Broker) Problem: How to run –a workflow where different nodes can be executed in different grids –and data files can also be taken from different grids?

Enabling Grids for E-sciencE EGEE-II INFSO-RI J1 J2 J3 J4 J5 Grid 1 Grid 2 DB2 FS2 DB1 FS1 Workflow engine J: Job FS: File storage system, e.g. SRB or SRM DB: Database management system Intra-workflow interoperation of grid data resources Generic requirements

Enabling Grids for E-sciencE EGEE-II INFSO-RI portlets implemented 1.Data Service Manager 2.Database Browser 3.Query Manager 4.Data Manipulation Static OGSA-DAI integration a set of OGSA-DAI portlets –Implementation of a set of JSR 168 portlets –Support file delivery capabilities in order to transfer the data- sets before and after job execution

Enabling Grids for E-sciencE EGEE-II INFSO-RI GridFTP, SRM, SRB and OGSA-DAI workflow level integration Portal server Grid infrastructure Computing resources GridFTP servers REMOTE INPUT FILES REMOTE OUTPUT FILES LOCAL INPUT FILES LOCAL OUTPUT FILES LOCAL INPUT FILES LOCAL OUTPUT FILES User level storage EGEE Storage elements Control of remote input/output Workflow level Interoperation of local, GridFTP, SRM and SRB file catalogues and databases exposed by OGSA-DAI SRB servers OGSA- DAI services Data manipulation Input to workflows Output from workflows

Enabling Grids for E-sciencE EGEE-II INFSO-RI Lessons learnt P-GRADE portal became popular because it provides –Easy-to-use but powerful workflow system (graphical editor, wf manager, etc.) –Easy-to-use parameter sweep concept support –Easy-to-use MPI program execution support –Multi-grid/multi-VO access mechanism (job submission grid interoperation at workflow level) for LCG-2, gLite and GT2 Its extension with GEMLCA, SRB and OGSA-DAI enables –The usage of legacy codes as grid-enabled services –The usage of service/job repository –Access to SRB data resources and OGSA-DAI database services –Multi-grid/multi-VO access mechanism for LCG-2, gLite, GT2 and GT4 –Solving grid interoperation for grid data services

Enabling Grids for E-sciencE EGEE-II INFSO-RI Lessons learnt If you use P-GRADE portal you are not connected to any grid Your application can seamlessly run on many grid systems where you have valid certificate As the grid technology progress you do not have to port your application for the new grid middlewares (it is the portal that should support the new grid middlewares). The sustainability of P-GRADE portal is guaranteed since –It is OSS –There is a Portal Developer Alliance that continuously improve the portal The P-GRADE Portal Developer Alliance is open for everyone to join. See details at:

EGEE-II INFSO-RI Enabling Grids for E-sciencE Next steps How to get involved

Enabling Grids for E-sciencE EGEE-II INFSO-RI Portal installations P-GRADE Portal services: –SEE-GRID infrastructure –Central European VO of EGEE –GILDA: Training VO of EGEE –Many national Grids (UK National Grid Service, HunGrid, Turkish Grid, etc.) –US Open Science Grid, TeraGrid –OGF Grid Interoperability Now (GIN) VO –… Portal services and account request: Account request form on portal login page

Enabling Grids for E-sciencE EGEE-II INFSO-RI Installing P-GRADE Portal –(If you are the administrator of a Grid/VO then) get the installation package from SourceForge   Server requirements  Local user support must be provided by you –If you are member of an EGEE / Globus VO that does not have P-GRADE Portal yet, then  Ask SZTAKI to add your VO to the Multi-grid P-GRADE Portal Installation  Ask the manager of your VO to install P-GRADE Portal

Enabling Grids for E-sciencE EGEE-II INFSO-RI To configure your VO in Multi-grid P-GRADE Portal Send the following information to P-GRADE Portal team: Technical –WMS URL –BDII URL –LFC host URL –Default SE of the VO –VOMS server URL, certificate subject –VOMS server certificate –Extra CA packages if VO members use other than certificates. Administrative –Homepage of VO/community –Use cases for P-GRADE –Number of users

Enabling Grids for E-sciencE EGEE-II INFSO-RI Support services Training: –Training events, trainer community: list: –Portal to access GILDA resources –Online tutorials User forum Installation and operational support team Application Development and Porting Support:

EGEE-II INFSO-RI Enabling Grids for E-sciencE Thank you! Learn once, use everywhere Develop once, execute anywhere