Introduction to the Application Hosting Environment

Slides:



Advertisements
Similar presentations
TeraGrid Deployment Test of Grid Software JP Navarro TeraGrid Software Integration University of Chicago OGF 21 October 19, 2007.
Advertisements

GridSAM Overview Grid Job S ubmission A nd M onitoring Service What is GridSAM? Funded by the OMII Managed Programme (Started in Sept, 04) Client Perspective.
Legacy code support for commercial production Grids G.Terstyanszky, T. Kiss, T. Delaitre, S. Winter School of Informatics, University.
OMII-UK Steven Newhouse, Director. © 2 OMII-UK aims to provide software and support to enable a sustained future for the UK e-Science community and its.
CERN LCG Overview & Scaling challenges David Smith For LCG Deployment Group CERN HEPiX 2003, Vancouver.
Member of the ExperTeam Group Ralf Ratering Pallas GmbH Hermülheimer Straße Brühl, Germany
Grid Execution Management for Legacy Code Applications Exposing Application as Grid Services Porto, Portugal, 23 January 2007.
Minimum intrusion GRID. Build one to throw away … So, in a given time frame, plan to achieve something worthwhile in half the time, throw it away, then.
Minimum intrusion GRID. Build one to throw away … So, in a given time frame, plan to achieve something worthwhile in half the time, throw it away, then.
Grid Programming Environment (GPE) Grid Summer School, July 28, 2004 Ralf Ratering Intel - Parallel and Distributed Solutions Division (PDSD)
Presented by IBM developer Works ibm.com/developerworks/ 2006 January – April © 2006 IBM Corporation. Making the most of Creating Eclipse plug-ins.
70-290: MCSE Guide to Managing a Microsoft Windows Server 2003 Environment Chapter 8: Implementing and Managing Printers.
70-290: MCSE Guide to Managing a Microsoft Windows Server 2003 Environment, Enhanced Chapter 8: Implementing and Managing Printers.
MCTS Guide to Microsoft Windows Server 2008 Network Infrastructure Configuration Chapter 8 Introduction to Printers in a Windows Server 2008 Network.
70-290: MCSE Guide to Managing a Microsoft Windows Server 2003 Environment Chapter 8: Implementing and Managing Printers.
The SAM-Grid Fabric Services Gabriele Garzoglio (for the SAM-Grid team) Computing Division Fermilab.
- 1 - Grid Programming Environment (GPE) Ralf Ratering Intel Parallel and Distributed Solutions Division (PDSD)
From Campus Resources to Federated International Grids: Bridging the Gap with the Application Hosting Environment Joohyun Kim Center for Computation and.
Connecting OurGrid & GridSAM A Short Overview. Content Goals OurGrid: architecture overview OurGrid: short overview GridSAM: short overview GridSAM: example.
1 AHE Server Deployment and Hosting Applications Stefan Zasada University College London.
COMP3019 Coursework: Introduction to GridSAM Steve Crouch School of Electronics and Computer Science.
Condor Birdbath Web Service interface to Condor
1 Overview of the Application Hosting Environment Stefan Zasada University College London.
Contents 1.Introduction, architecture 2.Live demonstration 3.Extensibility.
CSF4 Meta-Scheduler Name: Zhaohui Ding, Xiaohui Wei
London e-Science Centre GridSAM Job Submission and Monitoring Web Service William Lee, Stephen McGough.
DIRAC Review (13 th December 2005)Stuart K. Paterson1 DIRAC Review Exposing DIRAC Functionality.
Grid Execution Management for Legacy Code Applications Grid Enabling Legacy Code Applications Tamas Kiss Centre for Parallel.
Ganga A quick tutorial Asterios Katsifodimos Trainer, University of Cyprus Nicosia, Feb 16, 2009.
Tool Integration with Data and Computation Grid GWE - “Grid Wizard Enterprise”
Giuseppe Codispoti INFN - Bologna Egee User ForumMarch 2th BOSS: the CMS interface for job summission, monitoring and bookkeeping W. Bacchi, P.
Styx Grid Services: Lightweight, easy-to-use middleware for e-Science Jon Blower Keith Haines Reading e-Science Centre, ESSC, University of Reading, RG6.
© Geodise Project, University of Southampton, Geodise Middleware & Optimisation Graeme Pound, Hakki Eres, Gang Xue & Matthew Fairman Summer 2003.
The NGS Grid Portal David Meredith NGS + Grid Technology Group, e-Science Centre, Daresbury Laboratory, UK
Grid Execution Management for Legacy Code Applications Grid Enabling Legacy Applications.
Interactive Workflows Branislav Šimo, Ondrej Habala, Ladislav Hluchý Institute of Informatics, Slovak Academy of Sciences.
Introduction to Taverna Online and Interaction service Aleksandra Pawlik University of Manchester.
Testing Grid Software on the Grid Steven Newhouse Deputy Director.
© Geodise Project, University of Southampton, Geodise Middleware Graeme Pound, Gang Xue & Matthew Fairman Summer 2003.
Standards driven AAA for Job Management within the OMII-UK distribution Steven Newhouse Director, OMII-UK
8 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. BI Publisher Server: Administration and Security.
Tool Integration with Data and Computation Grid “Grid Wizard 2”
SPI NIGHTLIES Alex Hodgkins. SPI nightlies  Build and test various software projects each night  Provide a nightlies summary page that displays all.
PROGRESS: GEW'2003 Using Resources of Multiple Grids with the Grid Service Provider Michał Kosiedowski.
The NGS Grid Portal David Meredith NGS + Grid Technology Group, e-Science Centre, Daresbury Laboratory, UK
IBM Express Runtime Quick Start Workshop © 2007 IBM Corporation Deploying a Solution.
INFSO-RI Enabling Grids for E-sciencE Ganga 4 Technical Overview Jakub T. Moscicki, CERN.
Grid Execution Management for Legacy Code Architecture Exposing legacy applications as Grid services: the GEMLCA approach Centre.
EGEE-II INFSO-RI Enabling Grids for E-sciencE A Glance Towards the Future Mike Mineter Training Outreach and Education University.
Active-HDL Server Farm Course 11. All materials updated on: September 30, 2004 Outline 1.Introduction 2.Advantages 3.Requirements 4.Installation 5.Architecture.
V7 Foundation Series Vignette Education Services.
InSilicoLab – Grid Environment for Supporting Numerical Experiments in Chemistry Joanna Kocot, Daniel Harężlak, Klemens Noga, Mariusz Sterzel, Tomasz Szepieniec.
Grid Execution Management for Legacy Code Applications Grid Enabling Legacy Applications.
NAREGI PSE with ACS S.Kawata 1, H.Usami 2, M.Yamada 3, Y.Miyahara 3, Y.Hayase 4 1 Utsunomiya University 2 National Institute of Informatics 3 FUJITSU Limited.
Packaging and Deploying Windows Applications
Contents Software components All users in one location:
Reading e-Science Centre
Database System Concepts and Architecture
Autonomic Resource Virtualization in Cloud-like Environments A
How to connect your DG to EDGeS? Zoltán Farkas, MTA SZTAKI
BOSS: the CMS interface for job summission, monitoring and bookkeeping
BOSS: the CMS interface for job summission, monitoring and bookkeeping
BOSS: the CMS interface for job summission, monitoring and bookkeeping
Grid Portal Services IeSE (the Integrated e-Science Environment)
Interoperability & Standards
Copyright © 2011 Pearson Education, Inc. Publishing as Pearson Addison-Wesley Chapter 2 Database System Concepts and Architecture.
Module 01 ETICS Overview ETICS Online Tutorials
From Prototype to Production Grid
GRID Workload Management System for CMS fall production
Introduction to the SHIWA Simulation Platform EGI User Forum,
Presentation transcript:

Introduction to the Application Hosting Environment Stefan Zasada

Contents Motivation for the AHE Concepts & functionality Meeting the AHE design constraints Architecture of the AHE AHE client interaction Deploying the AHE

Motivation for the AHE Problems with current middleware solutions: Difficult for an end user to configure and/or install Dependent on lots of supporting software also being installed Require modified versions of common libraries Require non-standard ports to be opened on firewall Large footprint – memory/disk space

The Application Hosting Environment Based on the idea of applications as web services Lightweight hosting environment for running unmodified applications on grid resources (NGS, TeraGrid) and on local resources (departmental clusters) Community model: expert user installs and configures an application and uses the AHE to share it with others Simple clients with very limited dependencies No intrusion onto target grid resources

Virtualizing Applications Application Instance/Simulation is central entity; represented by a stateful WS-Resource. State properties include: simulation owner target grid resource job ID simulation input files and urls simulation output files and urls job status Application exposed as web service clients and end-users deal with applications as services and application instances, not jobs higher level of abstraction: applications are first class entities in the architecture applications may be multi-component (need several jobs to instantiate an application) some applications may be interactive (e.g. steerable) WSRF provides clean interface, web services communicate with resource services which allow data to be stored and retrieved

AHE Functionality Launch simulations on multiple grid resources Single interface to monitor and manipulate all simulations launched on the various grid resource Run simulations without manually having to stage files and GSISSH in Retrieve files to local machine when simulation is done Can use a combination of different clients – PDA, desktop GUI, command line

Accessing Resources NGS UK NGS Local UCL resources DEISA GridSAM/ Leeds Manchester Oxford RAL HPCx UK NGS NGS GridSAM/ Globus Local UCL resources DEISA GridSAM/ SGE GridSAM/ UNICORE

AHE Design Constraints Client does not have Globus installed locally Client is NAT'd and firewalled Client does not have to be a single machine Client needs to be able to upload and download files but doesn’t have local installation of GridFTP Client doesn’t maintain information on how to run the application Client doesn’t care about changes to the backend resources

Meeting the Constraints AHE Client behind firewall => polls server to update job state etc. Uses intermediate filestaging area => GridFTP not installed All application specific information for running simulations on the grid resource is maintained on a central service => user can switch clients etc. Location of binary on grid resource configured on server => user doesn’t need to know GridSAM provides interface to job queue

Layered Architecture of the AHE AHE client installed on users machine - can be combination of GUI, command line and PDA 2) Client interacts with AHE constituent web services. Services include a registry of installed applications and a registry of previously instantiated applications. Each time an application is launched, a new stateful web service is created to represent the application instance. The user interacts with this web service to monitor, terminated the job etc 3) The AHE sends a Job Submission Description Language (JSDL) document to the GridSAM job submission service, representing the application run. Each machine that the AHE is submitting on requires one GridSAM instance for submission. 4) GridSAM submits the job to the machine’s DRM (currently GridSAM plug-ins exist for Grid Engine, Globus and Condor, and a UNICORE plug-in is under development by the DEISA middleware team).

Service Architecture of the AHE

AHE Server Implementation WSRF::Lite => services developed in Perl WebDAV server GridSAM => Globus grid => Sun Grid Engine => Condor pool => Unicore MyProxy PostgreSQL database Apache/Tomcat container

AHE Server Deployment The expert user must: Set up container to host services: Apache/WSRF::Lite or modified Tomcat/WSRF::Lite Set up PostgreSQL database and WebDAV server If not already running set up GridSAM instance for grid resource Deploy and configure the AHE services in the container OMII stack installer will do all of this automatically Once deployed, any number of applications can be hosted

Hosting a New Application Expert user must: Install and configure application on all resources on which it is being shared Create a JSDL template for the application (easily cloned from exiting template) Add the application to the RMInfo.xml file Run a script to reread the configuration Documentation covers whole process of deploying AHE & applications on NGS and TeraGrid

Client Implementation GUI & command line clients implemented in Java Client allows user to: Discover appropriate resources Launch application Monitor running jobs Query registry of running jobs Stage files to and from resource Terminate jobs GUI client implements application launching as a wizard

Client Extensibility Plugins can be added to process application input files to automatically discover the input and output files that need to be staged If no plugin is available then a default case will allow users to specify input and output files manually Plugins implement AHEConfParser interface and follow specific naming convention Plugin .class files dropped into plug-in directory and picked up by GUI/command line clients

AHE Client Deployment Deploying client is trivial for the end user: User’s machine must have Java installed User downloads and untars client package Imports X.509 certificate into Java keystore using provided script Configures client with endpoints of AHE services supplied by expert user Ready to go!

Constructing workflows with the AHE By calling command line clients from Perl script complex workflows can be achieved Easily create chained or ensemble simulations E.g. HIV equilibration protocol implemented by: ahe-prepare  prepare a new simulation for the first step ahe-start  start the step ahe-monitor  poll until step complete ahe-getoutput  download output files repeat for next step

Current Deployed Applications Currently hosting: NAMD LAMMPS DL_POLY LB3D Gromacs CHARMM Plan to host: Trubal POLCOMS

Summary The AHE provides a lightweight, easily deployable environment for running unmodified scientific applications on the grid and local resources The AHE server is designed to be deployed by an expert user who uses it to share applications installed on grid resources The client is easily installed by any end user, requiring no intervention by system/network administrators By calling the command line clients from scripts, complex scientific workflows can be implemented

Any Questions? Released in OMII 3.2.0 http://www.omii.ac.uk/downloads/ RealityGrid web site: http://www.realitygrid.org/AHE NeSCForge: http://forge.nesc.ac.uk/projects/ahe/ Mailing list: http://www.mailinglists.ucl.ac.uk/mailman/listinfo/ahe- discuss

Exercise 1 Installing and configuring the AHE client Tasks: Outcome Install the AHE client on your system Set up a keystore containing your grid certificate Configure the client with settings for UCL’s AHE server Confirm that the client is installed and working Outcome Installed and configured AHE client

Exercise 2 Launching an Application on the NGS using the AHE Tasks: Launch the sort application with the AHE GUI client Launch the sort application with the AHE command line client Manually specify input and output files for an application Retrieve application output from NGS machine Outcome Successfully run applications on NGS machines