Grid Programming Environment: Implementation & Globus Interface Ralf Ratering Thomas Kentemich Alexander Lukichev Igor Odintsov Grid Day in CGSL November 3, 2005

*Other brands and names are the property of their respective owners © Copyright 2005 Intel Corporation. All Rights Reserved. November Agenda  Intel GPE Component Overview  Intel GPE Implementation  Setting up a GPE Grid  Globus Interface  Current State and Next Steps

*Other brands and names are the property of their respective owners © Copyright 2005 Intel Corporation. All Rights Reserved. November Intel GPE Component Overview

*Other brands and names are the property of their respective owners © Copyright 2005 Intel Corporation. All Rights Reserved. November Application Programming Layer GPE Component Overview Applications FSI, Energy, Mfg, Telco, Pharma, Retail,... Resource Virtualization Layer Service Oriented Architecture/Infrastructure (SOA & SOI) Grid Middleware Compute Storage Network Operating System Compute Storage Network Operating System... ManagementSecurity Application Programming Layer Grid Programming Library Grid SDK Client FrameworkGridBeans Globus Toolkit- based Open Source implementation (GPE4GTK) Prototype Examples and Templates

*Other brands and names are the property of their respective owners © Copyright 2005 Intel Corporation. All Rights Reserved. November Grid Platform Grid Application I/F (WSRF, JSDL,...) Mgmt Console Application Programming Layer Grid Middleware Grid Platform I/F (WSRF, WSDM, JSDL, …) Workflow Engine Scheduler / Broker Target System Registry Database Access H/W Platform I/F (CIM & WS–based) Service OS Job Management Service (JMS) Target System Service (TSS) Storage Management Service (SMS) File Import Service (FIS) File Export Service (FES) Target System Factory (TSF) MRTE Virtual Machine OS Target System Interface (TSI) Virtual Machine OS Target System Interface (TSI) create & manage Application Deployment Service Application ClientWeb portals Grid Platform... Admin Client GPE Architecture (Fitting in Grid SW Architecture) = GPE components under development Hosting Environments and Server Implement. (UNICORE/GS, GTK4, …) Expert Client Firmware ( Platform S/W) Platform H/W & Platform *Ts (Compute, Storage, Network) Remote File Manager GridBeans = GPE components (not started) Firmware ( Platform S/W) Platform H/W & Platform *Ts (Compute, Storage, Network) Applications

*Other brands and names are the property of their respective owners © Copyright 2005 Intel Corporation. All Rights Reserved. November  OGSA (Open Grid Services Architecture) –Share components with other architectures  WS* (WS-Addressing, WSRF, WSN, etc.) –Interoperation with other Grid Middleware  JSDL (Job Submission Description Language) –High level job description that can be submitted to all target systems offering a JSDL interface  BPEL (Business Process Execution Language) –Integration of Grid Bean services into larger business process workflows  CIM (Common Information Model) –Used to describe resources –Usage of CIM management interfaces for Grid administration Emerging Standards

*Other brands and names are the property of their respective owners © Copyright 2005 Intel Corporation. All Rights Reserved. November Intel GPE Implementation

*Other brands and names are the property of their respective owners © Copyright 2005 Intel Corporation. All Rights Reserved. November Expert Client GPE Client Framework  Full access to Grid for expert users and administrators  Workflow Editor  Supports multiple applications and user identities Application Client  Light-weight Java application running on mobile devices  Simple application specific interface Portal Client  JSR-168 compliant application portlets that can be integrated into existing portals like UPortal, Jetspeed or GridSphere

*Other brands and names are the property of their respective owners © Copyright 2005 Intel Corporation. All Rights Reserved. November Expert Client Application Client Portal Client Write GridBeans for your applications GridBean Service GridBean GridBean SDK GPE Client API, libraries and tools GridBean  A GridBean dynamically integrates application-specific user interfaces in different clients

*Other brands and names are the property of their respective owners © Copyright 2005 Intel Corporation. All Rights Reserved. November Expert Client Application Client Portal Client GPE as interoperability framework UNICORE/GS Globus Toolkit 4 Other OGSA- compliant Grid servers China Grid Support Package SOAP, WSDL, WSRF, WS-Addressing, WS-Security, JSDL and other specifications Atomic Services Atomic Service Client API

*Other brands and names are the property of their respective owners © Copyright 2005 Intel Corporation. All Rights Reserved. November Atomic Services Overview Target System Factory (TSF)  Atomic service interfaces define basic set of operations and properties that have to be available on a Grid system –Operations describe the functionality –Properties describe the state (WS-Resource Framework)  Different implementations of interfaces for different infrastructures –Protocols, languages, security models,... Target System Service (TSS) Job Management Service (JMS) Storage Management Service (SMS) File Transfer Service (FTS) Manage files on storage Add a new target system to the Grid Manage transfers to/from storage Manage Jobs on target system Manage target system

*Other brands and names are the property of their respective owners © Copyright 2005 Intel Corporation. All Rights Reserved. November Job Management Service (JMS) Storage Management Service (SMS) File Transfer Service (FTS) OperationsServiceResource TypeProperties Atomic Service Interfaces based on WSRF Extends WS-RP/LT start() abort() hold() resume() Extends WS-RP/LT Job-EPR submit(JSDL, initialTT) Job3 Job2 Status=Running OriginalJSDL= ExecutionJSDL= etc. Job1 create Extends WS-RP/LT listDirectory(path) listFileProperties(path) createDirectory(path) changePermissions(path, perm) copy(path, path) rename(path, path) delete(path) import(path, protocol, isPipe) export(path, protocol, isPipe) Home Root AvailableFileSpace=127kb SupportedProtocols= {GridFTP, UPL, scp} etc. TS-Property Temp Work create Target System Factory (TSF) Target System Service (TSS) Target System Memory=1Gb Running Jobs=23 CPU Type=ia64 Storage=Temp, Root etc. create TSR-EPR create(config) Extends WS-RP/LT File Transfer 1 File Transfer 2 Sourcefile=c:/tmp/test.txt DestFile=/tmp/test.txt Transferred=300kb etc. create

*Other brands and names are the property of their respective owners © Copyright 2005 Intel Corporation. All Rights Reserved. November Hiding platform-specific information with Application Resources  Abstract job concept borrowed from UNICORE –No concrete platform specific information in job decription (paths, libraries, etc.) –Job will be incarnated on target system –For security and portability reasons!  Available applications can be queried from target system resource properties  Use JSDL Posix extensions to specify required application resources in submitted job

*Other brands and names are the property of their respective owners © Copyright 2005 Intel Corporation. All Rights Reserved. November Transparency Virtualization of the computational resources requires the job definitions to be abstract. Non-abstract definition: /usr/bin/povray /var/uspaces/ec82a0f0-4b7d-bf2585/povray.pov Abstract definition: povray.pov

*Other brands and names are the property of their respective owners © Copyright 2005 Intel Corporation. All Rights Reserved. November Abstract Job Definitions Advantages of abstract job definitions:  Easy to use with brokers  Don’t need to disclose private information about target system configuration  One step job submission

*Other brands and names are the property of their respective owners © Copyright 2005 Intel Corporation. All Rights Reserved. November File Transfers  Atomic Services support different protocols –FTP, GridFTP, plain HTTP(S), SOAP with Attachements, any other  Storage management announces available protocols via its resource properties –Client queries available protocols and selects appropriate one –GridFTP for large high-performance transfers –HTTPS or SOAP w/a to work with firewall limitations

*Other brands and names are the property of their respective owners © Copyright 2005 Intel Corporation. All Rights Reserved. November Client Advanced Services: Workflow Execution Workflow Execution Service (TSS) BPEL- enabled JSDL Target System Custom Web-service orchestrate  First implementation combining BPEL, JSDL and WSRF  Use Grid-specific BPEL subset –Allows integration into larger business processes  Implemented as WSRF service itself –Information about workflow (state, input/output files, etc.) is available through resource properties

*Other brands and names are the property of their respective owners © Copyright 2005 Intel Corporation. All Rights Reserved. November Advanced Services: Target System Registry  Registry keeps track of static and dynamic information –Hardware properties, available software,... –Workload, available disk space,...  TSS contacts registry on startup and when properties change (WS- Notification)  Clients and services query informations about target systems from registry Client Target System Register/ update Query Resource Broker Target System Registry

*Other brands and names are the property of their respective owners © Copyright 2005 Intel Corporation. All Rights Reserved. November GridBeans Internals GridBean Job Definition GPE Workflow Engine: Atomic Target System: Contact arbitrary web-services Perform XML evaluations Incarnate user’s task Perform computations

*Other brands and names are the property of their respective owners © Copyright 2005 Intel Corporation. All Rights Reserved. November GridBeans POVRay Visit Compiler Gaussian GridBean API Financial Simulation Interface GPE Client API GPE Client Framework Web Portals GPE Application Client GPE Expert Client Higher-level application services Engineering Applications Financial Simulations etc. Grid Programming API Easy to implement Portable Easy to maintain Advanced services Atomic Services China Grid Support Package Target System Registry UNICORE/GS Globus Toolkit Resource Broker Workflow Higher-level custom services take advantage of basic functionality provided by atomic and advanced services Optional deployment of user interfaces through GridBeans

*Other brands and names are the property of their respective owners © Copyright 2005 Intel Corporation. All Rights Reserved. November Setting up a GPE Grid

*Other brands and names are the property of their respective owners © Copyright 2005 Intel Corporation. All Rights Reserved. November Adding a target system to a Grid Target System Static Properties Incarnation Database (IDB) Target System Interface (TSI) Lightweight Java component to perform the actual work Service container Configuration file describing hardware and software properties Configuration file defining systems specific paths to commands, libraries, etc.  Start Target System Interface  Define Static Properties and Incarnation Rules  Start Service Container Atomic services

*Other brands and names are the property of their respective owners © Copyright 2005 Intel Corporation. All Rights Reserved. November Target System TSI Web Server Service container Target System TSI Target System TSI Atomic services Site A Target System TSI Service container Atomic services Target System TSI Service container Atomic services Site B Web Server Service container Advanced services Application services Site C A GPE Inter-Grid Clients

Globus Interface

*Other brands and names are the property of their respective owners © Copyright 2005 Intel Corporation. All Rights Reserved. November Why Globus?  De-facto standard, widely used  Powerful, fast evolving  Has enough features to host GPE services

*Other brands and names are the property of their respective owners © Copyright 2005 Intel Corporation. All Rights Reserved. November Differences between Globus and UNICORE/GS hosting environments Globus Toolkit 4UNICORE/GS Transport-level securityTLS v1 (SSL v3)SSL v2 DelegationProxy certificatesExtended Trust Delegation WSRF specsDraft v1.0Latest specs de-facto standard

*Other brands and names are the property of their respective owners © Copyright 2005 Intel Corporation. All Rights Reserved. November Portability GPE client core Globus client API implementation Unicore/GS client API implementation Globus atomic services interface GPE service core UNICORE/GS

*Other brands and names are the property of their respective owners © Copyright 2005 Intel Corporation. All Rights Reserved. November Incarnations Database povray +I +O +F +W … fields Fields are taken from the abstract job definition and are merged with the default values for user or application. The system is very flexibly configurable. GPE uses incarnations database to store the templates of user’s jobs to incarnate.

*Other brands and names are the property of their respective owners © Copyright 2005 Intel Corporation. All Rights Reserved. November Current State and Next Steps

*Other brands and names are the property of their respective owners © Copyright 2005 Intel Corporation. All Rights Reserved. November GPE & GPE4GTK  GPE Alpha version has been released at UNICORE SourceForge project on June 1, 2005 –Full working version for demos and developers  GPE-enhanced GTK4 implementation –SourceForge project „GPE4GTK“ –First release on October 28, 2005 –Beta release on December 31, 2005  UNICORE/GS support in European UniGridS project  Cooperation with China Grid to integrate GPE until end of 2006  Possible cooperation with SimDAT

*Other brands and names are the property of their respective owners © Copyright 2005 Intel Corporation. All Rights Reserved. November Summary Intel GPE......provides easy portable solution for different Grid infrastructures including UNICORE/GS and GTK4...provides a client framework to give users access to the infrastructure...provides the GridBean concept and a programming API for Grid developers...is available as Open Source today

Thank you !