1. SC 2003 Demo, NCSA booth GridLab Project Funded by the EU (5+ M€), January 2002 – December 2004 Application and Testbed oriented Cactus Code, Triana Workflow, all the other applications that want to be Grid-enabled Main goal: to develop a Grid Application Toolkit (GAT) and set of grid services and tools...: resource management (GRMS), data management, monitoring, adaptive components, mobile user support, security services, portals,... and test them on a real testbed with real applications

2. SC 2003 Demo, NCSA booth GridLab Members n PSNC (Poznan) - coordination n AEI (Potsdam) n ZIB (Berlin) n Univ. of Lecce n Cardiff University n Vrije Univ. (Amsterdam) n SZTAKI (Budapest) n Masaryk Univ. (Brno) n NTUA (Athens) Sun Microsystems Compaq (HP) n ANL (Chicago, I. Foster) n ISI (LA, C.Kesselman) n UoWisconsin (M. Livny) collaborating with: Users! EU Astrophysics Network, DFN TiKSL/GriKSL NSF ASC Project other Grid projects Globus, Condor, GrADS, PROGRESS, GriPhyn/iVDGL, Most of the other European Grid Projects (GRIDSTART) GWEN

3. SC 2003 Demo, NCSA booth GridLab Aims Get Computational Scientists using the “Grid” and Grid services for real, everyday, production work (AEI Relativists, EU Network, Grav Wave Data Analysis, Cactus User Community), all the other potential grid apps Make it easier for applications to make flexible, efficient, robust, use of the resources available to their virtual organizations Dream up, prototype, and test new application scenarios which make adaptive, dynamic, wild, and futuristic uses of resources.

4. SC 2003 Demo, NCSA booth What GridLab isn’t We are not developing low level Grid infrastructure, Addressing Grids and P2P We do not want to repeat work which has already been done (want to incorporate and assimilate it …) Globus APIs, OGSA, ASC Portal (GridSphere/Orbiter), GPDK, GridPort, DataGrid, GriPhyn,...

5. SC 2003 Demo, NCSA booth GridLab End User Requirements Application oriented environment, Applications on one or more virtual organizations, Today we are able to run jobs between GridLab and Progress VOs Flexible, easy-to-use, simple interfaces resources, jobs, and data (including compiling, tracking jobs, cataloguing data), Means to make efficient and effective use of resources, Robustness smart adaptivity, complete control and fail safety are available on all levels, The ability to work in a disconnected environment, Mobile working,

6. SC 2003 Demo, NCSA booth What do our users want? Larger computational resources Memory/CPU Faster throughput Cleverer scheduling, configurable scheduling, co-scheduling, exploitation of un-used cycles Easier use of resources Portals, grid application frameworks, information services, mobile devices Remote interaction with simulations and data Notification, steering, visualization, data management Collaborative tools Notification, visualization, video conferencing, portals Dynamic applications, New scenarios Grid application frameworks connecting to services

7. SC 2003 Demo, NCSA booth GridLab end user requirements From laptops to fully deployed Virtual Organisations, Complexity hidden as much as possible, Collaborative infrastructure, The infrastructure for all classes of applications The infrastructure must provide capabilities to customise choice of service implementation (e.g. using efficiency, reliability, first succeeding, all)

8. SC 2003 Demo, NCSA booth Application Scenarios Dynamic Staging move to faster/cheaper/bigger machine Multiple Universe create clone to investigate steered parameter Automatic Convergence Testing from initial data or initiated during simulation Look Ahead spawn off and run coarser resolution to predict likely future Spawn Independent/Asynchronous Tasks send to cheaper machine, main simulation carries on Application Profiling best machine/queue choose resolution parameters based on queue Dynamic Load Balancing inhomogeneous loads multiple grids Portal User/VO interface to the grid Intelligent Parameter Surveys farm out to different machines Make use of Running with management tools such as Condor, Entropia, etc. Scripting thorns (management, launching new jobs, etc) Dynamic use of eg. MDS for finding available resources

9. SC 2003 Demo, NCSA booth Motivation for GAT Why do applications need a framework for using the Grid? Our application developers need a layer between applications and grid infrastructure: Higher level than existing grid APIs, hide complexity, abstract grid functionality through application oriented APIs Insulate against rapid evolution of grid infrastructure Choose between different grid infrastructures Make it possible for grid developers to develop new infrastructures Make it possible for application developers to use and develop for the grid independent of the state of deployment of the grid infrastructure

10. SC 2003 Demo, NCSA booth Solution... GAT – a layer between apps and emerging grid technologies GridLab testbed/VO Close cooperation between developers and deployers End Users GAT Tool Developers Grid Infrastructure Developers GAT-API Developers

11. SC 2003 Demo, NCSA booth GridLab Architecture

12. SC 2003 Demo, NCSA booth GridLab services Software environment for Grid-enabling scientific applications GridLab services, third party services and various core-grid services will be supported by GAT In the advent of the Open Grid Service Architecture (OGSA), GridLab's architecture will revolve around the notion of services, all the GridLab services will be OGSA compliant currently all the services are Web Services based roadmap for Web Services to OGSA transformation is being prepared (3-6 months from now)

13. SC 2003 Demo, NCSA booth GridLab services A primary aim of this project is to produce a GridLab GAT containing a set of high quality services which provide a complete environment for Grid-enabling generic applications GridLab services: implement common (strict) security use common service conneciton protocols (WSDL/OGSA) are built primarily for Globus infrastructure

14. SC 2003 Demo, NCSA booth What are the GL services? Authorisation Service (WP6) Adaptive Services (WP7) Data Management Services (WP8) Resource Management System (GRMS) (WP9) Information Services (WP10) Monitoring Services (WP11) Mobile User Support (WP12)

15. SC 2003 Demo, NCSA booth Security (WP6) Security WP focuses right now on the Authorization Service (AS) The main requirement is flexibility The AS is about to provide universal way of defining security policy for the whole Grid, independent of technologies used at lower levels It should be able to implement most security models for Grids and use many different scenarios at the same time

16. SC 2003 Demo, NCSA booth Adaptive Components (WP7) Adaptive Components Service (ACS) and the Local Adaptive Components (LAC). ACS provides an interface to query the adaptive system. It currently supports calls to: rank resources estimate transfer time estimate usage (of some given metric) LAC uses the monitoring system (shown in blue), to continuously collect data about the resource and applications running on it (load information, queue lengths, network bandwidth to other machines, etc.).

17. SC 2003 Demo, NCSA booth Data Management Services (WP8) replica catalog prototype was ready at Zakopane meeting data movement/copy service also since Zakopane meeting supports reliable gridftp file transfer is gsi enabled with authentication and delegation scalable and fault-tolerant replica catalog in M12, based on ongoing research at ZIB Soap Host A Host B Host C Host D automatic load-balance, fail-over between replica catalogs external access via SOAP and OGSA internal communication via more efficient protocol (Corba)

18. SC 2003 Demo, NCSA booth GRMS - the plan Job Receiver Jobs Queue BROKER Execution Unit Resource Discovery Scheduler Resource Reservation Prediction Unit File Transfer Unit Information Services Data Management Authorization System Adaptive Workflow Manager SLA Negotiation GRMS Monitoring GLOBUS, other Local Resources (Managers)

19. SC 2003 Demo, NCSA booth GridLab and Condor

20. SC 2003 Demo, NCSA booth GridLab and GriPhyN

21. SC 2003 Demo, NCSA booth Information Services Client Information Service (OGSA) SOAP over GSI Users Software Services Firewall V.O. C. A. Cluster Job Queues MDS GSI-SASL

22. SC 2003 Demo, NCSA booth Mercure Monitoring System Implements GGF’s GMA architecture Fast and robust Small resource usage Can monitor hosts and jobs Can deliver event notifications

23. SC 2003 Demo, NCSA booth Mobile User Support (WP12) Grid Services Network Environment / Grid Portal Applications User Mobile device

24. SC 2003 Demo, NCSA booth The Grid is complex … Monitoring Resource Management Information Security Data Management GLOBUS Application Manager Logging Notification Migration Profiling SOAPWSDLCorbaOGSAOther Other Grid Infrastructure? Application “Is there a better resource I could be using?” UNICORE

25. SC 2003 Demo, NCSA booth …need to make it easier to use GAT Application “Is there a better resource I could be using?” GAT_FindResource( ) The Grid

26. SC 2003 Demo, NCSA booth Grid Application Toolkit The GAT provides functionality through a carefully constructed set of generic high-level APIs, through which an application will be able to call the underlying grid services, Set of application developer APIs for Grid tools, services and software libraries, (and example implementations) that support the development of grid-enabled applications (open source!) Usable from any high level “application” (any generic code, Cactus, Triana, Portals, Scripts, …)

27. SC 2003 Demo, NCSA booth GAT: AIM Abstract Grid capabilities (services) from the application developer. Application developer concentrates on the functionality as needed by the application. Hide complexity. Provides a layer (buffer zone) between applications and the Grid.

28. SC 2003 Demo, NCSA booth GAT Goals The GAT provides an API and an associated set of tools which enable end-users and application developers to make easy and flexible use of the Grid, The infrastructure, and in particular the GAT, must allow developers to develop their applications independently of the deployment of grid services, Users must be able to make use of such applications in the absence of a fully-deployed infrastructure.

29. SC 2003 Demo, NCSA booth Cactus/GAT Integration GAT Library Cactus Flesh Thorn CGAT Thorn Physics and Computational Infrastructure Modules Cactus GAT wrappers Additional functionality Build system GridLab Service GridLab Service

30. SC 2003 Demo, NCSA booth GAT Adaptor Interface between GAT Engine and one or more capabilities Translates user requests to appropriate interface syntax for a capability provider Active adaptors change dynamically Includes “security context” Return appropriate error codes Examples OGSA adaptor (provides many capabilities) Globus adaptor (directly talk to gatekeepers) Adaptors for each GridLab service provider “Local” adaptors (GAT_MoveFile => “cp”, GATFindResource => “localhost”)

31. SC 2003 Demo, NCSA booth The Same Application … Application GAT Application GAT Application GAT Laptop The Grid Super Computer No network! Firewall issues!

32. SC 2003 Demo, NCSA booth Philosophy Application makes GAT API calls for operations which may be Grid-related. Application links against the GAT Engine Application runs irrespective of actual underlying infrastructure deployment Engine loads adaptors which are valid in the environment extant when the application starts Adaptors try to do Grid operations on request, on failure another adaptor provided function may be called. Application can thus be compiled, linked and tested without any Grid services Same application executable can run in a full Grid enviroment.

33. SC 2003 Demo, NCSA booth Philosophy The GAT uses whatever underlying Grid infrastructure there is and that people have developed adaptors for, GAT is not about replacing already developed infrastructure, but instead to provide a simple, clear interface which can be used with many different infrastructures. Different versions of Globus Condor Unicore...

34. SC 2003 Demo, NCSA booth The GAT Architecture GAT: Grid Application Toolkit API and Toolkit for developing portable Grid applications independently of the underlying Grid infrastructure and available services Implements the GAT-API Used by applications (different languages) GAT Adaptors Connect to capabilities/services GAT Engine Provides the function bindings for the GAT-API

35. SC 2003 Demo, NCSA booth API Goals The GAT must support applications written in any language which people write Grid Applications in: C, C++, Fortran, Java, Perl, Python,... The use of the GAT API should be as natural as possible for users of these languages. It must also not require a steep learning curve to move from the API in one language to the API in another language APIs in different languages should be as similar as possible

36. SC 2003 Demo, NCSA booth Migration Scenario Application migrates beacause of bad performance The Goal: Involve all the WPs! GAT application Portal GRMS Adaptive Monitoring GIS Mobile user support Security Data mgmt Testbed

37. SC 2003 Demo, NCSA booth Migration Scenario

38. SC 2003 Demo, NCSA booth Migration Scenario

39. SC 2003 Demo, NCSA booth Migration Scenario

40. SC 2003 Demo, NCSA booth Migration Scenario

41. SC 2003 Demo, NCSA booth The „behind the scenes” goal of GL „Let’s make the most advanced grid in the world” Michael Russell, AEI...and that’s what we do...

42. SC 2003 Demo, NCSA booth More info / summary www.GridLab.org gridlab@gridlab.orggridlab@gridlab.org, news@gridlab.orgnews@gridlab.org office@gridlab.org Check the GridLab tutorials available at the Web Bring your application and test it with the GAT and our services!