1. www.cactuscode.org www.gridlab.org Developing Applications on Today’s Grids Tom Goodale Max Planck Institute for Gravitational Physics goodale@aei.mpg.de

2. Grid Apps 2 Grid Projects n Globus n GrADS n Condor n GridLab n DataGrid Many more, see for example: n http://www-fp.mcs.anl.gov/~foster/grid-projects/ n http://www.gridcomputing.com/

3. Grid Apps 3 Globus n http://www.globus.org n Large and established project which has contributed much Grid middleware n Based at Argonne National Laboratories (USA) n Globus is the most widely deployed software for Grid computing n “The Globus Project is developing fundamental technologies needed to build computational grids.”

4. Grid Apps 4 GrADS n http://hipersoft.cs.rice.edu/grads/ n Grid Application Development Software n “The goal of the Grid Application Development Software (GrADS) Project is to simplify distributed heterogeneous computing in the same way that the World Wide Web simplified information sharing over the Internet. The GrADS project will explore the scientific and technical problems that must be solved to make grid application development and performance tuning for real applications an everyday practice.”

5. Grid Apps 5 Condor n http://www.cs.wisc.edu/condor/ n “The goal of the Condor Project is to develop, implement, deploy, and evaluate mechanisms and policies that support High Throughput Computing (HTC) on large collections of distributively owned computing resources. Guided by both the technological and sociological challenges of such a computing environment, the Condor Team has been building software tools that enable scientists and engineers to increase their computing throughput. “

6. Grid Apps 6 GridLab n http://www.gridlab.org n “ The GridLab project will develop an easy-to-use, flexible, generic and modular Grid Application Toolkit (GAT), enabling todays applications to make innovative use of global computing resources. The project is grounded by two principles, (i) the co-development of infrastructure with real applications and user communities, leading to working scenarios, and (ii) dynamic use of grids, with self- aware simulations adapting to their changing environment.”

7. Grid Apps 7 DataGrid n http://eu-datagrid.web.cern.ch/eu-datagrid/ n “DataGrid is a project funded by European Union. The objective is to build the next generation computing infrastructure providing intensive computation and analysis of shared large-scale databases, from hundreds of TeraBytes to PetaBytes, across widely distributed scientific communities.”

8. Grid Apps 8... n There are many more projects – the preceding was just a sample n More projects are starting all the time n The “Grid” is gaining interest...

9. Grid Apps 9 Grid Infrastructure n There is already a lot of infrastructure out there to help one run applications on grids n Not so much infrastructure so far for tailoring applications to run on grids, but that doesn't stop existing legacy applications being able to run in grid environments. n Lots of effort currently underway to develop portals to facilitate use of the existing infrastructure with existing applications.

10. Grid Apps 10 Globus n Globus project has developed the most widely deployed grid infrastructure n This infrastructure splits into l Security l Data management l Resource management l Information systems n Additionally the project have developed an MPI implementation which helps MPI applications to run across multiple computational resources.

11. Grid Apps 11 GSI n The Globus Toolkit uses the Grid Security Infrastructure for enabling secure authentication and communication over an open network. GSI provides a number of useful services for Grids, including mutual authentication and single sign-on. n GSI uses public key encryption and X.509 certificates, along with SSL, with extensions to allow single sign on and credential delegation. n Globus implementation GSSAPI compliant

12. Grid Apps 12 Data Management n The Globus Toolkit includes various data management components n GridFTP l GSI enabled FTP including multiple parallel streams to increase overall throughput n Data Replication l Multiple copies of data distributed to allow faster access l Toolkit provides replica catalogue and replica management software n GASS l Global Access to Secondary Sources l Can access data from anywhere with a URL

13. Grid Apps 13 Resource Management n Globus Resource Allocation Manager (GRAM) n GRAM processes the requests for resources for remote application execution, allocates the required resources, and manages the active jobs. It also returns updated information regarding the capabilities and availability of the computing resources to the Monitoring and Discovery Service (MDS). n GRAM provides an API for submitting and canceling a job request, as well as checking the status of a submitted job. The specifications are written by the user in the Resource Specification Language (RSL), and is processed by GRAM as part of the job request.

14. Grid Apps 14 Information Systems n Monitoring and Discovery Service (MDS) n The MDS contains static and dynamic information about compute resources, as well as static and dynamic information about the network performance between compute resources. n LDAP based database n Hierarchical

15. Grid Apps 15 MPICH-G2 n Can be used to run across multiple distributed parallel resources n Based on the widely available MPICH MPI implementation from Argonne – in fact a standard device which may be built if globus is installed on the system n May use vendor's native MPI implementation for intra- machine communication n Uses Globus infrastructure to launch jobs on remote resources

16. Grid Apps 16 Condor n Condor converts collections of distributively owned workstations and dedicated clusters into a distributed high-throughput computing facility. n Uses ClassAds to specify resource requirements for jobs. n Contains checkpointing and process migration n Can use Globus to be batch system across multiple resources

17. Grid Apps 17 How Do Applications Use These ? n To use Condor or MPICH-G2 no changes need to be made to the application to make use of new distributed features n So simple applications used in their current mode may be able to use infrastructure transparently. n More complicated applications have more needs...

18. Grid Apps 18 What is an Application ? n Sometimes causes much confusion in conversations n Is an application a single process, or many processes collaborating to perform some task ? n For the purposes of this tutorial I will define an application as the latter n An application is one or more processes which perform a particular task such as a simulation or a calculation on behalf of a user n E.g. all processes in an MPI Job

19. Grid Apps 19 Application Developers n This is very similar to the requirements for an application to be able to run on many different architectures n Need now to also think that not all processes in an application are necessarily running on the same resource or even the same architecture n Not all processes have access to the same environment, or may be able to reach the same set of remote resources What do Application Developers Need to Think About in Grid Environments ?

20. Grid Apps 20 IO n As discussed for frameworks, files must be in some format which is readable on all architectures n Not all processes may have access to the same file systems, so may need to use communication technologies to access files remotely n The user may not ordinarily have access to any of the filespaces accessible to the application, so there must be some way to migrate files to and from the space available to the application.

21. Grid Apps 21 Parallel Issues n If using MPI, must be an MPI version which can run heterogeneously l e.g. MPICH-G2, PACX. n When running across multiple resources, the bandwidth and latencies of communication between processes on different resources is much greater than between processes on a single resource l Need to think about communication patterns – is it possible to reduce the amount of communication by, for example, buffering data for longer and sending larger batches of data.

22. Grid Apps 22 Inter Process Communication n Need to locate other processes in application n These may be on remote resources n Remote resources may be firewalled

23. Grid Apps 23 Portability n Need to be able to compile and run in heterogeneous environments n Not all resources have the same sets of software available n When starting a distributed application, how does one make sure that there is a suitable executable there ? n Should base code on standards, not on individual compiler vendors' specific features.

24. Grid Apps 24 Firewalls n In the modern world a lot of resources are protected by firewalls. These restrict the ports which may be access from the outside world, and often the locations in the outside world from which these ports may be opened. n Not generally a problem for an application running on this resource n A real problem for monitoring such an application n A real problem for running an application across multiple such resources

25. Grid Apps 25 Testbeds - What and Why ? n A testbed is a (heterogeneous) set of machines which you may test your application on. n May or may not have a uniform distribution of grid infrastructure. n Why use one ? l Set of resources which you can find out about and have accounts on. l Can ask the sysadmins what went wrong. l Can request installation of other software l Can thus test your application in a Grid environment with less pain than on a random set of machines

26. Grid Apps 26 Grid Programming Tools n While there are many Grid projects, and much grid middleware, there is, to date, very little in the way of toolkits which make it easy for an application developer to write an application which makes full use of the possibilities of the Grid. n Both MPICH-G2 and Condor allow specific classes of applications to make use of the power of the grid to run distributed applications, however access to resource and data management is still hard to do from an application, and IPC for distributed application is still hard.

27. Grid Apps 27 GAT - What ? n The Grid Application Toolkit (GAT), which is currently being developed by the GridLab project aims to make this easier n The GAT aims to develop an API to enable application developers to make use of the best Grid infrastructure when and as it becomes available n The GAT API allows access to “fundamental grid operations” n The GAT abstracts these operations allowing access to alternative implementations or instances of entities providing these operations

28. Grid Apps 28 GAT - Why ? n People want to use the Grid n However they don't want to have to learn all about the various Grid technologies n Users want to just submit a job and get results back n Application developers want to be able to write applications which can access Grid resources and run in a Grid environment; they don’t want to have to rewrite parts of their application when new technologies come along n Want to be able to have applications developed today, so they can use the Grid as it emerges. n Provides a “buffer zone” between applications and the Grid.

29. Grid Apps 29 The Grid is complex … Monitoring Resource Management Information Security Data Management GLOBUS Application Manager Logging Notification Migration Profiling SOAPWSDLCorbaOGSAOther Other Grid Infrastructure? Cactus “Is there a better resource I could be using?”

30. Grid Apps 30 …need to make it easier to use GAT Cactus “Is there a better resource I could be using?” GAT_FindResource( ) The Grid

31. Grid Apps 31 GridLab Architecture n The GridLab architecture is split in several pieces: n The application itself n A library which interfaces between the application and the Grid middleware n The Grid middleware. n The GridLab project aims to develop the library – the GAT Engine - and a set of middleware – GridLab services.

32. Grid Apps 32 GAT: What is It? GAT: Grid Application Toolkit n Implements the GAT-API l Used by applications (different languages) n GAT Adaptors l Connect to capabilities/services n GAT Engine l Provides the function bindings for the GAT-API

33. Grid Apps 33 The GAT Engine n This is a library which applications link against to make use of Grid infrastructure n It provides stub calls for the basic Grid operations n Applications can always make calls to any of these operations, and will get an error back if it is not available. n Thus an application need not be re-written, recompiled or re-linked to make use of new middleware. n The actual access to middleware is provided by dynamically loadable modules which provide access to specific implementations of these grid operations

34. Grid Apps 34 GAT Engine When an application makes a GAT-API call, the engine searches through an internal database of adaptors for the requested capability and calls it

35. Grid Apps 35 GAT Adaptor n Interface between GAT Engine and one or more capabilities l Translates user requests to appropriate interface syntax for a capability provider l Active adaptors change dynamically l Includes “security context” l Return appropriate error codes n Examples l OGSA adaptor (provides many capabilities) l GRAM adaptor (directly talk to gatekeepers) l Adaptors for each GridLab service provider l “Local” adaptors (GAT_MoveFile => “cp”, GATFindResource => “localhost”)

36. Grid Apps 36 GAT Adaptor

37. Grid Apps 37 GAT Adaptor Initialisation

38. Grid Apps 38 GAT Adaptor Call (I)

39. Grid Apps 39 GAT Adaptor Call (ii)

40. Grid Apps 40 Current Status n Many GridLab services in development or available in prototype or alpha release l Data management, resource brokering, application monitoring, information systems, access for mobile devices n GridLab portal under development n GAT Engine available in prototype form l Usable with a test API l Allows access to available GridLab services and some other Grid middleware n Grid Operations for GAT API being identified and codified and actual API being developed.

41. Grid Apps 41 The Same Application … Application GAT Application GAT Application GAT Laptop The Grid Super Computer No network! Firewall issues!

42. Grid Apps 42 Getting Ready For The Grid n Grid Toolkits and middleware aren't a magic wand which will 'grid enable' your application. You still need to think ! n Unless your application is very simple, or makes use of a framework, you are very likely to need to modify it. n Use standards ! l This is basic to any portable application n If you want to locate data you need to be able to describe it. l The grid will not magically decrease data processing time for data intensive applications unless the data can be described adequately.

43. Grid Apps 43 Getting Ready... n As said before, simple applications can be 'grid enabled' in a basic way by use of MPICH-G2 or Condor. n In fact any basic MPI application is 'grid enabled' ! However it may need modification to run optimally in a Grid environment.

44. Grid Apps 44 Various Scenarios n Application steering l This should be done in some standard way so that in the future it can be replaced by some actuators from some toolkit which gives authentication and authorisation n Checkpointing and IO l Should be in some standard file format l May want to advertise files to some data managements system n Visualisation l Again standard file or data formats allow grid middleware to operate l Can be linked to file advertising

45. Grid Apps 45 What Frameworks and Toolkits Give You n Frameworks such as Cactus give you a lot of these things. Using such a framework frees you as an application developer from having to worry about a lot of issues; the framework developers have hopefully done it instead. n Similarly toolkits such as the GAT free you from having to worry about specific Grid infrastructure. All the access to Grid middleware, and worrying about how it is deployed can be delegated to the toolkit. n Using frameworks and toolkits frees you from having to worry about a lot of generic things, leaving you more time and energy to work on application specific things.

46. Grid Apps 46 Example n A simple of example of an GAT enabled application will be presented and run. n This will be available from http://www.gridlab.org/WorkPackages/wp-1/Examples