1. TeraGrid Software Integration: Area Overview (detailed in 2007 Annual Report Section 3) Lee Liming, JP Navarro TeraGrid Annual Project Review April, 2008

2. Software Integration (SI) Activities GIG Software Integration –Area Coordination –Operations Define and maintain six CTSS capability definitions CTSS software packaging Operate infrastructure services –Development Information Services TeraGrid Client Software Distribution Application Hosting Prototype Instrumentation Resource Providers –CTSS Deployment & Maintenance –RP Capability Development April 20082Software Integration Overview

3. Coordinated TeraGrid Software & Services (CTSS) What is CTSS? The collection of capabilities that TeraGrid resource providers coordinate across their compute and storage resources Examples: –Remote job submission interface –Compilers, some development libraries and tools –Data movement CTSS is NOT: –All the software available across all TeraGrid resources –Software used in centralized TeraGrid services (User Portal, TeraGrid website, TeraGrid central accounting and allocations, gateway software) –User, application, or community software History: CTSS 1: DTF construction and initial production common software and services (2003) CTSS 2: ETF common software and services (2004) CTSS 3: Incremental evolution (with Globus 4) common software and services (2006) CTSS 4: A new paradigm (2007) April 20083Software Integration Overview

4. 4 CTSS 4 Capability Kits Capability KitDescription TeraGrid Core IntegrationMinimal components that integrate RP resources into the TeraGrid Remote LoginRemote login using TeraGrid credentials to a coordinated Unix environment Remote ComputeRemote job submission Application Development and Runtime Compile and execute applications Data MovementRemote data movement to/from RP resources Parallel ApplicationCompile and execute parallel (MPI) applications Science WorkflowRun an orchestrated collection of interdependent jobs Wide Area GPFSLocal access to TeraGrid wide GPFS file-systems VisualizationCompile and execute visualization applications Availability: First kit (Core) is required and available everywhere Next 6 kits are available (almost) everywhere Last 2 kits are available on a subset of resources April 2008Software Integration Overview

5. CTSS 4 – A New Paradigm Significant change in how we define CTSS –CTSS 1 thru 3: Monolithic software stack –CTSS 4: Modular user capabilities –Improved many aspects of capability delivery Better descriptions of the capabilities (esp. for users) Better documentation Clearer availability information More focused delivery process (package, deploy, and configure) Improved process for RPs to select and publish their offerings Delivery timeline –Designed in 2006 –Capability kits defined in Q1-Q2 2007 –Capabilities rolled out in Q2-Q3 2007 April 20085Software Integration Overview

6. Motivation for a Paradigm Shift Transform CTSS from a complex puzzle of interlocking pieces to a menu of recognizable user functions Improve agility in response to new user requirements Easier for users to discover and learn about capabilities Operational benefits –Improves delivery scalability thru distributing ownership –Streamlined change management process –Makes it possible to evaluate and adopt competitors to CTSS components (or eliminate existing components) –Easier to correlated component usage data to specific user benefits and types April 20086Software Integration Overview

7. Information Service The linchpin in TeraGrid’s coordinated user environment –How resource providers register availability of CTSS capabilities –How users discover coordinated capabilities and how to access them –Integrated with user documentation, user portal, Inca, gateways GIG contributions –Architecture and technology selection –Information schema –New information providers –Presentation views and services –High-availability central information service RP contributions –Local information services –Data! April 2008Software Integration Overview7 See section 3.3 of annual report

8. Software Integration Challenges Expected issues with integrating diverse high-end systems –Integrating with local RP specialized services –Integrating with local RP security policies/practices –Meshing with local resource hardware and software operations schedules Scalability and reliability issues in specific capabilities –LEAD gateway; NCSA, IU, UofC/ANL RPs; Globus team; PSC End-to-End data movement group; and GIG Gateways and Software Integration teams working for several months to address operational, configuration, and software issues with Grid services. –Human Arterial Tree simulation team; IU RP; MPICH-G2/mpiG team; and GIG SI team worked to resolve deployment and customization issues in software. Complexity in new capabilities –New Remote Compute version 4 capability will integrate 5 distinct web services, and 1 non-web service to deliver functionality April 20088Software Integration Overview

9. Operations Activities Coordinate the CTSS capability definition process –Document the capabilities – user scenarios and key interfaces –Define/maintain a recommended implementation –Track user requirements –Track/address software component issues –Includes formal change management process Maintain six of the CTSS capability definitions Generate CTSS software packages –Build binaries and package 11 CTSS components (9 with OSG’s VDT) –Package 24 other CTSS components –Why? Security vulnerabilities, software issues, RP technical requirements, new resources and platforms, new operational requirements Operate GIG-provided services April 2008Software Integration Overview9

10. Capability Development Activities TeraGrid client software distribution Application hosting capability Usage reporting Build and test – Metronome service Fostering interoperability with other major grids Fostering international grid standards Several development projects at IU, NCAR, NCSA, PSC, TACC, UC/ANL April 2008Software Integration Overview10