TeraGrid CTSS Plans and Status Dane Skow for Lee Liming and JP Navarro OSG Consortium Meeting 22 August, 2006

Common TeraGrid Software & Services (CTSS) Science Gateway TeraGrid Service Level 1: Login Access Level 2: Grid Service Level 3: TeraGrid Service Science Gateway TG Compute Resource CTSS TG Data Resource CTSS TG Compute Resource CTSS TG Viz Resource

CTSS v3 Status GT4 based services –Description at –Cutover to GT4 services production on June 19 –Moving Aggressively to WS services Continue to work through deployment configuration issues –(primarily MPI related) –15 unique platforms to build –Difficulties keeping up with resource upgrades and with new 64 bit architectures

Focus Areas Community Software Areas –Persistent area for community supported software –Very useful for multi-grid work MDS4 –Initial focus is support of user portal overview of resources –Extend to general service registry INCA, Gateways Coordination with VDT Build & Test

Planned TeraGrid/NMI Automated Build Framework TG submit host grandcentral TeraGrid Software Sources Binaries NMI Build/Test Framework Condor / Condor-G / DAGMan TG build host NMI Framework Wrapper Condor Startd TG Software Build Scripts TG Build Tools TeraGrid CVS Build Scripts Build Specs NMI Results DB Build Job Info Existing NMI component Evolved TG component New TG component

General Character of CTSSv4 To meet project milestones, CTSS changes must accelerate in the coming years. Process –Process will be the focus of CTSSv4. –Significant changes in who and how, not so much in what. –Process changes now will enable us to more effectively manage content changes in the future. Content –Newer component versions that include features we need –More allowable versions –Support for more platforms

CTSS 4 Process Goals Change the focus from software packages to capabilities. –Software should be deployed to meet user capability requirements, not to satisfy abstract agreements. –Which capabilities ought to be coordinated, and why? Be explicit about which capabilities are expected to be on which systems. –The CTSS core (mandatory capabilities) is radically smaller. –Each RP explicitly decides which additional capabilities it will provide, based on the intended purpose of each system. Make the process of defining CTSS more open and inclusive and more reflective of the TeraGrid project structure (GIG + multiple RPs, working groups, RATs, etc.). –GIG/RP working groups and areas have an open mechanism for defining, designing, and delivering CTSS capabilities. –Expertise is distributed, so the process should be distributed also.

Goals Continued Improve coordination significantly. –Changes are coordinated more explicitly with more TeraGrid sub-teams. –Each sub-team has a part in change planning.

CTSS 4 Strategy Break the CTSS monolith into multiple capability modules. Employ a formal change planning process.

CTSS “Kits” Reorganize CTSS into a series of kits. –A kit provides a small set of closely related capabilities. (job execution service, dataset hosting service, high-performance data movement, global filesystem, etc.) –A kit is (as much as possible) independent from other kits. Each kit includes: –a definition of the kit that focuses on purpose, requirements, and capabilities, including a problem statement and a design; –a set of software packages that RP administrators can install on their system(s) in order to implement the design; –documentation for RP administrators on how to install and test the software; –inca tests that test whether a given system satisfies the stated requirements; –softenv definitions that allow users to access the software.

The “Core” Kit Provides the capabilities that are absolutely necessary for a resource to meet the most basic integrative requirements of the TeraGrid. –Common Authentication, Authorization, Auditing, and Accounting capabilities –A system-wide registry of capabilities and service information –A Verification & Validation mechanism for capabilities –System-wide Usage Reporting capabilities This is much smaller than the current set of “required” CTSSv3 components. Unlike other capability kits, the Core Kit is focused on TeraGrid operations, as opposed to user capabilities.

Core Kit Provides Integrative Services Authentication, Authorization, Auditing, Accounting Mechanisms –Supports TeraGrid allocation processes –Allows coordinated use of multiple systems –Supports TeraGrid security policies –Goal: Forge a useful link between campus authentication systems, science gateway authentication systems, and TeraGrid resources Service Registry –Goal: Provide a focal point for registering presence of services and capabilities on TeraGrid resources –Goal: Support documentation, testing, automatic discovery, and automated configuration for distributed services (tgcp) Verification & Validation –Independently verifies the availability of capabilities on each resource –Goal: Focus more clearly on the specific capabilities each resource is intended to offer Usage Reporting –Goal: Support the need to monitor and track usage of TeraGrid capabilities

CTSS Capability Kits Each CTSS capability kit is an opportunity for resource providers to deploy a specific capability in coordination with other RPs. –Focal point for collecting and clarifying user requirements (via a RAT) –Focal point for designing, documenting, and implementing a capability (via a WG) –Focal point for deploying the capability (via the software WG) RPs can explicitly decide and declare which capabilities they intend to provide on each resource. –What is appropriate for each resource? –What is the RP’s strategy for delivering service to the community? TeraGrid’s system-wide registry tracks which CTSS capabilities are provided by each resource. –By declaration, by registration, and by verification

CTSS Capability Kits Kits may be defined, designed, implemented, packaged, documented, and supported by a broad range of people. –RATs –Working groups –GIG areas –Resource providers –Other communities The key feature of a CTSS capability is that its deployment is coordinated among RPs.

CTSS 4 Capability Kits –TeraGrid Core Capabilities Authorization, Usage Reporting, Service Registration, V & V –Application Development & Runtime Globus libraries, compilers, softenv, MPIs, etc –Remote Compute GRAM, WS-GRAM –Remote Login GSI-SSH, clients –Science Workflow Support Condor (G), GridShell –Data Management RLS, SRB –Data Movement GridFTP, RFT –Wide Area Filesystem GPFS-WAN

Change Coordination Process Motivation –New CTSS kit structure results in more potential sources of changes. –Scaling of resources (both number and diversity) results in more potential points of confusion/coord failure. –In general, we’d like to do this better. Goals –Clarity of purpose for changes –Help for documentation –Help in identifying points requiring coordination –Tracking deployment steps and progress –Easy to use for small changes, helpful for large changes.

Change Coordination Structure Change Description Data Sheet –Collects the basic facts about a proposed change –Provides everyone with the information needed to understand what is planned and who is involved and to identify potential risks –Will provide a record of changes Change Planning Checklist –Helps the planning team brainstorm about what the necessary points of coordination are –Provides opportunities for recording the coordination plans for each sub-team (docs, user services, RP admins, security, etc.) –Helps get coordination started early