Development Timelines Ken Kennedy Andrew Chien Keith Cooper Ian Foster John Mellor-Curmmey Dan Reed

What We Have Today Prototype Testbeds with middleware Prototype of execution components in ScaLAPACK and Cactus Design for the execution environment —Implementation of resource specification (AART) —Interfaces for Scheduler/Resource Negotiator –Prototype resource scheduler (Dail) –Prototype renegotiator (Sievert) —Prototype Contract Monitoring System Grid ready libraries with performance models Tools for extracting information from program executables —For performance estimation on single nodes DSL design for Signal/Image Processing

What We Will Have on CGrADS Day One Prototype Execution System for Heterogeneous Clusters —Prototype Scheduler/Resource Negotiator —Binder —Run-time system —Contract monitor (working together) ScaLAPACK re-implemented using the prototype Cactus using generic resource selector Prototype automatic performance modeling for black boxes Script-based application composition (without optimization) Testbeds —Microgrid running ScaLAPACK and Cactus —Integrated NWS and prediction in MacroGrid and configuration tools

Currently Targeted Application Milestones Three codes concurrently, each engaged for two-three years 2002 —Cactus: traditional PDE solver, aggressive application scenarios —CAPS: dynamic data acquisition and real-time data ingest —ChemEng Workbench: application service scenarios prototypes 2003 —Cactus: by now transitioned to operational use by application group —CAPS: adaptive execution for high-speed prediction —ChemEng Workbench: application service scenarios operational —CMS/GriPhyN: query estimation and dynamic scheduling —BIRN-like distributed bioscience: emergent behavior issues 2004 —CAPS: by now transitioned to operational use by application group —CMS/GriPhyN: large-scale experimentation in production settings —NEES: application service and real-time data analysis scenarios

NCSA Linux Cluster 5) Secondary reports complete to master Master Condor job running at Caltech 7) GridFTP fetches data from UniTree NCSA UniTree - GridFTP-enabled FTP server 4) 100 data files transferred via GridFTP, ~ 1 GB each Secondary Condor job on WI pool 3) 100 Monte Carlo jobs on Wisconsin Condor pool 2) Launch secondary job on WI pool; input files via Globus GASS Caltech workstation 6) Master starts reconstruction jobs via Globus jobmanager on cluster 8) Processed objectivity database stored to UniTree 9) Reconstruction job reports complete to master CMS Data Reconstruction Example Scott Koranda, Miron Livny, others

Program Preparation System Milestones 2002 —Preliminary automated support for performance models ( black boxes ) —Binder includes local optimization, inserts probes and actuators —Prototype DSL for signal processing —Evaluate original COP design 2003 —Binder support for global optimization —Experiment with contract monitoring/reporting in applications —Evaluate and extend DSL support for signal processing 2004 —First dynamic optimizer prototype, plan for retargeting Binder —Initial telescoping language prototype based on experience with signal processing DSL —Whole Program Compiler generates initial COPs

Program Execution System Milestones 2002 —Virtual organization management tools —Resource selector and application manager prototypes —Temporal contract violation specification —Rescheduling models including distribution costs & info quality 2003 —Integrated resource monitoring and prediction prototype —Resource selector and application manager tools —Composable contract specification and tools —Reconfigurable object program specification —Scheduling models for highly parallel and data Grid applications 2004 —Enhanced application and resource measurement infrastructure —Enhanced resource scheduling infrastructure with adv. resv., etc. —Performance economics for global resource scheduling

Execution Environment Milestones

Software Development Process Andrew A. Chien Department of Computer Science and Engineering University of California, San Diego

Software Development Process Development Efficiency versus Robust Output —Key is choosing where to live on this spectrum CGrADS Objectives —Ground breaking research —Rapid development of functionality and experimentation —Difficult to know what will become widely used –Hardening of selected components

Software Development Approach Experimental development with lightweight process Core development with close to full process Targeted hardening of what becomes core —Enable researchers inside and outside CGrADS to build on core —Software output not supported as a product

Software Development Process Project Software Manager defines and enforces process; drives progress Infrastructure —Revision control system (CVS,SourceSafe,Clearcase) –Code, documents —Defect tracking system (ClearQuest) —Design, documentation, coding guidelines —Software infrastructure Process —Lightweight Process —Industrial Process

Two Flavors of Process “Lightweight” Process (Research Strength) —Requirements and Design; Review —Implementation and Test; Review —Integration; Test —Iterative Improvement Full Process (Industrial Strength) —Requirements; Review —Design; Review —Implementation; Test and Review —Integration; Test —Iterative Improvement