1. Projects using Cactus Gabrielle Allen gallen@cct.lsu.edu Cactus Retreat Baton Rouge, April 2004

2. Cactus Projects @ AEI/LSU n User Support n Applications n Research and Development l Support for more numerical models (Saturday) l Large scale computing l Frameworks l Visualization l Data models and formats l Grid computing

3. Community Development n Adaptive Mesh Refinement l PAGH (Wash U) l Carpet (Albert Einstein Institute) n Application Performance Modeling l University of Chicago n Data Formats and Visualization l Lawrence Berkeley Lab l Konrad Zuse Zentrum l Albert Einstein Institute n Optimization and Performance l NCSA, Intel, Cray, Lawrence Berkeley Lab, Absoft n User GUIs l Wash U

4. Interoperability n Pending proposal to NSF ITR program l “Hypercode: Interoperable Infrastructure Initiative” n Addressing interoperability for general computational infrastructures, l Cactus, Chombo, Paramesh and CCA-based frameworks n Focused around applications: l Numerical Relativity l Computational Astrophysics l Coastal Modeling l Climate Modeling l Cosmology l Computational Fluid Dynamics

5. Interoperability

6. n Develop common mechanisms and abstractions to enable different simulation codes to use modules and data interchangeably n Partners: l LSU l LBL (John Shalf, Phil Colella, Julian Borill) l U. Maryland (Kevin Olsen, Joan Centrella) l U. Indiana (Denis Gannon) l NCSA (Greg Daues)

7. Interoperability n Main developments for Cactus: l Incorporate other AMR drivers into Cactus –Chombo, Paramesh l Incorporate other elliptic solvers into Cactus l Develop a community toolkit for CFD l Add new features to Cactus –Adaptivity –Dynamic component loading l Develop common data model –New visualization tools available

8. Visualization n Ongoing visualization projects at AEI, LSU, LBL with Cactus n Pending DST-NSF proposal with the computer science department and C-DAC to build visualization infrastructure to allow data to be analyzed and visualized on the fly l Additions to Cactus I/O infrastructure l Thorns for visualization l Web-based visualization tools

9. Large Scale Computing n NSF Software Technologies for High End Computing l (being written) n Incorporate fault tolerant MPI in Cactus driver layer l University of Tennessee n Develop HTTPD thorn into an interactive, real time parallel debugger n Detect and exploit memory and network connection hierarchy from processor cache, through node layout on clusters of SMPs, to cluster interconnections on the Grid n Performance monitoring and adaption using e.g. PAPI library

10. Grid Computing … infrastructure enabling the integrated, collaborative use of high-end computers, networks, databases, scientific instruments owned and managed by multiple organizations … … applications often involve large amounts of data and/or computing, secure resource sharing across organizational boundaries, not easily handled by today’s Internet and Web infrastructures … “resource sharing and coordinated problem solving in dynamic, multi-institutional virtual organizations”

11. Remote Monitoring/Steering: n HTTPD Thorn which allows simulation to any to act as its own web server n Connect to simulation from any browser anywhere … collaborate n Monitor run: parameters, basic visualization,... n Change steerable parameters n Running example at www.CactusCode.org www.CactusCode.org n Wireless remote viz, monitoring and steering

12. User Portal (ASC and GridSphere) n Collaboration focal point for a virtual organization n Interact, share data n Start jobs on remote resources n Move/browse files n Track and monitor announced jobs n Access to new Grid technologies n www.ascportal.org n www.gridsphere.org

13. Portal Server SMS Server Mail Server “TestBed” R u n n i n g A p p li c a t i o n s Notification (Announce)

14. OpenDX, Amira, … HDF5 GridFTP VFD Stream VFD Visualization Tools GridFTP Remote Data Server IOStreamedHDF5 Simulation Hyperslabbing, Downsampling Remote Data and Visualization (GriKSL)

15. Dynamic Adaptive Distributed Computation SDSC IBM SP 1024 procs 5x12x17 =1020 NCSA Origin Array 256+128+128 5x12x(4+2+2) =480 OC-12 line (But only 2.5MB/sec) GigE:100MB/sec 17 12 5 4 2 5 2 These experiments: n Einstein Equations (but could be any Cactus application) Achieved: n First runs: 15% scaling n With new techniques: 70-85% scaling, ~ 250GF “Gordon Bell Prize” (Supercomputing 2001, Denver) Dynamic Adaptation: Number of ghostzones, compression, …

16. New Grid Scenarios n Intelligent Parameter Surveys, Monte Carlo n Dynamic Migration: faster/cheaper/bigger machine n Multiple Universe: create clone to investigate steered parameter n Automatic Component Loading (Needs of process change) n Automatic Convergence Testing n Look Ahead n Spawn Independent/Asynchronous Tasks n Routine Profiling: best machine/queue, parameters n Dynamic Load Balancing: inhomogeneous loads, multiple grids

17. GridLab n Cactus experiments with grid computing on the E-Grid l Cactus Worm: thorns which allowed simulations to migrate themselves from machine to machine l Spawning: sending (asynchronous) calculations in analysis thorns to different machines l We wrote the GridLab proposal to be able to do these, and other scenarios, in a better way

18. GridLab Project n http://www.gridlab.org n EU Funded ($5M) by 5th Framework l January 2002-December 2004): n Many partners in Europe and US l PSNC (Poland), AEI & ZIB (Germany), VU (Netherlands), MASARYK (Czech), SZTAKI (Hungary), ISUFI (Italy), Cardiff (UK), NTUA (Greece), Chicago, ISI & Wisconsin (US), Sun/Compaq l LSU now a collaborating partner

19. GridLab n Developing an easy-to-use, flexible, generic and modular Grid Application Toolkit (GAT), enabling applications to make innovative use of global computing resources n Focused on two principles: l co-development of infrastructure with real applications and user communities (Badly needed in grid computing!!) dynamic use of grids, with self-aware simulations adapting to their changing environment.

20. GridLab n 12 Work Packages covering: Grid Portals Mobile Users Different Grid Services Applications (Development) Test Bed Grid Application Toolkit (GAT)

21. Grid Application Toolkit Need a layer between applications and grid infrastructure: l Higher level than existing grid APIs, hide complexity, abstract grid functionality through application oriented APIs l Insulate against rapid evolution of grid infrastructure and state of grid deployment l Choose between different grid infrastructures l Make it possible for application developers to use and develop for the grid independent of the state of deployment of the grid infrastructure

22. Monitoring Resource Management Information Security Data Management GLOBUS Application Manager Logging Notification Migration Profiling SOAPWSDLCorbaOGSAOther Other Grid Infrastructure? Application “Is there a better resource I could be using?”

23. Application “Is there a better resource I could be using?” GAT_FindResource( ) The Grid

24. The Same Application … Application GAT Application GAT Application GAT Laptop The Grid Super Computer No network! Firewall issues!

25. GAT: Grid Application Toolkit n Standard API and Toolkit for developing portable Grid applications independently of the underlying Grid infrastructure and available services n Implements the GAT-API l Used by applications (different languages) n GAT Adaptors l Connect to capabilities/services n GAT Engine l Provides the function bindings for the GAT-API n http://www.gridlab.org/software/GAT

26. GAT Architecture

27. Cactus/GAT Integration GAT Library Cactus Flesh Thorn CGAT Thorn Physics and Computational Infrastructure Modules Cactus GAT wrappers Additional functionality Build system GridLab Service GridLab Service Grid Scenario Thorn Grid Scenario Thorn

28. TFM Task Farming on the Grid TFM implemented in Cactus GAT used for starting remote TFMs Designed for the Grid Tasks can be anything

29. Grid-Black Holes n Task farm small Cactus black hole simulations across testbed n Parameter survey: black hole corotation parameter n Results steer a large production black hole simulation

30. Physicist has new idea ! S1S1 S2S2 P1P1 P2P2 S1S1 S2S2 P2P2 P1P1 S Brill Wave Dynamic Grid Computing Found a horizon, try out excision Look for horizon Calculate/Output Grav. Waves Calculate/Output Invariants Find best resources Free CPUs!! NCSA SDSC RZG LRZ Archive data SDSC Add more resources Clone job with steered parameter Queue time over, find new machine Archive to LIGO public database