The Cactus Code: A Problem Solving Environment for the Grid Gabrielle Allen, Gerd Lanfermann Max Planck Institute for Gravitational Physics

What is Cactus? Cactus is a freely available, modular, portable and manageable environment for collaboratively developing parallel, high- performance multidimensional simulations

Cactus Core “Flesh” Plug-In “Thorns” (modules) driver input/output interpolation SOR solver coordinates boundaryconditions boundary conditions black holes equations of state remote steering wave evolvers multigrid parameters gridvariables grid variables errorhandling error handling scheduling extensibleAPIs extensible APIs makesystem make system ANSI C Fortran/C/C++Java/Perl/Python

Cactus Architecture Configure CST Flesh Computational Toolkit Operating Systems AIX NT Linux Unicos Solaris HP-UX Thorns Cactus SuperUX Irix OSF Make

Thorn Architecture Make Information Source Code Documentation! Configuration Files Parameter Files and Testsuites ???? Fortran Routines C++ Routines C Routines Thorn

State of the Art Numerical Relativity Simulations Albert Einstein Institute Washington University Viz: Werner Benger

Current Version Cactus 4.0 n Cactus 4.0 beta 1 released September 1999 n Flesh and many thorns distributed under GNU GPL n Currently: Cactus 4.0 beta 8 n Supported Architectures: l SGI Origin l SGI 32/64 l Cray T3E l Dec Alpha l Intel Linux IA32/IA64 l Windows NT l HP Exemplar l IBM SP2 l Sun Solaris l Hitachi SR8000-F l NEC SX-5 l Mac Linux

Flesh API n Abstract Flesh API for l Driver functions (storage, communication) l Interpolation l Reduction l IO, checkpointing l Coordinates l etc, etc n In general, thorns overload or register their capabilities with the Flesh, agreeing to provide a function with the correct interface n e.g. CCTK_SyncGroup (overloaded) n e.g. CCTK_OutputVar(“variable”,“IOASCII”)(registered)

Application View Flesh CCTK_(…)CST Application Toolkit Computational Toolkit

Parallelism in Cactus n Cactus is designed around a distributed memory model. Each thorn is passed a section of the global grid. n The actual parallel driver (implemented in a thorn) can use whatever method it likes to decompose the grid across processors and exchange ghost zone information - each thorn is presented with a standard interface, independent of the driver. n Standard driver distributed with Cactus (PUGH) is for a parallel unigrid and uses MPI for the communication layer n PUGH can do custom processor decomposition and static load balancing

Configuration files n Each thorn provides 3 configuration files, detailing its interface with the Flesh and with other thorns n CCL: Cactus Configuration Language n interface.ccl l implementation, this thorn’s variables and variables used from other thorns n param.ccl l this thorn’s parameters, parameters used and extended from other thorns n schedule.ccl l when and how this thorn’s routines should be executed, optionally with respect to routines from other thorns

Cactus Computational Toolkit n CactusBase l Boundary, IOUtil, IOBasic, CartGrid3D, IOASCII, Time n CactusBench l BenchADM n CactusExample l WaveToy1DF77, WaveToy2DF77 n CactusElliptic l EllBase, EllPETSc, EllSOR, EllTest n CactusPUGH l Interp, PUGH, PUGHSlab n CactusPUGHIO l IOFlexIO, IOHDF5, IsoSurfacer n CactusTest l TestArrays, TestCoordinates, TestInclude1, TestInclude2, TestComplex, TestInterp n CactusWave l IDScalarWave, IDScalarWaveC, IDScalarWaveCXX, WaveBinarySource, WaveToyC, WaveToyCXX, WaveToyF77, WaveToyF90, WaveToyFreeF90 n external l IEEEIO, RemoteIO, TCPXX

Cactus can make use of... Autopilot FlexIO (IEEEIO/HDF5) Globus GrACE HDF5 MPI Panda IO PAPI PETSc

AutoPilot n Dynamic performance instrumentation, on-the-fly performance data reduction, resource management algorithms, real-time adaptive control mechanism n Cactus provides a mechanism to register timers, and Autopilot is currently being integrated

FlexIO (IEEEIO) n FlexIO is a compact multi-platform API for storing multidimensional scientific data. It hides the differences between underlying file formats including HDF5 and IEEEIO. IEEEIO readers for: Amira AVS IDL LCA Vision NAG Explorer n IEEEIO is a compact library for storing multidimensional scientific data in a binary format that can be transported between different computer systems. n Cactus thorn CactusPUGHIO/IOFlexIO outputs multidimensional data using the IEEEIO library Documentation in thorns CactusBase/IOUtil and CactusPUGHIO/IEEEIO

Globus Toolkit n Globus Toolkit: Enables application of Grid concepts to scientific and engineering computing n Cactus (with the default MPI driver) compiles with Globus (1.0/1.1), using MPICH-G. n Cactus can then be run using RSL scripts as usual with Globus The Grid: Dependable, consistent, pervasive access to [high-end] resources Collaborative engineering Browsing of remote datasets Use of remote software Data-intensive computing Very large-scale simulation Large-scale parameter studies

HDF5 n Hierarchical data format for scientific data management (I/O libraries and tools). n Future standard, overcomes limitations of HDF4. Simple but powerful model, includes hyperslabs, datatype conversion, parallel IO. n Used for 2D/3D output in Computational Toolkit (CactusPUGHIO/IOHDF5) n Much development in (remote) visualization and steering with Cactus uses HDF5 n Readers for Amira, OpenDX, (LCA Vision) Documentation in thorns CactusBase/IOUtil and CactusPUGHIO/IOHDF5

Panda IO n Data management techniques for I/O intensive applications in high- performance scientific computing. n Simpler, more abstract interfaces, efficient layout alternatives for multidimensional arrays, high performance array I/O operations. n Thorn IOPanda

PAPI n Standard API for accessing the hardware performance counters on most microprocessors. n Useful for tuning, optimisation, debugging, benchmarking, etc n Java GUI available for monitoring the metrics n Cactus thorn CactusPerformance/PAPI

Grid-Enabled Cactus n Cactus and its ancestor codes have been using Grid infrastructure since 1993 n Support for Grid computing was part of the design requirements for Cactus 4.0 (experiences with Cactus 3) n Cactus compiles “out-of-the-box” with Globus [using globus device of MPICH-G(2)] n Design of Cactus means that applications are unaware of the underlying machine/s that the simulation is running on … applications become trivially Grid-enabled n Infrastructure thorns (I/O, driver layers) can be enhanced to make most effective use of the underlying Grid architecture

Grid Experiments n SC93 l remote CM-5 simulation with live viz in CAVE n SC95 l Heroic I-Way experiments leads to development of Globus. Cornell SP-2, Power Challenge, with live viz in San Diego CAVE n SC97 l Garching 512 node T3E, launched, controlled, visualized in San Jose n SC98 l HPC Challenge. SDSC, ZIB, and Garching T3E compute collision of 2 Neutron Stars, controlled from Orlando n SC99 l Colliding Black Holes using Garching, ZIB T3E’s, with remote collaborative interaction and viz at ANL and NCSA booths n 2000 l Single simulation LANL, NCSA, NERSC, SDSC, ZIB, Garching, … l Dynamic distributed computing … spawning new simulations

Cactus + Globus Cactus Application Thorns Distribution information hidden from programmer Initial data, Evolution, Analysis, etc Grid Aware Application Thorns Drivers for parallelism, IO, communication, data mapping PUGH: parallelism via MPI (MPICH-G2, grid enabled message passing library) Grid Enabled Communication Library MPICH-G2 implementation of MPI, can run MPI programs across heterogeneous computing resources Standard MPI Single Proc

Grand Picture Remote steering and monitoring from airport Origin: NCSA Remote Viz in St Louis T3E: Garching Simulations launched from Cactus Portal Grid enabled Cactus runs on distributed machines Remote Viz and steering from Berlin Viz of data from previous simulations in SF café DataGrid/DPSS Downsampling Globus http HDF5 IsoSurfaces

Grid Related Projects n ASC: Astrophysics Simulation Collaboratory l NSF Funded (WashU, Rutgers, Argonne, U. Chicago, NCSA) l Collaboratory tools, Cactus Portal l Currently setting up testbed (Globus, Cactus, Portal at NCSA, ZIB, AEI) n E-Grid: European Grid Forum l Members from academic and government institutions, computer centers and industry l Test application: Cactus+Globus l Currently working towards distributed computing project for SC2000 (spawning Cactus jobs to new machines) n GrADs: Grid Application Development Software l NSF Funded (Rice, NCSA, U. Illinois, UCSD, U. Chicago, U. Indiana...) l Application driver for grid software

Grid Related Projects (2) n Grid Forum l Experiments l Transparency appearances n Distributed Runs l AEI, Argonne, U. Chicago l Working towards running on several computers, 1000’s of processors (different processors, memories, OSs, resource management, varied networks, bandwidths and latencies) n TIKSL l German DFN funded: AEI, ZIB, Garching l Remote online and offline visualization, remote steering/monitoring n Cactus Team l Dynamic distributed computing … l Testing of alternative communication protocols … MPI, PVM, SHMEM, pthreads, OpenMP, Corba, RDMA,...

Dynamic Distributed Computing n Make use of l Running with management tools such as Condor, Entropia, etc. l Scripting thorns (management, launching new jobs, etc) l Dynamic use of MDS for finding available resources n Applications l Portal for simulation launching and management l Intelligent parameter surveys (Cactus control thorn) l Spawning off independent jobs to new machines e.g. analysis tasks l Dynamic staging … seeking out and moving to faster/larger/cheaper machines as they become available (Cactus worm) l Dynamic load balancing (e.g. inhomogeneous loads, multiple grids)

Remote Visualization IsoSurfaces and Geodesics Contour plots (download) Grid Functions Streaming HDF5 Amira LCA Vision OpenDX

Remote Visualization n Streaming data from Cactus simulation to viz client l Clients: OpenDX, Amira, LCA Vision,... n Protocols l Proprietary: –Isosurfaces, geodesics l HTTP: –Parameters, xgraph data, JPegs l Streaming HDF5: –HDF5 provides downsampling and hyperslabbing –all above data, and all possible HDF5 data (e.g. 2D/3D) –two different technologies Streaming Virtual File Driver (I/O rerouted over network stream) XML-wrapper (HDF5 calls wrapped and translated into XML)

Remote Visualization (2) n Clients l Proprietary: –Amira l HTTP: –Any browser (+ xgraph helper application) l HDF5: –Any HDF5 aware application h5dump Amira OpenDX LCA Vision (soon) l XML: –Any XML aware application Perl/Tk GUI Future browsers (need XSL-Stylesheets)

OpenDX n Open source, (free), multiplatform, large active development community, easy to program n Reads HDF5 (Cactus) data from file or remotely streamed from Cactus n Simple GUI, select different hyperslabs from 3D data n Also support for streamed ASCII data from Cactus

Remote Visualization - Issues n Parallel streaming l Cactus can do this, but readers not yet available on the client side n Handling of port numbers l clients currently have no method for finding the port number that Cactus is using for streaming l development of external meta-data server needed (ASC/TIKSL) n Generic protocols n Data server l Cactus should pass data to a separate server that will handle multiple clients without interfering with simulation l TIKSL provides middleware (streaming HDF5) to implement this n Output parameters for each client

Remote Steering Remote Viz data XML HTTP HDF5 Amira Any Viz Client

Remote Steering n Stream parameters from Cactus simulation to remote client, which changes parameters (GUI, command line, viz tool), and streams them back to Cactus where they change the state of the simulation. n Cactus has a special STEERABLE tag for parameters, indicating it makes sense to change them during a simulation, and there is support for them to be changed. n Example: IO parameters, frequency, fields n Current protocols: l XML (HDF5)to standalone GUI l HDF5to viz tools (Amira) l HTTPto Web browser (HTML forms)

Thorn http n Thorn which allows simulation to act as a web server n Connect to simulation from any browser n Monitor run: parameters, basic visualization,... n Change steerable parameters n See running example at n Wireless remote viz, monitoring and steering

Remote Steering - Issues n Same kinds of problems as remote visualization l generic protocols l handling of port numbers l broadcasting of active Cactus simulations n Security l Logins l Who can change parameters? n Lots of issues still to resolve...

Remote Offline Visualization Viz Client (Amira) HDF5 VFD DataGrid (Globus) DPSS FTP HTTP Visualization Client DPSS Server FTP Server Web Server Remote Data Server Downsampling, hyperslabs Viz in Berlin 4TB at NCSA Only what is needed

Remote Offline Visualization n Accessing remote data for local visualization n Should allow downsampling, hyperslabbing, etc. n Access via DPSS is working (TIKSL) n Waiting for DataGrid support for HTTP and FTP to remove dependency on the DPSS file systems.

More Information... n Web site: n Users Guide n Development projects n HOWTOs l References for common questions, includes HOWTO-QuickStart n Tutorials n Help desk l n Bug tracking n Mailing lists