1. Cactus Tools for the Grid Gabrielle Allen Max Planck Institute for Gravitational Physics, (Albert Einstein Institute)

2. Cactus THE GRID: Dependable, consistent, pervasive access to high-end resources CACTUS is a freely available, modular, portable and manageable environment for collaboratively developing parallel, high- performance multi-dimensional simulations www.CactusCode.org

3. What is Cactus n Flesh (ANSI C) provides code infrastructure (parameter, variable, scheduling databases, error handling, APIs, make, parameter parsing, ) n Thorns (F77/F90/C/C++/Java/Perl/Python) are plug-in and swappable modules or collections of subroutines providing both the computational instructructure and the physical application. Well-defined interface through 3 config files n Just about anything can be implemented as a thorn: Driver layer (MPI, PVM, SHMEM, …), Black Hole evolvers, elliptic solvers, reduction operators, interpolators, web servers, grid tools, IO, … n User driven: easy parallelism, no new paradigms, flexible n Collaborative: thorns borrow concepts from OOP, thorns can be shared, lots of collaborative tools n Computational Toolkit: existing thorns for (Parallel) IO, elliptic, MPI unigrid driver, n Integrate other common packages and tools: HDF5, Globus, PETSc, PAPI, Panda, FlexIO, GrACE, Autopilot, LCAVision, OpenDX, Amira,... n Trivially Grid enabled!

4. Current Version Cactus 4.0 n Cactus 4.0 beta 1 released September 1999 n Community code: Distributed under GNU GPL n Currently: Cactus 4.0 beta 8 n Supported Architectures: l SGI Origin l SGI 32/64 l Cray T3E l Dec Alpha l Intel Linux IA32/IA64 l Windows NT l HP Exemplar l IBM SP2 l Sun Solaris l Hitachi SR8000-F l NEC SX-5 l Mac Linux l...

5. Cactus Computational Toolkit: Parallel utilities (thorns) for computational scientist CactusBase l Boundary, IOUtil, IOBasic, CartGrid3D, IOASCII, Time CactusBench l BenchADM CactusConnect l HTTPD, HTTPDExtra CactusExample l WaveToy1DF77, WaveToy2DF77 CactusElliptic l EllBase, EllPETSc, EllSOR, EllTest CactusPUGH l Interp, PUGH, PUGHSlab, PUGHReduce CactusPUGHIO l IOFlexIO, IOHDF5, IsoSurfacer CactusIO l IOJpeg CactusTest l TestArrays, TestCoordinates, TestInclude1, TestInclude2, TestComplex, TestInterp, TestReduce CactusWave l IDScalarWave, IDScalarWaveC, IDScalarWaveCXX, WaveBinarySource, WaveToyC, WaveToyCXX, WaveToyF77, WaveToyF90, WaveToyFreeF90 CactusExternal l FlexIO, jpeg6b BetaThorns (In Development) l IOStreamedHDF5, Renderer, IOHDF5Util,…, many more

6. Cactus Community DLR Geophysics (Bosl) Numerical Relativity Community Cornell Crack prop. NASA NS GC Livermore SDSS (Szalay) Intel Microsoft Clemson “Egrid” NCSA, ANL, SDSC AEI Cactus Group (Allen) NSF KDI (Suen) EU Network (Seidel) Astrophysics (Zeus) US Grid Forum DFN Gigabit (Seidel) “GRADS” (Kennedy, Foster, Dongarra, et al) ChemEng (Bishop) San Diego, GMD, Cornell Berkeley

7. Grid Computing n AEI Numerical Relativity Group has access to high-end resources in over ten centers in Europe/USA n They want: l Bigger simulations, more simulations and faster throughput l Intuitive IO at local workstation l No new systems/techniques to master!! n How to make best use of these resources? l Provide easier access … noone can remember ten usernames, passwords, batch systems, file systems, … great start!!! l Combine resources for larger productions runs (more resolution badly needed!) l Dynamic scenarios … automatically use what is available l Remote/collaborative visualization, steering, monitoring n Many other motivations for Grid computing...

8. Grand Picture Remote steering and monitoring from airport Origin: NCSA Remote Viz in St Louis T3E: Garching Simulations launched from Cactus Portal Grid enabled Cactus runs on distributed machines Remote Viz and steering from Berlin Viz of data from previous simulations in SF café DataGrid/DPSS Downsampling Globus http HDF5 IsoSurfaces

9. Remote Visualization IsoSurfaces and Geodesics Contour plots (download) Grid Functions Streaming HDF5 Amira LCA Vision OpenDX

10. Remote Steering Remote Viz data XML HTTP HDF5 Amira Any Viz Client

11. Remote Offline Visualization Viz Client (Amira) HDF5 VFD DataGrid (Globus) DPSS FTP HTTP Visualization Client DPSS Server FTP Server Web Server Remote Data Server Downsampling, hyperslabs Viz in Berlin 4TB at NCSA Only what is needed

12. Remote Monitoring: Thorn HTTPD n Thorn which allows simulation any to act as its own web server n Connect to simulation from any browser anywhere … collaborate n Monitor run: parameters, basic visualization,... n Change steerable parameters n See running example at www.CactusCode.org n Wireless remote viz, monitoring and steering

13. Remote Viz: OpenDX n Open source, (free), multiplatform, large active development community, easy to program n Reads HDF5 (Cactus) data from file or remotely streamed from Cactus n Simple GUI, select different hyperslabs from 3D data n Also support for streamed ASCII data from Cactus

14. Remote Viz: IsoView n Lightweight, free, isosurface viewer n Distributed from Cactus website n Displays isosurfaces streamed from Cactus (thorn IsoSurfacer calculates isosurfaces inline with the parallel simulation) n Connect to running simulation on any remote machine n Steer isosurface value from client

15. Cactus Portal n KDI ASC Project n Technology: Globus, GSI, Java Beans, DHTML, Java CoG, MyProxy, GPDK, TomCat, Stronghold n Locates resources, builds/finds executables, central management of parameter files/job output, submit jobs to local batch queues, tracks active jobs n Submission of distributed runs n Adding more collaborative features (eg group accounting, …) n Accesses the ASC Grid Testbed (SDSC, NCSA, Argonne, ZIB, LRZ, AEI)

16. New Grid Applications n Dynamic Staging: move to faster/cheaper/bigger machine l “Cactus Worm” n Multiple Universe l create clone to investigate steered parameter (“Cactus Virus”) n Automatic Convergence Testing l from intitial data or initiated during simulation n Look Ahead l spawn off and run coarser resolution to predict likely future n Spawn Independent/Asynchronous Tasks l send to cheaper machine, main simulation carries on n Thorn Profiling l best machine/queue l choose resolution parameters based on queue l ….

17. New Grid Applications (2) n Dynamic Load Balancing l inhomogeneous loads l multiple grids n Portal l resource choosing l simulation launching l management n Intelligent Parameter Surveys l farm out to different machines n Make use of l Running with management tools such as Condor, Entropia, etc. l Scripting thorns (management, launching new jobs, etc) l Dynamic use of eg MDS for finding available resources

18. Go! Dynamic Grid Computing Clone job with steered parameter Queue time over, find new machine Add more resources Found a horizon, try out excision Look for horizon Calculate/Output Grav. Waves Calculate/Output Invariants Find best resources Free CPUs!! NCSA SDSC RZG SDSC LRZArchive data

19. Users View

20. Cactus Grid Development n Projects: GrADs, KDI Astrophysics Collaboratory, TIKSL/GriKSL, EGrid/GGF, Globus team, AEI n Grid Application Development Toolkit: Application developer should be able to build simulations with tools that easily enable dynamic grid capabilities e.g. Query information server, Network Monitoring, Simulation instrumenation/Contract specification, Spawning and Migrating, Authentification Server, Data handling, Data transfer, Visualization, Steering, … n Remote visualization, steering, data management n User Portal n Dynamics Applications

21. More Information... n Cactus: l Web Site: www.CactusCode.org (Documentation/Tutorials etc) l Cactus Worm: www.CactusCode.org/Development/Egrid.html n Global Grid Forum (Egrid) l www.egrid.org l www.gridforum.org n ASC Portal l www.ascportal.org n TIKSL Gigabit Computing l www.zib.de/Visual/projects/TIKSL/ n Black Holes and Neutron Star: Pictures and Movies l jean-luc.aei.mpg.de n Any questions: cactus@cactuscode.org