1. Cactus and Solving Einstein’s Equations Edward Wang 2002.04.09Yfwang@mail.ustc.edu.cn

2. 2 Episode one: What is cactus? http://www.cactus.org

3. 3 Description Cactus is an open source problem solving environment designed for scientists and engineers. Its modular structure easily enables parallel computation across different architectures and collaborative code development between different groups. Cactus originated in the academic research community, where it was developed and used over many years by a large international collaboration of physicists and computational scientists.

4. 4 The name Cactus comes from the design of a central core (or "flesh") which connects to application modules (or "thorns") through an extensible interface. Thorns can implement custom developed scientific or engineering applications, such as computational fluid dynamics. Other thorns from a standard computational toolkit provide a range of computational capabilities, such as parallel I/O, data distribution, or checkpointing.

5. 5 Cactus runs on many architectures. Applications, developed on standard workstations or laptops, can be seamlessly run on clusters or supercomputers. Cactus provides easy access to many cutting edge software technologies being developed in the academic research community, including the Globus Metacomputing Toolkit, HDF5 parallel file I/O, the PETSc scientific library, adaptive mesh refinement, web interfaces, and advanced visualization tools.

6. 6 PETSc http://www.globus.org/default.asp The Globus Project is developing fundamental technologies needed to build computational grids. http://hdf.ncsa.uiuc.edu/HDF5/ HDF5 is a library and file format for storing scientific data. http://www-fp.mcs.anl.gov/petsc/ PETSc is a suite of data structures and routines for the scalable (parallel) solution of scientific applications modeled by partial differential equations.

7. 7 Features Highly Portable Supported on most architectures Sophisticated make system Powerful Application Programming Interface User modules (thorns) plug-into compact core (flesh) Configurable interfaces, schedules and parameters

8. 8 Advanced Computational Toolkit Accessible MPI-based parallelism for finite difference grids Access to a variety of supercomputing architectures and clusters Several parallel I/O layers Fixed and Adaptive mesh refinement under development Elliptic solvers ? Parallel interpolators and reductions ? Metacomputing and distributed computing

9. 9 Collaborative Development Interactive monitoring, steering and visualization Enables sharing code base. TestSuite checking technology Visualization tools Exhaustive Numerical Relativity and Astrophysical Applications Black Hole coalescence Neutron star collisions Other cataclysms

10. 10 Applications If your only goal is to implement a simulation of a 1D wave equation, you probably don't want to use Cactus. You should consider using Cactus if you need to perform any of the following tasks - now or in future: perform parallel programming in an easy but powerful manner using the language of your choice: F77, F90, C, C++. run on a wide range of architectures and operating systems. develop your code on the most convenient machine available, for example your workstation or laptop...

11. 11 and simply move your code to a supercomputer (like a Cray T3E or Origin) to go into production right away! engage in high performance cluster computing or get first parallel experience by turning your networked PC pool into a computing cluster. work with collaborators on the same code and avoid having your programs fragmented. make use of the latest software technology: e.g. take advantage of research groups that spend their time thinking about the fastest way to bring simulation data to disk or how to visualize it while the code is running.

12. 12 Supported Architectures Intel IA32 Linux IBM SP SGI 32 bit Intel IA32 Windows NT Cray T3E SGI 64 bit Intel IA32 Windows 2000 Intel IA64 Linux Compaq Alpha Sun Sparc Hitachi SR8000-F1 MacOS X Fujitsu(Beta 11)

13. 13 Cactus uses the cygwin package to provide a unix-like environment in windows os. cvgwin

14. 14 Associated Packages MPI LAM Version 6.5.4 and earlier MPICH Version 1.2.2.1 and earlier MPICH-G2 HDF5 Version 1.4 and 1.2

15. 15 The Computational Toolkit Cactus is architectured to consist of modules (we call them thorns) which plug into a core code (we call it the flesh) which contains the APIs and infrastructure to glue the thorns together. The flesh on it's own doesn't actually do anything. The thorns contain all the real activity, and can usually be divided into application thorns (typically written by scientists, solving problems in physics, astrophysics, engineering, etc) and infrastructure thorns (typically written by computational scientists, providing IO, interpolations, drivers, elliptic solvers etc). We group thorns together into arrangements, depending on their functionality and applicability.

16. 16 Drivers Boundary Conditions Coordinate Systems Output Methods Interpolators and Reduction Operators Elliptic Solvers Utilities

17. 17

18. 18 Live Demos WaveToy Demo Our standard demonstration, see the description page for an explanation and how to run this yourself. Cactus Worm Our prototype dynamic Grid computing example... this is being developed to add new features and fault tolerance... please be understanding if it is down!

19. 19 E-Grid and Cactus The Cactus Team are closely involved with the activities of the European Grid Forum, in particular with the Working Group for Testbeds, which is headed by Ed Seidel. Cactus is being used by members of the E-Grid, along with the distributed computing toolkit Globus as an initial test application in a number of institutions.Globus

20. 20 Through work with the EGrid Consortium, Cactus will benefit from the development of Access to a Europe wide Grid-TestBed, joining computing resources at over ten institutions. More stable and secure methods of streaming data between simulation and various visualisation clients Development of procedures and thorns for automatic migrating Cactus jobs Ability to read streamed data into Cactus, e.g. checkpoint files Ability to stream checkpoint files from Cactus Increasing expertise in performing distributed runs

21. 21 E-Grid TestBed The E-Grid TestBed comprises a varied set of computational resources which was set up for demonstrations at SC2000 in Dallas, and provides a common and supported grid infrastructure with each site implementingdemonstrationsSC2000 Globus 1.1.4 gsiftpd gsisshd (using port 2222) MPICH-G2 (MPICH 1.2.1 with Globus device) HDF5 1.3.30 or above GRIS reporting to TestBed GIIS Cactus 4.0 Beta 9

22. 22 Episode Two: Solving Einstein ’ s Equations http://www.computer.org/computer /articles/einstein_1299_1.htm

23. 23 Einstein ’ s Equations Globally distributed scientific teams, linked to the most powerful supercomputers, are running visual simulations of Einstein ’ s equations on the gravitational effects of colliding black holes.

24. 24 In 1916, Albert Einstein published his famous general theory of relativity, which contains the rules of gravity and provides the basis for modern theories of astrophysics and cosmology. It describes phenomena on all scales in the universe, from compact objects such as black holes, neutron stars, and supernovae to large- scale structure formation such as that involved in creating the distribution of clusters of galaxies. For many years, physicists, astrophysicists, and mathematicians have striven to develop techniques for unlocking the secrets contained in Einstein ’ s theory of gravity; more recently, computational-science research groups have added their expertise to the endeavor.

25. 25 Those who study these objects face a daunting challenge: The equations are among the most complicated seen in mathematical physics. Together, they form a set of 10 coupled, nonlinear, hyperbolic-elliptic partial differential equations( 一组十对非线性双曲线－椭圆偏微分方程 ） that contain many thousands of terms. Despite more than 80 years of intense analytical study, these equations have yielded only a handful of special solutions relevant for astrophysics and cosmology, giving only tantalizing snapshots of the dynamics that occur in our universe.

26. 26 Scientists have gradually realized that numerical studies of Einstein ’ s equations will play an essential role in uncovering the full picture. Realizing that this work requires new tools, developers have created computer programs to investigate gravity, giving birth to the field of numerical relativity. Progress here has been initially slow, due to the complexity of the equations, the lack of computer resources, and the wide variety of numerical algorithms, computational techniques, and underlying physics needed to solve these equations.

27. 27 A realistic 3D simulation based on the full Einstein equations is an enormous task: A single simulation of coalescing neutron stars or black holes would require more than a teraflop per second for reasonable performance, and a terabyte of memory. Even if the computers needed to perform such work existed, we must still increase our understanding of the underlying physics, mathematics, numerical algorithms, high- speed networking, and computational science. These cross-disciplinary requirements ensure that no single group of researchers has the expertise to solve the full problem.

28. 28 Further, many traditional relativists trained in more mathematical aspects of the problem lack expertise in computational science, as do many astrophysicists trained in the techniques of numerical relativity. We need an effective community framework for bringing together experts from disparate disciplines and geographically distributed locations that will enable mathematical relativists, astrophysicists, and computer scientists to work together on this immensely difficult problem.

29. 29 Some examples Figure 1. Gravitational waves from a full 3D grazing merger of two black holes. The image shows the objects immediately after the coalescence, when the two holes (seen just inside) have merged to form a single, larger hole at the center. The distortions of the horizon (the Gaussian curvature of the surface, 高斯曲面 ) appear as a color map, while the resulting burst of gravitational waves (the even-parity polarization or real part of  4 ) appears in red and yellow.

30. 30 Figure 2. Additional polarization of gravitational waves (imaginary part of   ) from a 3D merging collision of two black holes. The merged single black-hole horizon can just be seen through the cloud of radiation emitted in the process.

31. 31 Figure 3. Close-up of the horizon—with Gaussian curvature to show the distorted nature of the surface—of a black hole formed by the collision of two black holes. The two individual horizon surfaces can just be seen inside the larger horizon formed during the collision process.

32. 32 Figure 4. Gravitational waves and horizon of a newly formed black hole, caused by massive collapse of a strong gravitational wave on itself. The dotted surface shows the horizon, where green colors indicate high curvature and yellow means zero curvature. The highest curvature indicates the largest gravitational radiation. The “distortion” (the non- sphericity) of the black hole radiates away over time, in accordance with mathematical theorems about black holes.

33. 33 Figure 5. Horizon of a gravitational wave that implodes to form a black hole, with leftover waves escaping, shown at a later time in the same evolution presented in Figure 4. We see that the horizon curvature is affected by, and correlated with, the evolving gravitational wave.

34. 34 Episode Three: WaveToy Demo

35. 35 including Remote monitoring and steering of an application from any web browser Streaming of isosurfaces( 等值面） from a simulation, which can then be viewed on a local machine Remote visualization of 2D slices from any grid function in a simulation as jpegs in a web browser

36. 36 The application we are using is the simulation of the 3D scalar field produced by two orbiting sources. The solution is found by finite differencing a hyperbolic partial differential equation （双曲线偏微分方程） for the scalar field. This is a very simple application, however it is representative of a large class of more complex systems, including Einstein's Equations, Maxwell's Equations, or the Navier-Stokes Equations. We use it for demonstrations since the simulation is not computationally intensive, is very robust, has simple parameter choices, and has reasonable graphics.

37. 37 Before you start, make sure you have the following items GetCactus the perl script for easily checking a Cactus application out from our CVS server. GetCactus WaveDemo.th the ThornList for the demo, this is used to tell GetCactus which thorns to get. WaveDemo.th WaveDemo.par a parameter file for running the demonstration. WaveDemo.par IsoView the isosurface visualization client. IsoView A web browser. Note that you'll need a live network connection to checkout the code, but you can run the demo on a single machine, the remote tools will look more impressive though if you use two networked machines, preferably a long way apart.

38. 38 Check out and compile Checkout the source code using something like perl GetCactus WaveDemo.th You should be able to use the default answers for all the questions Move into the Cactus directory (cd Cactus), and compile the application using either gmake WaveDemo-config gmake WaveDemo

39. 39 or if you have a configuration file (this is easiest, you don't need to remember the options you use) gmake WaveDemo-config options= gmake WaveDemo Hopefully that went OK, and you now have an executable, exe/cactus_WaveDemo. Check it really worked by running the testsuites, just type gmake WaveDemo-testsuite and use the default answers to each question.

40. 40 Run the demo Move the downloaded demo parameter file into the Cactus directory. To start the simulation, run your new executable with the demo parameter file, if you have a single processor executable./exe/cactus_WaveDemo WaveDemo.par If you compiled with MPI and have a multiprocessor version, you will need to use the appropriate mpi command for running.

41. 41 When the simulation starts, you will see output describing for example the activated thorns and the scheduling tree. If you have the simple visualization client xgraph installed, you can look at the 1D output

42. 42 Connecting with a web browser To connect to the simulation, move to another machine if you have one, and start up a web browser. Connect to http:// :5555 where :5555 is the name of the machine where the simulation is running. Note that this information was part of the standard output when the simulation started for example Server started on http://gullveig.aei-potsdam.mpg.de:5555/ Now you should see a screen with information about the simulation.

43. 43

44. 44 Viewing IsoSurfaces Start up the IsoView client, using IsoView -h -dp 5557 -cp 5558 Again, this information can be found in the standard output, for example Isosurfacer listening for control connections on gullveig.aei-potsdam.mpg.de port 5558/ Isosurfacer listening for data connections on gullveig.aei-potsdam.mpg.de port 5557/

45. 45 You should now see rotating blobs appearing in the client, looking something like this

46. 46 Steering the Simulation

47. 47 If you are watching the isosurfaces you should see the blobs move together.

48. 48 Episode Four: Some Other Sites NCSA DatalinkNCSA Datalink Article about Cactus from the NCSA Datalink news letter. http://archive.ncsa.uiuc.edu/datalink/9911/Cactus1.html http://archive.ncsa.uiuc.edu/datalink/9911/Cactus1.html TASC ReviewTASC Review Review of Cactus Framework by NASA/TASC http://sdcd.gsfc.nasa.gov/ESS/esmf_tasc/Files/Cactus_b.ht ml http://sdcd.gsfc.nasa.gov/ESS/esmf_tasc/Files/Cactus_b.ht ml IEEE ComputingIEEE Computing Cover story article about Cactus in IEEE Computing. http://www.computer.org/computer/articles/einstein_1299_1.htm http://www.computer.org/computer/articles/einstein_1299_1.htm Albert Einstein InstitutAlbert Einstein Institut: The birth place of the Cactus Code. http://www.aei-potsdam.mpg.de/ http://www.aei-potsdam.mpg.de/

49. 49 Thank you!