1. Cactus Workshop - NCSA Sep 27 - Oct 1 1999 Cactus For Relativistic Collaborations Ed Seidel Albert Einstein Institute eseidel@aei-potsdam.mpg.de

2. Cactus for Relativistic Collaborations n Back to Physics (the main goal of most of us today)… l Yesterday will make achieving this easier l But don’t lose sight of our physics goals... n What we hope to achieve through community code n Question: is it a good idea to have one code for a whole community? l Yes! But Cactus is a metacode; it is many codes; it a glue for codes… n Grand Challenges l First attempt to get large scale collaborations across disciplines to work l Now can really do it with such a collaborative, connective infrastructure to support entire communities! n Present overview today of: l What physics we have done with Cactus l What modules have been created and tested l Which are already in Cactus 4, what is coming from Cactus 3, what is NOT there yet

3. The Promise of a Community Code for Relativity and Astrophysics n Intellectual Property Philosophy l Use cactus to solve your own problems through standards for the community l Use publicly available thorns as you need them l Develop your own private thorns for your own research l Make them public when and only when you are ready (e.g., maybe you want to publish first!) n Sharing of work, modules, results much easier n Providing numerical relativity functionality to groups previously working in other areas, from mathematics to astrophysics l Much more activity in the field!! n Advanced developments in computational science made available through KDI, other research projects n Cactus Again, a new community code for 3D simulations

4. Publications so far with/about Cactus n “The Cactus Computational Collaboratory: Enabling Technologies for Relativistic Astrophysics, and a Toolkit for Solving PDE’s by Communities in Science and Engineering”, G. Allen, T. Goodale, and E. Seidel, 7th Symposium on the Frontiers of Massively Parallel Computation-Frontiers ’99, IEEE, (1999). n “Technologies for Collaborative, Large Scale Simulation in Astrophysics and a General Toolkit for solving PDE's in Science and Engineering”, E. Seidel, to appear in “Forschung und wissenschaftliches Rechnen”, T. Plesser and P. Wittenburg, eds., (Max-Planck-Gesellschaft, 1999). n “Numerical Relativity in a Distributed Environment”, W. Benger, I. Foster, J. Novotny, E. Seidel, J. Shalf, W. Smith, and P. Walker, SIAM PPP, (1999). n “Three Dimensional Numerical Relativity with a Hyperbolic Formulation”, C. Bona, J. Massó, E. Seidel, P. Walker, Physical Review D15, in press… n “Numerical Relativity As A Tool For Computational Astrophysics”, E. Seidel and Wai-Mo Suen, Journal of Computational and Applied Mathematics”, (1999). n “Test-beds and applications for apparent horizon finders in numerical relativity”, M. Alcubierre, S. Brandt, B. Brügmann, C. Gundlach, J. Masso, E. Seidel, and P. Walker, gr-qc/9809004 Physical Review D, (1998).

5. More publications... n “Collapse of 3D Gravitational Waves to form a Black Hole”, M. Alcubierre, G. Allen, B. Brügmann, G. Lanfermann, Edward Seidel, W.-M. Suen, and Malcolm Tobias, submitted to Physical Review Letters, (1999). n “Axisymmetry without Axisymmetry: A New Method for Spherical and Axisymmetric Simulations without Coordinate Singularities”, M. Alcubierre, B. Brügmann, E. Seidel, and J. Thornburg, in preparation for Computer Physics Communications, (1999). n “Towards an understanding of the stability properties of the 3+1 evolution equations in general relativity”, Miguel Alcubierre, Gabrielle Allen, Bernd Bruegmann, Edward Seidel, Wai-Mo Suen, submitted to Physical Review D, gr-qc/9908079, (1999). n “The Shapiro Conjecture: Prompt or Delayed Collapse in the head-on collision of neutron stars?”, Mark Miller, Wai-Mo Suen, Malcolm Tobias, gr-qc/9904041(1999). n “A Conformal Hyperbolic Formulation of the Einstein Equations”, Miguel Alcubierre, Bernd Brugmann, Mark Miller, Wai-Mo Suen, submitted to PRD, (1999) gr-qc/9903030. n “Three Dimensional Numerical General Relativistic Hydrodynamics I: Formulations, Methods, and Code Tests”, J. A. Font, M. Miller, W. Suen, M. Tobias, submitted to PRD, (1998), gr-qc/9811015. n “Robust evolution system for Numerical Relativity”, A. Arbona, C. Bona, J. Masso, J. Stela, submitted to PRD, (1999), gr-qc/9902053. n … (probably forgot a few…) n Many others in the pipeline, hopefully soon some from you...

6. Recent Large Scale Simulations of Cactus n Black Holes (prime source for GW) l Largest Ever Prod. Simulations: 384 3 l (80GB Memory Required) l Full 3D Distorted BH Evolutions, Extracting Waves l Increasingly complex collisions: now doing full 3D grazing collisions n Gravitational Waves l Study linear waves as testbeds l Move on to fully nonlinear waves l Interesting Physics: BH formation in full 3D! n Neutron Stars l Developing capability to do full GR hydro l Now can follow full orbits!

7. Evolving Pure Gravitational Waves n Probe GR in highly nonlinear regime l Form BH? l Critical Phenomena in 3D? –Choptuik, many others in 1D now –2D example of Abrahams and Evans l Naked singularities? l … Little known about generic 3D behavior n Take “Brill Wave” Initial data ds 2 =  4 (e 2q (d  2 +dz 2 )+  2 d  2 ) q = A f( ,z,  ) l Choose K ij (take time symmetry for now…) l Solve constraints Highly nonlinear waves

8. Subcritical Waves: Everything radiates away... Newman-Penrose  4 (showing gravitational waves) with lapse underneath

9. Supercritical Waves (form Black Hole!)  4 showing collapse to BH, AH with gaussian curvature

10. Comparison of full non-axisymmetric and axisymmetric supercritical collapse to BH l=2,m=2 wave extraction 3D Axisym. simulation QNM fit to two lowest modes for BH of appropriate mass BH formingFormed BH Ringing

11. Now try first 3D “Grazing Collision”: Spinning, “orbiting”, unequal mass BHs merging. n Preliminary evolution results l Can be evolved beyond through coalescence l AH’s merge early l 384 3 (largest we’ve attempted) But just grazing collision: have a long way to go to get full orbits...

12. Apparent Horizon of Coalescing Holes z-axis x-axis t=5M Merging of AH’s Without excision, BH’s will not move across grid...

13. Gravitational Radiation from the Grazing Collision (inner 256 3 of 384 3 simulation shown) Re(  4 ) (red: even-parity radiation) Im(  4 ) (blue: odd-parity radiation)

14. More than a pretty movie: Extracting Waveforms   l=2,m=0 even-parity   l=2,m=2 even-parity   l=2,m=0 odd-parity Waves extracted very close to the final BH: 8M Very preliminary, but modes show QNM-like ringing… Each carry < 0.001M adm in energy

15. ( M AH ) 2 = (M ir ) 2 + J 2 /(4 (M ir ) 2 ) M AH = 3.08 Growth in Area of Apparent Horizon M ir = (A AH /16   ≈  Spurious Growth (error) Merged AH Appears (A AH /16   Time (M)

16. Total Mass of Spacetime: M ADM = 3.11 Total Mass of Final BH: M AH = 3.08 M ADM - M AH = 0.03 = 0.0097M ADM Compare to… Total energy radiated in all modes (integrated only to t=30M): M RAD ~ 0.007 - 0.008M ADM M RAD ≈ M ADM - M AH !! Total Energy Accounting of Grazing Collision

17. Specific Thorn of interest: “Cartoon 2D” n Axisymmetric systems difficult in axisym coords. Singularities (e.g. 1/(r sin  )) n Special coordinate systems introduce problems to be solved only there…don’t carry over to 3D (e.g. special “diagonal gauge”) n Routines develop for 3D cartesian systems will usually not work in 2D axisymmetric coordinate systems, and vice versa. n Would like to be able to study axisymmetric systems as testbeds in 3D, yet cartesian simulations take forever, require huge amounts of memory (~N 3 )… n Would like to use the SAME tools in 2D or 3D, SAME equations, SAME gauge conditions, etc...

18. Cartoon 2D n Basic Idea: l Axisymmetric system lives in a single plane l Can do a “thin slab” in x-z plane, 3 zones thick l Then need boundary conditions on “ghost” zones, which are provided by rotations of scalar, vector, and tensor quantities being evolved n Now implemented in Cactus: use on, e.g., BH problems studied yesterday!! n All 3D cartesian modules should work... x-z plane Ghost boundary zones Rotate interior data after each time step to boundaries

19. Scheduling n Basic Skeleton idea l Initial data l Evolution loop –Evolve –Analysis and IO n Extend it with scheduling groups l A certain thorn A must be called before another one B and after C –Example: Initialization of different pieces some elliptic solve to perform after initializing the fields Then, some other fields are initialized based on the elliptic solution l Some thorn must be called while some condition is met n Future redesign l The scheduler is really a runtime selector of the computation flow. l We can add much more power to this concept