1. July 2011 High-order schemes for high-frequency Helmholtz equation 1 Parallel solution of high-frequency Helmholtz equation using high- order finite difference schemes Dan Gordon Computer Science University of Haifa Rachel Gordon Aerospace Eng. Technion

2. July 2011 High-order schemes for high-frequency Helmholtz equation 2Outline  Background on Helmholtz equation  The CARP-CG parallel algorithm  Comparative results using low- and high-order finite difference schemes

3. July 2011 High-order schemes for high-frequency Helmholtz equation 3 The Helmholtz Equation  Eqn: -Δu - k 2 u = g (k = "wave no.")  c = speed of sound, f = frequency  Wave length: = c/f = 2/k  No. of grid pts per : N g = /h, h= mesh size  Shifted Laplacian approach: –Bayliss, Goldstein & Turkel, 1983 –Erlangga, Vuik & Oosterlee, 2004/06 introduced imaginary shift: -Δu – ( i  k 2 u = f

4. July 2011 High-order schemes for high-frequency Helmholtz equation 4 The Helmholtz Equation  Some other approaches: –Elman, Ernst & O'Leary, 2001 –Plessix & Mulder, 2003 –Duff, Gratton, Pinel & Vasseeur, 2007 –Bollhöfer, Grote & Schenk, 2009 –Osei-Kuffuor & Saad, 2010  This work: hi-order schemes following –Singer & Turkel, 2006 –Erlangga & Turkel, 2011 (to appear)

5. July 2011 High-order schemes for high-frequency Helmholtz equation 5 Difficulties with Helmholtz  High frequencies  small diagonal  2 nd order schemes require many grid points/wavelength  "Pollution effect": high frequency requires more than fixed number of grid points/wavelength (Babuška & Sauter, 2000)   high-order schemes required

6. July 2011 High-order schemes for high-frequency Helmholtz equation 6 CARP: block-parallel Kaczmarz  Given: Ax=b  "Normal equations": AA T y=b, x=A T y  Kaczmarz algorithm (1937) "KACZ" is SOR on normal equations  Relaxation parameter of KACZ is the usual relax. par. of SOR  Cyclic relax. par.: each eq. gets its own relax. par.

7. July 2011 High-order schemes for high-frequency Helmholtz equation 7 July 1, 2010Parallel solution of the Helmholtz equation7 KACZ: Geometric Description eq. 1 eq. 2 eq. 3 initial point

8. July 2011 High-order schemes for high-frequency Helmholtz equation 8 CARP: Component-Averaged Row Projections  A block-parallel version of KACZ  Equations divided into blocks (not necessarily disjoint)  Initial estimate: vector x=(x 1,…,x n )  Suppose component x 1 appears in 3 blocks  x 1 is “cloned” as y 1, z 1, t 1 in the different blocks.  Perform a KACZ iteration on each block (independently, in parallel)

9. July 2011 High-order schemes for high-frequency Helmholtz equation 9 CARP – Explanation (cont)  The internal iterations in each block produce 3 new values for the clones of x 1 : y 1 ’, z 1 ’, t 1 ’  The next iterative value of x 1 is x 1 ’ = (y 1 ’ + z 1 ’ + t 1 ’)/3  The next iterate is x’ = (x 1 ’,..., x n ’)  Repeat iterations as needed for convergence

10. July 2011 High-order schemes for high-frequency Helmholtz equation 10 CARP as Domain Decomposition xx y 0 1 1 domain A domain B external grid point of A clone of clone of x 1 Note: domains may overlap

11. July 2011 High-order schemes for high-frequency Helmholtz equation 11 Overview of CARP domain Adomain B KACZ iterations KACZ iterations averaging cloning KACZ in some superspace (with cyclic relaxation)

12. July 2011 High-order schemes for high-frequency Helmholtz equation 12 Convergence of CARP  Averaging Lemma: the component- averaging operations of CARP are equivalent to KACZ row-projections in a certain superspace (with =1)   CARP is equivalent to KACZ in the superspace, with cyclic relaxation parameters – known to converge

13. July 2011 High-order schemes for high-frequency Helmholtz equation 13 CARP Applications  Elliptic PDEs w/large convection term result in stiff linear systems (large off-diagonal elements) –CARP very robust on such systems, compared to leading solver & preconditioner combinations –Downside: Not always efficient  Electron tomography (ET) – joint work with J.-J. Fernández

14. July 2011 High-order schemes for high-frequency Helmholtz equation 14 CARP-CG: CG acceleration of CARP  CARP is KACZ in some superspace (with cyclic relaxation parameters)  Björck & Elfving (1979): developed CGMN, which is a (sequential) CG- acceleration of KACZ (double sweep, fixed relax. parameter)  We extended this result to allow cyclic relaxation parameters  Result: CARP-CG

15. July 2011 High-order schemes for high-frequency Helmholtz equation 15 CARP-CG: Properties  Same robustness as CARP  Very significant improvement in performance on stiff linear systems derived from elliptic PDEs  Very competitive runtime compared to leading solver/preconditioner combinations on systems derived from convection-dominated PDEs  Highly scalable on Helmholtz eqns

16. July 2011 High-order schemes for high-frequency Helmholtz equation 16 CARP-CG: Properties  On one processor, CARP-CG is identical to CGMN  Particularly useful on systems with LARGE off-diagonal elements –example: convection-dominated PDEs  Discontinuous coefficients are handled without requiring domain decomposition (DD)

17. July 2011 High-order schemes for high-frequency Helmholtz equation 17 Robustness of CARP-CG  KACZ inherently "normalizes" the eqns (eqn i is divided by ║ A i ║ 2 )  Normalization is generally useful for discontinuous coefficients  After normalization, the diagonal elements of AA T are all 1, and strictly greater than the off-diagonal elements  This is not diagonal dominance, but it makes the normal eqns manageable  Also: when diag of A decreases, sum of off-diag of AA T decreases.

18. July 2011 High-order schemes for high-frequency Helmholtz equation 18 Experiments with Hi-Order  Relax. par. = 1.5 for all problems  2 nd, 4 th & 6 th order central difference schemes, following –Singer & Turkel, 2006 –Erlangga & Turkel, 2011  Hi-order schemes  9-pt. stencil  Complex eqns: separated real & imag., interleaved equations (following Day & Heroux, 2001)

19. July 2011 High-order schemes for high-frequency Helmholtz equation 19 Problem 1 (with analytic sol'n)  Based on Erlangga & Turkel, 2011  Eqn: (Δ+k 2 )u = 0, on [-0.5,0.5]  [0,1]  bndry condition: Dirichlet on 3 sides: –u=0 for x=-0.5 and x=0.5 –u=cos(x) for y=0 –Sommerfeld: u y +iβu=0 for y=1, β 2 =k 2 -   Analytic solution: u = cos(x)exp(-iβy)  Grid points per : Ng = 9,12,15,18  Approx. 186,000 – 742,000 complex variables  One processor  k = 300

20. July 2011 High-order schemes for high-frequency Helmholtz equation 20 Prob. 1: rel-res for 2 nd, 4 th, 6 th order schemes

21. July 2011 High-order schemes for high-frequency Helmholtz equation 21 Prob. 1: rel-err for 2 nd, 4 th, 6 th order schemes

22. July 2011 High-order schemes for high-frequency Helmholtz equation 22 Prob. 1: rel-err for 2 nd, 4 th, 6 th order schemes

23. July 2011 High-order schemes for high-frequency Helmholtz equation 23 Problem 2 (with analytic soln)  Eqn: Δu + k 2 u = 0  Domain: [0,1][0,1]  Analytic sol'n: u=sin(x)cos(βy), β 2 =k 2 -   Dirichlet bndry cond determined by u on the boundaries  Grid points per : Ng = 9 to 18  Approx. 186,000 – 742,000 real variables  One processor  k = 300  2 nd, 4 th, 6 th order schemes

24. July 2011 High-order schemes for high-frequency Helmholtz equation 24 Prob. 2: rel-res for 2 nd, 4 th, 6 th order schemes

25. July 2011 High-order schemes for high-frequency Helmholtz equation 25 Prob. 2: rel-err for 2 nd, 4 th, 6 th order schemes

26. July 2011 High-order schemes for high-frequency Helmholtz equation 26 Prob. 2: rel-err, 6 th order, N g =9–18

27. July 2011 High-order schemes for high-frequency Helmholtz equation 27 Problem 3 (no analytic soln)  Eqn: Δu + k 2 u = 0  Domain: [0,1]  [0,1]  Bndry cond on y=0: discontinuity at midpt.: u(0.5,0)=1, u(x,0) = 0 for x ≠ 0 other sides: 1st order absorbing  Approx. 515,000 complex variables  Grid points per : N g = 15  One processor  k = 300  2 nd, 4 th, 6 th order schemes

28. July 2011 High-order schemes for high-frequency Helmholtz equation 28 Problem 3: evaluating the error  No analytic solution  Run 6 th order scheme to rel-res=10 -13  Saved result as “true” solution  Compared results of 2 nd, 4 th and 6 th order schemes with the “true” solution

29. July 2011 High-order schemes for high-frequency Helmholtz equation 29 Prob. 3: rel-err for 2 nd, 4 th, 6 th order schemes

30. July 2011 High-order schemes for high-frequency Helmholtz equation 30 Parallel Performance, 1 to 16 Proc. # proc12481216 Prob 1288135164634612544784983 Prob 2384739814328477455615691 Prob 3734473787441757277107842 No. iter for rel-res=10 -7, 6 th order, Ng=15, ~515,000 var.

31. July 2011 High-order schemes for high-frequency Helmholtz equation 31 Parallel Performance, 1 to 16 Proc. # proc12481216 rel-res= 10 -4 28816387504137 rel-res= 10 -7 810459243139113103 Problem 3: time (s), 6 th order scheme, Ng=15, ~515,000 var. Times taken on a 12-node cluster, 2 quad proc. per node

32. July 2011 High-order schemes for high-frequency Helmholtz equation 32 Prob. 2 & 3: rel-res for 1 to 16 processors

33. July 2011 High-order schemes for high-frequency Helmholtz equation 33 Summary  Hi-freq Helmholtz require hi-order schemes  CARP-CG is applicable to hi-freq Helmholtz with hi-order schemes  Parallel and simple  General-purpose – for problems with large off-diagonal elements and discontinuous coefficients

34. July 2011 High-order schemes for high-frequency Helmholtz equation 34 Other Potential Applications  Hi-order schemes for Helmholtz in homog & heterog 3D domains  Maxwell equations  Other physics equations  Saddle-point problems  Circuit problems  Linear solver in some eigenvalue methods

35. July 2011 High-order schemes for high-frequency Helmholtz equation 35 Publications and Software http://cs.haifa.ac.il/~gordon/pub.html CARP: SIAM J Sci Comp 2005 CGMN: ACM Trans Math Software 2008 Microscopy: J Parallel & Distr Comp 2008 Large convection + discont coef: CMES 2009 CARP-CG: Parallel Comp 2010 Normalization for discont coef: J Comp & Appl Math 2010 CARP-CG software: http://cs.haifa.ac.il/~gordon/soft.html THANK YOU!