1 Titanium Review: Immersed Boundary Armando Solar-Lezama Biological Simulations Using the Immersed Boundary Method in Titanium Ed Givelberg, Armando Solar-Lezama, Hormozd Gahvari, Meling Ngo, Omair Kamil and Kathy Yelick Computer Science Division, UC Berkeley

2 Titanium Review, Sep. 9, 2004 Project Goals Construction of large scale computational models based on much finer fluid grid than is currently possible Development of distributed algorithms for immersed boundary computations on systems with hundreds of processors Development of a general-purpose software suite to support immersed boundary computations

3 Titanium Review, Sep. 9, 2004 Achievements Since last review Beating Heart Automatic Tissue partitioning 2D fluid partitioning (in progress) Second Order Method (in progress)

4 Titanium Review, Sep. 9, 2004 Immersed Boundary Method A general method for simulating systems containing elastic and possibly active tissue immersed in a viscous, incompressible fluid ANDPeskin AND McQueen – blood flow in the heart platelet aggregation during blood clotting (A. Fogelson) swimming organisms (L. Fauci) flow in collapsible tubes (M. Rozar) valveless pumping (E. Jung) two-dimensional cochlea model (R. Beyer) flapping of a flexible filament in a flowing soap film (L. Zhu)

5 Titanium Review, Sep. 9, 2004 Tissue activation & force calculation Interpolate Velocity Navier-Stokes Solver Spread Force Four Stage Algorithm Immersed Boundary Method

6 Titanium Review, Sep. 9, 2004 Heart Simulation Developed by Peskin and McQueen at NYU On a Cray C90: 1 heart-beat in 100 CPU hours point fluid mesh Heart represented by muscle fibers –Applications: Understanding structural abnormalities Evaluating artificial heart valves Source:

7 Titanium Review, Sep. 9, 2004 Heart Model Parameters Simulation time: Time Step: 24 microseconds, progressively refined down to 6 microseconds Approximately 100,000 time steps per experiment Total simulated time:.9 seconds Memory: <1 Giga-bytes Disk space for output files: Giga-bytes Fluid grid: 128 x 128 x 128 points Fluid grid mesh width: 1.3 millimeter Immersed material (4100 fibers, total ~650,000 points)

8 Titanium Review, Sep. 9, 2004 Beating Heart

9 Titanium Review, Sep. 9, 2004 Performance Challenges Fiber Partition and Distribution Communication reduction Vs Load Balance Fluid Partition Want to use more processors Take advantage of shared memory

10 Titanium Review, Sep. 9, 2004 Fiber Distribution A difficult problem Need a clever way to split fibers What’s good for load distribution may be bad for communication Fibers move around the fluid Expected motility of a tissue point depends on application

11 Titanium Review, Sep. 9, 2004 Clever Fiber Splitting Need a clever way to break fibers Force communication can kill performance Take advantage of linearity of spread All fiber-points in the same processor, no communication problems One point in different processor, need to communicate force across processors Duplicate point across processors, no force to communicate. Linearity keeps points synchronized

12 Titanium Review, Sep. 9, 2004 Formalizing the partitioning problem Key principles: Breaking fibers has a cost We want to maximize spread-box overlap This minimizes communication (x k1, y k1, z k1 ) (x k2, y k2, z k2 ) (x p2, y p2, z p2 ) (x p1, y p1, z p1 ) Amount overlap:= (4 - |x k1 – x k2 |) x (4 - |y k1 – y k2 |) x (4 - |z k1 – z k2 |)

13 Titanium Review, Sep. 9, 2004 Formalizing the partitioning problem We can encode this information in a graph The cost of an edge is determined as = Overlap(v1,v2) + Penalty(v1,v2) Penalty(v1,v2) is zero if v1 and v2 are not on the same fiber. Penalty(v1, v2) is a parameter that can be tuned based on how much fibers are expected to move The graph is too big it must be pruned Graph can be partitioned with Metis

14 Titanium Review, Sep. 9, 2004 Results of Fiber Splitting

15 Titanium Review, Sep. 9, 2004 Results of Fiber Splitting

16 Titanium Review, Sep. 9, 2004 Results of Fiber Splitting

17 Titanium Review, Sep. 9, 2004 Results of Fiber Splitting

18 Titanium Review, Sep. 9, 2004 Results of Fiber Splitting 15% performance increase over already tuned fiber distribution. Very general method Can be tuned for other applications with different tissue properties

19 Titanium Review, Sep. 9, 2004 Fluid Partitioning Current heart code partitions the fluid in only one dimension Partitioning in two dimensions increases the potential scalability Problem grows O(N^3), but with slab partitioning, processors can only grow O(N) Partitioning in more than 1D can greatly increase communication

20 Titanium Review, Sep. 9, 2004 SMP aware partitioning Give slabs to nodes rather than processors Processors in a node cooperate to compute on the slab We are still performance tuning this approach Need to address some false sharing issues

21 Titanium Review, Sep. 9, 2004 SMP aware partitioning Scalability of FFT

22 Titanium Review, Sep. 9, 2004 SMP aware partitioning Scalability of FFT

23 Titanium Review, Sep. 9, 2004 Second-Order Solver Implementation of Peskin and McQueen’s formally second-order accurate Immersed Boundary Method First ever implementation of this method in a distributed memory parallel environment This effort is currently in progress

24 Titanium Review, Sep. 9, 2004 Why a Second-Order Solver? Some problems require it when you use a finer grid to model the problem Captures information that you can’t see with the first-order method Will allow for a larger timestep in the heart

25 Titanium Review, Sep. 9, 2004 Current Implementation Status Version without sources and sinks has yet to be tested Version with sources-and-sinks breaks down on test case (pipe with one source and one sink) after 10 timesteps Reason: some extra math for the flow from the source needs to be taken into account

26 Titanium Review, Sep. 9, 2004 Future Work Continue work on Second Order Method Continue work on fluid partitioning Show feasibility of simulating a 256 x 256 x 256 heart