1. Parallel Simulation of Continuous Systems: A Brief Introduction
Oct. 19, 2005
CS6236 Lecture

2. Background Sample applications of continuous systems Computer
simulations
Sample applications of continuous systems
Civil engineering: building construction
Aerospace engineering: aircraft design
Mechanical engineering: machining
Systems biology: heart simulations
Computer engineering: semiconductor simulations
Discrete models
Continuous models

3. Outline Mathematical models and methods Parallel algorithm methodology
Some active research areas

4. Mathematical Models Ordinary/partial differential equations
Laplace equation:
Heat (diffusion) equation:
Steady-state v.s. time-dependent
Convert into discrete problem through numerical discretization
Finite difference methods: structured grids
Finite element methods: local basis functions
Spectral methods: global basis functions
Finite volume methods: conservation

5. Example: 1-D Laplace Equation
Laplace equation in one dimension with boundary conditions
Finite difference approximation with
Jacobi iteration

6. Example: 2-D Laplace Equation
Laplace equation in two dimension with boundary conditions at four sides

7. Parallel Programming Model
Parallel computation: two or more tasks executing concurrently
Task encapsulates sequential program and local memory
Tasks can be mapped to processors in various ways, including multiple tasks per processor

8. Performance Considerations
Load balance: work divided evenly
Concurrency: work done simultaneously
Overhead: work not present in serial computation
Communication
Synchronization
Redundant work
Speculative work

9. Example: 1-D Laplace Equation
Define n tasks, one for each yi
Program for task i, i=1,…,n
Initialize yi for k=1,… if i>1, send yi to task i-1 if i<n, send yi to task i+1 if i<n, recv yi+1 from task i+1 if i>1, recv yi-1 from task i-1 yi = (yi-1+yi+1)/2 end

10. Design Methodology
Partition (Decomposition): decompose problem into fine-grained tasks to maximize potential parallelism
Communication: determine communication pattern among tasks
Agglomeration: combine into coarser-grained tasks, if necessary, to reduce communication requirements or other costs
Mapping: assign tasks to processors, subject to tradeoff between communication cost and concurrency

11. Design Methodology

12. Types of Partitioning Domain decomposition: partition data
Example: grid points in 1-, 2-, or 3-D mesh
Functional decomposition: partition computation
Example: components in climate model (atmosphere, ocean, land, etc.)

13. Example: Domain Decomposition
3-D mesh can be partitioned along any combination of one, two, or all three of its dimensions

14. Partitioning Checklist
Identify at least an order of magnitude more tasks than processors in target parallel system
Avoid redundant computation or storage
Make tasks reasonably uniform in size
Number of tasks, rather than size of each task, should grow as problem size increases

15. Communication Issues Latency and bandwidth Routing and switching
Contention, flow control, and aggregate bandwidth
Collective communication
One-to-many: broadcast, scatter
Many-to-one: gather, reduction, scan
All-to-all
Barrier

16. Communication Checklist
Communication should be reasonably uniform across tasks in frequency and volume
As localized as possible
Concurrent
Overlapped with computation, if possible
Not inhibiting concurrent execution of tasks

17. Agglomeration
Communication is proportional to surface area of subdomain, whereas computation is proportional to volume of subdomain
Higher-dimensional decompositions have more favorable communication-to-computation ratio
Increasing task sizes reduces communication but also reduces potential concurrency and flexibility

18. Surface-to-Volume Ratio

19. Example: Agglomeration
Define p tasks, each with n/p of yi’s
Program for task j, j=1,...p initialize yl,...,yh for k=1,... if j>1, send yl to task j-1 if j<p, send yh to task j+1 if j<p, recv yh+1 from task j+1 if j>1, recv yl-1 from task j-1 for i=l to h zi = (yi-1+yi+1)/2 end y = z end

20. Example: Overlap Comm/Comp
Program for task j, j=1,...p initialize yl,...,yh for k=1,... if j>1, send yl to task j-1 if j<p, send yh to task j+1 for i=l+1 to h-1 zi = (yi-1+yi+1)/2 end if j<p, recv yh+1 from task j+1 zh = (yh-1+yh+1)/2 if j>1, recv yl-1 from task j-1 zl = (yl-1+yl+1)/2 y = z end

21. Mapping Two basic strategies for assigning tasks to processors:
Place tasks that can execute concurrently on different processors
Place tasks that communicate frequently on same processor
Problem: These two strategies often conflict
In general, finding optimal solution to this tradeoff is NP-complete, so heuristics are used to find reasonable compromise
Dynamic vs static strategies

22. Mapping Issues Partitioning Granularity Mapping Scheduling
Load balancing
Particularly challenging for irregular problems
Some software tools: Metis, Chaco, Zoltan, etc.

23. Example: Atmosphere Model
Partitioning
grid points in 3-D finite difference model
Typically yields 105 to 107 tasks
Communication
9-point stencil horizontally and 3-point stencil vertically
Physics computations in vertical columns
Global operations to compute total mass

24. Example: Atmosphere Model

25. Other Equations Heat (diffusion) equation: Laplace equation:
Advection equation:
Wave equation:
Classification of second-order equations
Parabolic, hyperbolic, and elliptic
Methods for time-dependent equations
Explicit v.s. implicit
Finite-difference, finite-volume, finite-element

26. CFL Condition for Stability
Necessary condition named after Courant, Friedrichs, and Lewy
Computational domain of dependence must contain physical domain of dependence
Implies time step must satisfy

27. Active Research Areas
DES of continuous systems

28. Active Research Areas Coupling of different physics Load balancing
Different mathematical models
Continuous v.s. discrete techniques
Load balancing
Manager-worker model
Irregular/unstructured problems
Dynamic load balancing

29. Summary Mathematical models for continuous systems
Ordinary and partial differential equations
Finite difference, finite volume, and finite element
Parallel algorithm design
Partitioning
Communication
Agglomeration
Mapping
Active research areas

30. References
I. T. Foster, Designing and Building Parallel Programs, Addison-Wesley, 1995
A. Grama, A. Gupta, G. Karypis, and V. Kumar, Introduction to Parallel Computing, 2nd. ed., Addison-Wesley, 2003
M. J. Quinn, Parallel Computing: Theory and Practice, McGraw-Hill, 1994
K. M. Chandy and J. Misra, Parallel Program Design: A Foundation, Addison-Wesley, 1988