Download presentation

Presentation is loading. Please wait.

Published byJarod Ailstock Modified over 2 years ago

1
**Michal Merta Alena Vašatová Václav Hapla David Horák**

Massively parallel implementation of Total-FETI DDM with application to medical image registration Michal Merta Alena Vašatová Václav Hapla David Horák DD21, Rennes, France

2
**Motivation solution of large-scale scientific and engineering problems**

possibly hundreds of millions DOFs linear problems non-linear problems non-overlapping, FETI methods with up to tens of thousands of subdomains usage of PRACE Tier-1 and Tier-0 HPC systems

3
**PETSc (Portable, Extensible Toolkit for Scientific computation)**

developed by Argonne National Laboratory data structures and routines for the scalable parallel solution of scientific applications modeled by PDE coded primarily in C language, but good FORTRAN support, can also be called from C++ and Python codes current version is 3.2 petsc-dev (development branch) is intensively evolving code and mailing lists open to anybody

4
PETSc components seq. / par.

5
**Trilinos developed by Sandia National Laboratories**

collection of relatively independent packages toolkit for basic linear algebra operations, direct and iterative solvers for linear systems, PDE discretization utilities, mesh generation tools etc. object oriented design, high modularity, use of modern C++ features (templating) mainly in C++ (Fortran and Python bindings) current version trilinos.sandia.gov

6
Trilinos components

7
**Both PETSc and Trilinos…**

Potential users PETSc „essential object orientation“ for programmers used to functional programming but seeking for more modular code recommended for C and FORTRAN users Trilinos „pure object orientation“ for programmers who are not scared of OOP, appreciate good SW design and have some experience with C++ even better extensibility and reusability, SW project management are parallelized on the data level (vectors & matrices) using MPI use BLAS and LAPACK – de facto standard for dense LA have their own implementation of sparse BLAS include robust preconditioners, linear solvers (direct and iterative) and nonlinear solvers can cooperate with many other external solvers and libraries (e.g. MATLAB, MUMPS, UMFPACK, …) support CUDA and hybrid parallelization are licensed as open-source

8
**Problem of elastostatics**

Let me introduce a simple model problem. Omega is a isotropic elastic body such as steel traverse. One side is fixed to the wall – Gamma_U – Dirichlet boundary. Others are loaded by prescribed surface traction. f

9
TFETI decomposition to apply TFET domain decomposition we tear the original body from the Dirichlet boundary decompose it into a system of elastic bodies of nearly the same size new artifical boundaries between them arise continuation on the artificial boundaries enforced by gluing conditions.

10
**Primal discretized formulation**

The FEM discretization with a suitable numbering of nodes results in the QP problem:

11
**Dual discretized formulation (homogenized)**

QP problem again, but with lower dimension and simpler constraints

12
**Primal data distribution, F action**

* very sparse … straightforward matrix distribution, given by a decomposition block diagonal embarrassingly parallel

13
**Coarse projector action**

* ? ? ? … can easily take 85 % of computation time if not properly parallelized!

14
**G preprocessing and action**

?

15
**Coarse problem preprocessing and action**

1 2 action ? 3 Currently used variant: B2 (PPAM 2011)

16
Coarse problem

17
HECToR phase 3 (XE6) the UK's largest, fastest and most powerful supercomputer supplied by Cray Inc., operated by EPCC uses the latest AMD "Bulldozer" multicore processor architecture 704 compute blades each blade with 4 compute nodes giving a total of 2816 compute nodes each node with two 16-core AMD Opteron 2.3GHz Interlagos processors → 32 cores per node total of cores each 16-core processor shares 16Gb of memory, in total 60 Tb theoretical peak performance over 800 Tflops

18
**Benchmark K+ implemented as direct solve (LU) of regularized K**

built-in CG routine used (PETSc.KSP, Trilinos.Belos) E = 1e6, = 0.3, g = 9.81 ms-2 HECToR

19
**Results # subds = # cores 1 4 16 64 256 1024 Prim. dim. Dual dim. 252**

31752 Dual dim. 252 1512 7056 30240 124992 508032 Solution time Trilinos 1.39 3.01 4.80 6.25 10.31 28.05 PETSc 1.14 2.66 4.16 4.74 4.92 5.84 # iterations 34 63 96 105 102 33 68 94 1 iter. time 4.48e-2 4.76e-2 5.00e-2 5.95e-2 9.81e-2 2.75e-1 3.46e-2 3.92e-2 4.42e-2 4.52e-2 4.69e-2 5.73e-2 stopping criterion: ||rk|| / || r0|| < 1e-5 without preconditioning

20
**Application to image registration**

Image registration is a crucial step of image processing if there is a need to compare or integrate information from two or more images. Given these images, a reference R and template T, the goal is to nd an optimal transformation in such way that T becomes, in a certain sense, similar to R. These images usually show the same scene, but taken at dierent times, from dierent viewpoints or by dierent sensors Usage in weather forecast, GIS, medicine, cartography, computer vision Process of integrating information from two (or more) different images Images from different sensors, different angles or/and times

21
**Application to image registration**

In medicine: Monitoring of growth of a tumour Therapy valuation Comparison of patient data with anathomical atlas Data from magnetic resonance (MR), computer tomography (CT), positron emission tomography (PET)

22
Elastic registration The task is to minimize the distance between two images 𝜑≔𝑥−𝑢(𝑥) 𝑇 𝑅

23
Elastic registration Parallelization using TFETI method

24
**Results # of subdomains 1 4 16 Primal variables 20402 81608 326432**

Dual variables 903 2641 8254 Solution time [s] 41 34.54 57.44 # of iterations 2467 990 665 Time/iteration [s] 0.01 0.03 0.08 stopping criterion: ||rk|| / || r0|| < 1e-5

25
Solution

26
**Conclusion and future work**

To consolidate PETSc & Trilinos TFETI implementation to the form of extensions or packages To further optimize the codes using core-hours on Tier-1/Tier-0 systems (PRACE DECI Initiative, HPC-Europa2) To extend image registration to 3D data

27
References KOZUBEK T. et al. Total FETI domain decomposition method and its massively parallel implementation. Accepted for publishing in Advances in Engineering Software. HORAK, D.; HAPLA, V. TFETI coarse space projectors parallelization strategies. Accepted for publishing in the proceedings of PPAM 2011, Springer LNCS, 2012. Zitova B., Flusser J., Image registration methods: a survey, Image and Vision Computing, Vol.21, No.11, 2003, pp

28
**Thank you for your attention!**

Similar presentations

OK

Domain Decomposition in High-Level Parallelizaton of PDE codes Xing Cai University of Oslo.

Domain Decomposition in High-Level Parallelizaton of PDE codes Xing Cai University of Oslo.

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google

Ppt on nature and history of democracy in nepal Ppt on chapter 3 atoms and molecules class Ppt on water resources for class 10 download Ppt on amplitude shift keying remote Ppt on gunn diode pdf Ppt on multiple myeloma Ppt on barack obama leadership characteristics Ppt on river pollution in hindi Ppt on object oriented programming with c++ textbook Sixth sense technology best ppt on surface