1. Image Re-slicing for Parallel Computing IM&T ADVANCED SCIENTIFIC COMPUTING Mark Sedrak | Supervised by Darren Thompson & Sam Moskwa 13 February 2013 | Big Day In - Summer Vacation Project

2. Aim: To improve the existing cluster re-slicing routine in X-TRACT with Parallel Computing. Moore’s Law: Hardware and Data expansion. To be covered: –Image Re-Slicing. –Parallel Computing and the use of Super Computers. –My work through-out the project Image Re-Slicing for Parallel Computing| Mark Sedrak Project Introduction 2 |

3. What is it? –Slices from a CT reconstruction Synchrotron MRI Image Re-slicing 3 | Image Re-Slicing for Parallel Computing| Mark Sedrak

4. X-ray imaging tools for HPC clusters and the Cloud | Darren Thompson Reconstructed Image 4 |

5. Uses: –Medical Imaging –Image reconstruction (Materials, Objects, etc) XTRACT –Software developed by CSIRO Data Sizes Image Re-slicing 5 | N / M * N 2 float (projection / slice) NM float (sinogram) N 2 M float (all sinograms) N 3 float (all slices) 1k / 7204 MB2.8 MB2.8 GB4 GB 2k / 1,44016 MB11¼ MB22½ GB32 GB 4k / 2,88064 MB45 MB180 GB256 GB 8k / 5,760256 MB180 MB1.4 TB2 TB Image Re-Slicing for Parallel Computing| Mark Sedrak

6. Serial vs. Parallel Programming –Serial: Instructions are executed one-by-one in sequence. –Parallel: Instructions can be executed simultaneously. Splits the work Aspects of Parallel Systems Communication –Embarrassingly Parallel, Coarse-Grain Parallel, Fine-grain Parallel Memory –Shared Memory, Distributed Memory Problem Definition –Data Parallel, Task Parallel Parallel Computing 6 | Image Re-Slicing for Parallel Computing| Mark Sedrak

7. Super Computers –Clusters TBI “Mini Cluster”, MASSIVE Bragg: Dual 8-Core CPU’s, 128GB RAM, 40Gb/s InfiniBand Burnet (Specs): Dual 6-Core CPU’s, 48/96 GB RAM, 40Gb/s InfiniBand –File Systems GPFS, HNAS Message Passing Interface (MPI) –A Framework for sharing information between distributed memory processes –Different communication types: 1-1, 1-Many, Many-to-Many –Synchronous vs. Asynchronous Communication Supercomputers and Message Parsing 7 | Image Re-Slicing for Parallel Computing| Mark Sedrak

8. Optimising the re-slicing routine –Generic and portable Three main aspects –Communication –Computation/Shuffling –File I/O (Input/Output) Developed Three Main Methods –Method 1: (Single Mass communication, High Memory) –Method 2: (Multiple Smaller Communication, High Memory) –Method 3: (Multiple Smaller Communication, Low Memory) My Project 8 | Image Re-Slicing for Parallel Computing| Mark Sedrak

9. Results - 2k dataset 9 | Image Re-Slicing for Parallel Computing| Mark Sedrak

10. Results - Overview 10 | Image Re-Slicing for Parallel Computing| Mark Sedrak Shows the effectiveness of outputs for the Different Data Sizes, of M1,2,3 on both Burnet and Bragg 4k Data Set, 256 GB 7-10 min Shows method 1 compared with M2, on Both Burnt and Bragg Issues Shared users Resource Limits Bottlenecks (File System)

11. Image Re-Slicing Using Parallel Computing to Solve the Data Problem File I/O bottleneck, recommend a parallel file system. Summary 11 | Image Re-Slicing for Parallel Computing| Mark Sedrak

12. IM&T ASC Mark Sedrak Student eMark.Sedrak@csiro.au wwww.csiro.au IM&T ADVANCED SCIENTIFIC COMPUTING Thank you