1. Part I Status of GATE-related projects (Clermont-Ferrand; May 2017)
Liliana Caldeira
Mirjam Lenz & U we Pietrzyk
Institute of Neurosciences and Medicine (INM), Research Center Juelich, Germany
School of Mathematics and Natural Sciences, University of Wuppertal, Germany

2. List of Activities  FZ-Jülich / Uni Wuppertal
Liliana Caldeira, PhD
Improvements for running GATE
on a HPC Cluster (CPU/GPU)

3. JURECA
JURECA (Juelich Research on Exascale Cluster Architectures) supercomputer at the Jülich Supercomputing Center:
1872 compute nodes with two Intel Xeon E v3 Haswell CPUs
75 of these nodes are additionally equipped with two NVIDIA K80 GPUs.
workload manager Slurm (Simple Linux Utility for Resource Management).

4. GATE on JURECA
GATE v7.2
Additionally the cluster tools of GATE were adapted to support the Slurm workload manager for both the CPU and GPU simulation
The GPU simulations were performed with a new version of the existing GPU code that additionally produces phantom scatter information
Phantom scatter flag is missing in output
Source position is also at the borders of phantom ( BOX instead of Brain-like shape)

5. Work in Progress
Comparison of the CPU and GPU version of GATE on the number of true, scatter, and random coincidence events
Differences between CPU and GPU are always under 2% for all cases (true, random and scatter)
# detected coincidences
Note: Now trues and scatter are separated,
before, scatter looked like trues

6. Configuration PER NODE
Work in Progress
Provide an overview for the full runtime of the simulation.
Compare the runtime of the CPU and GPU simulations.
code
Configuration PER NODE
runtime
GATE 7.2
GATE 7.2 with changes/sc
CPU: 1 simulation
GPU: 1 simulation
2:20:52
0:20:05
CPU: 24 simulations
(in parallel)
3:01:57
GPU: 48 simulations
8:48:02
7:02:22
3600s simulation was split in one-second frames.
Weak scaling tests  optimal: CPU 24 Jobs per node GPU 48 Jobs per node
GPU does not scale as well as CPU!
GPU group in Juelich is looking at it! * *
Full runtime GPU = 3600s/48s * 7 hours ?
Full runtime CPU = 3600s/24s * 3 hours ?

7. Work in Progress  Code
Bug - GPU simulation stops / crashes before end of simulation time (increasing time and simulation will go pass that point).
Incorporate the phantom scatter flag in ROOT output directly in GATE (and not a posteriori as it is now) – someone who knows could help us.
168s – 169s, duration 1 s
crashes at 168.9s, duration 0.9s
168s -170s, duration 2s
crashes at 169.8s, duration 1.75s

8. Contact
Prof. Dr. Uwe Pietrzyk
Institute of Neuroscience and Medicine (INM-4)
- Medical Imaging Physics -
Forschungszentrum Jülich GmbH
and
School of Mathematics and Natural Science
- Medical Physics -
Bergische Universität Wuppertal
Germany