1. Geant4 profiling for medical physics
6th International Geant4 Tutorial in Korea
KIRAMS, Seoul, Korea
July 24~28, 2017
Geant4 profiling for medical physics
Kihyeon Cho and Insung Yeo
(KISTI)

2. Overview of Geant4 & KISTI
Geant4 Collaboration
1-1. Physics List and Validation Tool Kit Group (Chair: Gunter) => 0.1 FTE
& Test => Part I. KISTI
Tachyon2 (CPU: 200,000 CPU*h) – Andrea
Physics List using Beam – Chanyoung Lee
1-2. Quality and Assurance Group (Group: Soon Jun) => Part II
Profiling for Medical Physics (cf. SimpliCarlo)
Fermilab (Parallel) W/ Soon
2-1. Performance Test using GPU (cf )
2-2. Hadron Physics Parallelization (cf. EM Physics)

3. Part I.

4. History Stage1: 2009~2011: IT Group Stage2: 2012~ Present: HEP Group
Needs by community => Tutorials & Papers
Stage2: 2012~ Present: HEP Group
Challenge Program (2012) => Collaboration
Computational Science (2013~) => R&D

5. Stage1: 2009~2011 Ref. 황순욱박사 Activities Geant4 Workshop and Tutorials
Geant4 Workshop ( , KISTI)
Geant4 & GATE Tutorial 2010 개최( ~2.10, KISTI)
2010 Summer Training Course in Geant4, GATE and Grid Computing( ~7.10, Seoul)
2011 Geant4/GATE/Grid Tutorial for Medical Applications( ~11.4, KISTI)
Papers: KPS, Medical Physics Society
Ref. 황순욱박사

6. Joining Collaboration
KISTI colleagues have already made a big contribution in testing
of recently released prototype of multi-threaded Geant4, and in
enhancing our support on AIX operating system. We find KISTI
is rapidly becoming one of the two major development and
user-support centers in Asia, complementing with KEK in Japan
Geant4 Spokesperson
2012
Joining Collaboration

7. Stage2: 2012~Present Geant4 R&D HEP Grid Farm Supercomputer Community
(FKKPPL etc.)
Collaboration
Collaboration
HEP
Medical Physics
HEP
Medical Physics
Applica-
tion
Fermilab
-HEP
Applications
UC San Francisco
- Proton accelerator
R&D
Geant4 Collaboration
code development
Geant4 R&D
Infra-
structure
Grid Farm
Supercomputer
CERN
IN2P3
KEK

8. Community Supports R&D => One stop system(hep@kisti.re.kr)
CPU ( CPU*hour) – Andrea
R&D (Profiling)
Support(Porting)
FKPPL (Beam Simulation)
International Geant4 Tutorials
Collaborative tools: EVO (SeeVogh) => “KISTI” Community

9.
Geant has been ported at KISTI supercom (tachyon2) For the environment for mt or sequential as follows:   >source /applic/applications/hep/conf_scripts_new/g4mt.10.03_user_conf.bash   >source /applic/applications/hep/conf_scripts_new/g4mt.10.03_user_conf.tcsh Others are same on board #47. Now, the following versions are available on tachyon2:   Geant   Geant p01   Geant p02   Geant p03   Geant p04   Geant   Geant p01   Geant p02   Geant   Geant

10. Homepages Geant4 & Indico Homepage Geant4 user board
Geant4 user board
ID/Passwd
Q&A Contact person:

11. Summary KISTI Doing major role in Geant4 in Korea.
The official contact point in Geant4 Collaboration with NCC.

12. Geant4 profiling for medical physics
Part II.
Geant4 profiling for medical physics

13. Contents Introduction (Why?) Brachytherapy Methodology Results Summary
Low Energy Geant4 profiling code needed
Brachytherapy
Methodology
Flow chart
Results
CPU/Memory
Profiling Mesh
Profiling Events
Summary

14. Introduction Why profiling? Diversity of physics applications
Evolving Computing
S/W development
Profiling tools
To draw community interests for collateral effort

15. There are many efforts to develop HEP software to save CPU Time and Memory.(
However, there are not enough codes for profiling system.
=> To use FNAL Geant4 computing performance profiling protocol and tools as a starting point

16. Current status 1. High energy physics profiling (Fermilab)
TeV Scale 이상
SimpliCarlo (Sequential)
CMSExp (Multi-Thread)
2. Low energy physics profiling
Less than MeV Scale
Does not exists
Using Brachytherapy code

17. Brachytherapy Using Brachytherapy simulation in advanced examples
Brachytherapy seed

18. Goal on Brachytherapy Profiling
Low Energy Physics Profiling
CPU/Memory Usage
Mesh Size
Version dependency
Scalability for new computing architecture (KISTI supercomputer)

19. Bench mark @ current machine
The 4th KISTI
Rpeak
300TFlops
Nodes Number
3,176
Processor
Intel Xeon X GHz
CPU Number
25,408
Memory
76.8 TB
Storage
1,061TB
Service date

20. Scalability to Future Hybrid machine
The 5th KISTI
Cray?
100 X Scalability ?
Mini component supercomputer? 1PFlops
Rpeak
25PFlops
Price
$90 Million
Processor
CPU+Intel XeonPhi
Service date
=> delayed
Memory ??
Storage

21. Methodology Flowchart Minimum Lists of library & versions
Implementing profiling algorithm
RUN Igprof and Sprof
Make Summary plots
Minimum Lists of library & versions
gcc
cmake
Geant p ….

22. Methodology

23. Machines Machines OS batch jobs Input file outputfile Install Geant4
tachyon2
Linux
PBS -
Compile
Brachytherapy code
Igprof Run
kumac
IgProf_iodine.FTFP_INCLXX_HP.35.0_MEM_LIVE_1.txt
Sprof Run
g4profiling_1_X.tgz
Draw plot
hepkisti
Ouput of ig and sprof
png
Web site
html

24. Versions Geant4.10.01 Geant4.10.02 Release Date 2014.12 2015.12
Language
C++
C++11
gcc
4.4.6
(4.8 or more)
module load compiler/gcc-4.9.3
cmake
(3.3 or more)
/applic/common/utils/bin
make
3.81
Comments
Geant
Geant
SimpliCarlo
Done
Brachytherapy

25. SimpliCarlo vs. Brachytherapy
Energy
More than TeV
Less than MeV
CPU time
Long
Too short
CPU time / event (sec)
0.01
0.0002
Snapshot time 30 5
Institute
Fermilab
KISTI
Extended mode: Nfunction
Print + N Step , N Track information additionally

26. Settings Code Macro file Physics list Brachytherapy (v10.02)
IodiumSourceMacro.mac (endocavitary brachytherapy)
IridiumSourceMacro.mac (interstitial brachytherapy)
LeipzigSourceMacro.mac (superficial brachytherapy)
Physics list
QGSP_BIC_EMY
QGSP_BIC_EMZ

27. Physics List Name Content Energy QGS Quark Gluon String Model
> ~20 GeV
FTF
Fritiof string model
> ~ 5GeV
BIC
Binary Cascade
< ~ 10GeV
BERT
Bertini-style cascade
< ~10GeV HP
High Precision neutron model
>20MeV P
G4Precompund model used for
deexcitation

28. Results 1. CPU time and Memory 2. Profiling on mesh size
3. Profiling on number of events

29. 1. CPU time and memory Geant4 10.02 Number of events: 50,000 &10,000
Trial time: 528
Mesh size: 300
Snap shot time: 5 sec

30. <Trial time : 528>
(Events : 5001)
(Events : 10001)
5,000
10,000

31. #### Define the geometry of the I-125 source
/source/switch Iodium
###### Generation of primary field
#### Generate gamma deriving from radioactive decay
/control/execute iodine_source_primary.mac
#### ... or generate radioactive decay of iodine
#/control/execute iodine_decay.mac #
# Scoring mesh is used to calculate
# the energy deposition in the phantom
/score/create/boxMesh boxMesh_2
# the voxels are 1 mm wide.
/score/mesh/boxSize cm
/score/mesh/nBin
This example shows
model a radioactive source in terms of radiation field and geometry
model the radiation field with the General Particle Source

32. Events :5001
Events :10001

33. Events :5001
Events :10001

34. Events :5001
Events :10001

35. Events :5001
Events :10001

36. Events :5001
Events :10001

37. 2. Profiling on mesh size Mesh information Number of events: 10,000
CPU time: total 2.19 sec
Trial time: 8
Mesh size: 30, 300, 900, 1500, 3000, 6000
Snap shot time: 5 sec
PhysicsList: QGSP_BIC_EMY , QGSP_BIC_EMX

38. <Processing time>
Mesh size
Events : 10001
Iodine
QGSP BIC EMY
<Processing time> 30
300
900
1500
3000
6000 45
(min.) 50 60 70
110
153

39. Mesh size Events : 10001 CPU Time (sec/evt) Chi2 = 1.20654e-08 NDf = 4
p = / e-05
p = e / e-08
Iodine
QGSP BIC EMY
Total Memory of First Event
Chi =
NDf =
p = /
p = / e-06
Total Memory of Last Event (MB)
Chi =
NDf =
p = /
p = / e-05

40. Mesh size
Events : 10001
Iodine
QGSP BIC EMZ

41. Mesh size Events : 10001 CPU Time (sec/evt) Chi2 = 1.838e-08 NDf = 4
p = / e-05
p = e / e-08
Iodine
QGSP BIC EMZ
Total Memory of First Event
Chi =
NDf =
p = /
p = / e-06
Total Memory of Last Event (MB)
Chi =
NDf =
p = /
p = / e-05

42. 3. Profiling on number of events
Mesh information
Number of events: 5,000 ~ 16,000
Trial time: 8
Mesh size: 300 Snap shot time: 5 sec

43. Events : 5000
Events : 8000
Events : 10000
Events : 13000

44. Events : 15000 Events : 16000 Iodine QGSP BIC EMY Chi2 = 1.15413e-07
NDf =
p = /
p = e / e-08
Chi2 =
NDf =
p = /
p = / e-06

45. Iodine QGSP BIC EMZ Iridium QGSP BIC EMY Chi2 = 0.00565949
Chi = e-07
NDf =
p = /
p = e / e-08
Chi =
NDf =
p = /
p = / e-06
Iridium
QGSP BIC EMY
Chi = e-07
NDf =
p = /
p = e / e-08
Chi =
NDf =
p = /
p = / e-06

46. Iridium QGSP BIC EMZ Leipzig QGSP BIC EMY Chi2 = 1.15413e-07 NDf = 4
p = e / e-08
Chi =
NDf =
p = /
p = / e-06
Leipzig
QGSP BIC EMY
Chi = e-07
NDf =
p = /
p = e / e-08
Chi =
NDf =
p = /
p = / e-06

47. Leipzig QGSP BIC EMZ Chi2 = 1.15413e-07 NDf = 4
p = e / e-08
Chi =
NDf =
p = /
p = / e-06

48. Summary Shown Future Low Energy Physics Profiling code
CPU/Memory usage
Comparing profiling using mesh size & Events number => Linearity
Future
To usage: Geant4 profiling test => G4 Q&A
Version dependency
Scalability for new computing architecture (KISTI supercomputer)

49. Acknowledgement
Soon Yung Jun
Wonqook Choi

50. Thank you.