1. DataGrid is a project funded by the European Commission under contract IST-2000-25182 3 rd EU Review – 19-20/02/2004 Parallelization of Monte Carlo simulations GATE for medical applications The scenario of a typical radiotherapy treatment Lydia Maigne, Yannick Legré CNRS/IN2P3 maigne@clermont.in2p3.fr legre@clermont.in2p3.fr WP10

2. Parallelization of GATE simulations on the DataGrid - n° 2 Lydia MAIGNE, Yannick Legré Radiotherapy is widely used to treat cancer 1°) Obtain scanner slices images 2°) Treatment planning 3°) Radiotherapy treatment The head is imaged using a MRI and/or CT scanner Calculation of deposit dose on the tumor (~1mn): A treatment plan is developed using the images Irradiation of the brain tumor with a linear accelerator

3. Parallelization of GATE simulations on the DataGrid - n° 3 Lydia MAIGNE, Yannick Legré Better treatment requires better planning u Today: analytic calculation to compute dose distributions in the tumor n For new Intensity Modulated Radiotherapy treatments, analytic calculations off by 10 to 20% near heterogeneities u Alternative: Monte Carlo (MC) simulations in medical applications u The GRID impact: reduce MC computing time to a few minutes PET camera Ocular brachytherapy treatment Radiotherapy WP10 Demo: gridification of GATE MC simulation platform on the DataGrid testbed

4. Parallelization of GATE simulations on the DataGrid - n° 4 Lydia MAIGNE, Yannick Legré Computation of a radiotherapy treatment on the Datagrid: Let’s go….

5. Parallelization of GATE simulations on the DataGrid - n° 5 Lydia MAIGNE, Yannick Legré 1°) Obtain the medical images of the tumor: Scanner slices: DICOM format u 38 scanner slices of the brain of a patient are obtained  Format of the slices: 512 X 512 X 1 pixels  Size of a voxel in the image: 0,625 X 0,625 X 1,25 mm

6. Parallelization of GATE simulations on the DataGrid - n° 6 Lydia MAIGNE, Yannick Legré 2°) Concatenate these slices in order to obtain a 3D matrix: Pixies software Scanner slices: DICOM format Concatenation

7. Parallelization of GATE simulations on the DataGrid - n° 7 Lydia MAIGNE, Yannick Legré 3°) Transform the DICOM format of the image into an Interfile format Pixies software Number of slices Size of the pixels Size of the matrix Binary image of the scanner slices Image: text file Binary file: Image.raw Size 19M Scanner slices: DICOM format Concatenation Anonymisation

8. Parallelization of GATE simulations on the DataGrid - n° 8 Lydia MAIGNE, Yannick Legré 4°) Register and replicate the binary image on SEs: StorageElement ComputingElement StorageElement ComputingElement StorageElement ComputingElement StorageElement ComputingElement Image: text file Binary file: Image.raw Size 19M Scanner slices: DICOM format Anonymisation Concatenation Visualization CCIN2P3 RAL NIKHEF MARSEILLE User interface Copy image.raw on a SE Replicate lfns on the other SEs

9. Parallelization of GATE simulations on the DataGrid - n° 9 Lydia MAIGNE, Yannick Legré 5°) Register the lfn of an image: u WP2 spitfire or local database Image: text file Binary file: Image.raw Size 19M Scanner slices: DICOM format Database Anonymisation Concatenation StorageElement ComputingElement StorageElement ComputingElement StorageElement ComputingElement StorageElement ComputingElement CCIN2P3 RAL NIKHEF MARSEILLE Register lfn in the database Query a lfn to the database User interface

10. Parallelization of GATE simulations on the DataGrid - n° 10 Lydia MAIGNE, Yannick Legré 6°) Split the simulations: JobConstructor C++ program u A GATE simulation generating a lot of particles in matter could take a very long time to run on a single processor n So, the big simulation generating 10M of particles is divided into little ones, for example s 10 simulations generating 1M of particles s 20 simulations generating 500000 particles s 50 simulations generating 200000 particles …… n All the other files needed to launch Monte Carlo simulations are automatically created with the program. jdl files jobXXX.jdl Script files scriptXXX.csh Required files Macro files macroXXX.mac Status files statusXXX.rndm

11. Parallelization of GATE simulations on the DataGrid - n° 11 Lydia MAIGNE, Yannick Legré A typical jdl file:

12. Parallelization of GATE simulations on the DataGrid - n° 12 Lydia MAIGNE, Yannick Legré A typical script file:

13. Parallelization of GATE simulations on the DataGrid - n° 13 Lydia MAIGNE, Yannick Legré 7°) Submission on the DataGrid: u GUI of WP1: StorageElement ComputingElement StorageElement ComputingElement StorageElement ComputingElement StorageElement ComputingElement GATE Image: text file Binary file: Image.raw Size 19M Scanner slices: DICOM format Database User interface CCIN2P3 RAL NIKHEF MARSEILLE Anonymisation Concatenation Submission of jdls to the CEs Copy the medical image from the SE to the CE Retrieving of root output files from CEs

14. Parallelization of GATE simulations on the DataGrid - n° 14 Lydia MAIGNE, Yannick Legré GATE: Geant4 Application for Tomographic Emission GEANT4 core Dedicated developments SPECT/PET GATE  Develop a simulation platform for SPECT/PET imaging  Ensure a long term development Based on Geant4 Enrich Geant4 with dedicated tools SPECT/PET User friendly Effort of shared development Collaboration: OpenGATE Framework C++ classes Gate User interface Geant4  Based on Geant4  C++ object oriented langage  Reliable cross sections  GATE development  Ease of use  Command scripts to define all the parameters of the simulation  Framework: interface  modelisation of detectors, sources, patient  movement (detector, patient)  time-dependent processes (radioactive decay, movement management, biological kinetics)

15. Parallelization of GATE simulations on the DataGrid - n° 15 Lydia MAIGNE, Yannick Legré Parallelization technique of GATE u The random numbers generator (RNG) in GATE n CLHEP libraries: HEPJamesRandom (deterministic algorithm of F.James) s Characteristics: n Very long period RNG: 2 144 n Creation of 900 million sub-sequences non overlapping with a length of 10 30 n Pregeneration of random numbers s The Sequence Splitting Method s Until now, 200 status files generated with a length of 3.10 10 x1x2x3x4x5x6x7x8x9x10x11x12x13x14x15 Status 1Status 2Status 3 status000.rndmstatus001.rndmstatus002.rndm Each status file is sent on the grid with a GATE simulation

16. Parallelization of GATE simulations on the DataGrid - n° 16 Lydia MAIGNE, Yannick Legré 8°) Analysis of output root files u Typical dosimetry:  Merging of all the root files  Computation of the root data Brain_radioth000.root: 20 MB Brain_radioth001.root: 20 MB Brain_radioth002.root: 20 MB Brain_radioth003.root: 20 MB Brain_radioth004.root: 20 MB Brain_radioth005.root: 20 MB Brain_radioth006.root: 20 MB Brain_radioth007.root: 20 MB Brain_radioth008.root: 20 MB Brain_radioth009.root: 20 MB transversal view Centre Jean Perrin Clermont-Ferrand

17. Parallelization of GATE simulations on the DataGrid - n° 17 Lydia MAIGNE, Yannick Legré Conclusion and future prospects u The parallelization of GATE on the DataGrid testbed has shown significant gain in computing time (factor 10) u It is not sufficient for clinical routine u Necessary improvements n Dedicated resources (job prioritization) n Graphical User interface

18. Parallelization of GATE simulations on the DataGrid - n° 18 Lydia MAIGNE, Yannick Legré Aknowledgements u WP1: n Graphical User Interface, JobSubmitter u WP2: n Spitfire u WP6: n RPMs of GATE u WP8 u WP10: n 4D Viewer (Creatis) n Centre Jean Perrin n LIMOS u System administrators n Installations on UIs