1. Maria Grazia Pia, INFN Genova Geant4 Electromagnetic Validation (mostly electromagnetic, but also a bit of hadronic…) K. Amako, G.A.P. Cirrone, G. Cuttone, F. Di Rosa, S. Guatelli, V. Ivanchenko, M. Maire, B. Mascialino, K. Murakami, P. Nieminen, L. Pandola, S. Parlati, M.G. Pia, G. Russo, T. Sasaki, L. Urban et al. Geant4 Workshop Bordeaux, 7-10 November 2005

2. Maria Grazia Pia, INFN Genova Geant4 Electromagnetic Validation Ample variety of physics models in the Geant4 Toolkit –complementary and alternative Electromagnetic physics –Standard, LowEnergy, Muon, Optical, Utilities (contains multiple scattering, energy loss!) Hadronic physics –in some experimental cases EM and hadronic physics cannot be separated Geant4 Physics Book –on-going project to document the performance of Geant4 physics against experimental data and in relevant experimental application domains

3. Maria Grazia Pia, INFN Genova Validation process Geant4 test process –Physics packages are subject to unit and system testing –Verification, validation of single processes/models performed by Working Groups Validation process –systematic –systematic : cover all models of a given process experimental data –comparison to experimental data and established reference databases –rigorous software process –rigorous software process to guarantee quality and reliability –statistical analysis –statistical analysis: quantitative mathematical evaluation Goals –evaluate quantitatively the accuracy of Geant4 physics models –document their respective strength –provide guidance to users to select appropriate models in their applications

4. Maria Grazia Pia, INFN Genova Scope of this talk Common project for Geant4 Electromagnetic Physics validation –systematic all –spanning all existing electromagnetic models –rigorous software process transparent –performed in a transparent way (agreed methods & tools, code in CVS) statistical analysis –quantitative results through rigorous statistical analysis publication-quality –goal: produce publication-quality results Other on-going “private” activities not covered in this talk –no time to present everything… –but also don’t know what is going on…

5. Maria Grazia Pia, INFN Genova Overview of recent validation activities Geant4 Physics Book: Electromagnetic Volume –comparison against the NIST databases –K. Amako, S. Guatelli, V. N. Ivanchenko, M. Maire, B. Mascialino, K. Murakami, P. Nieminen, L. Pandola, S. Parlati, M. G. Pia, M. Piergentili, T. Sasaki, L. Urban Comparison of Geant4 electromagnetic physics models against the NIST reference data IEEE Trans. Nucl. Sci., Vol. 52, Issue 4, Aug. 2005, 910-918 Current Physics Book projects (preliminary results) –Bremsstrahlung final state –Atomic relaxation and PIXE + Radioactivity from composite materials –Bragg peak (EM + hadronic) Other Geant4 validation activities –Validation of specific physics models done by each Geant4 Working Groups –LCG Simulation Validation Project: focus on hadronic physics –User own projects

6. Maria Grazia Pia, INFN Genova NIST Test Photon Mass Attenuation Coefficient Photon Partial Interaction Coefficient –related to the cross section of a specific photon interaction process Electron CSDA range and Stopping Power Proton CSDA range and Stopping Power  CSDA range and Stopping Power Elements Be, Al, Si, Fe, Ge, Ag, Cs, Au, Pb, U (span the periodic element table) Energy range photon 1 keV – 100 GeV electron 10 keV – 1 GeV proton 1 keV – 10 GeV  1 keV – 1 GeV Geant4 models : electrons and photons Standard Low Energy EEDL/EPDL Low Energy Penelope Geant4 models : protons and  Standard Low Energy ICRU49 Low Energy Ziegler 1977 Low Energy Ziegler 1985 Low Energy Ziegler 2000 (Low Energy: free electron gas + parameterisations + Bethe-Bloch) Simulation configuration reproducing NIST conditions (ionisation potential, fluctuations, production of secondaries etc.)

7. Maria Grazia Pia, INFN Genova G.A.P Cirrone, S. Donadio, S. Guatelli, A. Mantero, B. Mascialino, S. Parlati, M.G. Pia, A. Pfeiffer, A. Ribon, P. Viarengo “A Goodness-of-Fit Statistical Toolkit” IEEE- Transactions on Nuclear Science (2004), 51 (5): 2056-2063 Partly funded by ESA (SEPTIMESS Project)

8. Maria Grazia Pia, INFN Genova H0: Geant4 simulation = NIST data H1: Geant4 simulation ≠ NIST data Statistical analysis Goodness-of-Fit test (Statistical Toolkit) GoF test ( χ 2 test) Distance between Geant4 simulation and NIST reference data Test result p-value Geant4 simulation results + Reference Data The p-value represents the probability that the test statistics has a value at least at least as extreme as the one observed, assuming the null hypothesis is true 0 ≤ p ≤ 1 p < 0.05 p < 0.05 Geant4 simulation and NIST data differ significantly p > 0.05 p > 0.05 Geant4 simulation and NIST data do not differ significantly Alternative hypotheses under test:

9. Maria Grazia Pia, INFN Genova Photon mass attenuation coefficient Geant4 models: Standard Low Energy – EPDL Low Energy – Penelope Reference data: NIST - XCOM Mass attenuation coefficient in Fe Geant4 LowE Penelope Geant4 Standard Geant4 LowE EPDL NIST - XCOM Results All Geant4 models compatible with NIST Best agreement: Geant4 LowE models H 0 REJECTION AREA Monochromatic photon beam (I o ) Transmitted photons (I) p-value stability study Experimental set-up

10. Maria Grazia Pia, INFN Genova Compton interaction coefficient (cross section) Geant4 LowE Penelope Geant4 Standard Geant4 LowE EPDL NIST - XCOM Compton interaction coefficient in Ag p-value stability study H 0 REJECTION AREA Geant4 models: Standard Low Energy – EPDL Low Energy – Penelope Reference data: NIST - XCOM Results All Geant4 models compatible with NIST Best agreement: Geant4 LowE-EPDL

11. Maria Grazia Pia, INFN Genova Photoelectric interaction coefficient (cross section) Geant4 LowE Penelope Geant4 Standard Geant4 LowE EPDL NIST - XCOM Geant4 LowE Penelope Geant4 Standard Geant4 LowE EPDL NIST - XCOM Photoelectric interaction coefficient in Ge H 0 REJECTION AREA p-value stability study Geant4 models: Standard Low Energy – EPDL Low Energy – Penelope Reference data: NIST - XCOM Results All Geant4 models compatible with NIST Best agreement: Geant4 LowE models

12. Maria Grazia Pia, INFN Genova Pair production interaction coefficient (cross section) Geant4 LowE Penelope Geant4 Standard Geant4 LowE EPDL NIST - XCOM Pair production interaction coefficient in Au p-value stability study H 0 REJECTION AREA p-value (pair production interaction coefficient test) Geant4 models: Standard Low Energy – EPDL Low Energy – Penelope Reference data: NIST - XCOM Results All Geant4 models compatible with NIST and equivalent

13. Maria Grazia Pia, INFN Genova Rayleigh interaction coefficient (cross section) Results The Geant4 Low Energy models look in disagreement with the reference data for some materials Geant4 LowE Penelope Geant4 LowE EPDL NIST - XCOM Rayleigh interaction coefficient in Be Be0.991 Al0.32<0.05 Si0.77<0.05 Fe1<0.05 Ge<0.050.39 Ag0.360.08 Cs<0.05 Au<0.05 Pb<0.05 U EPDL XCOM Penelope XCOM Geant4 models: Low Energy – EPDL Low Energy – Penelope (no standard Rayleigh process) Reference data: NIST - XCOM

14. Maria Grazia Pia, INFN Genova Zaidi H., 2000, Comparative evaluation of photon cross section libraries for materials of interest in PET Monte Carlo simulation IEEE Transaction on Nuclear Science 47 2722-35 The disagreement is evident between 1 keV and 1 MeV photon energies For what concerns the Geant4 Low Energy EPDL model, the effect observed derives from an intrinsic inconsistency between Rayleigh cross section data in NIST-XCOM and the cross sections of EPDL97, on which the model is based Differences between EPDL97 and NIST-XCOM have already been highlighted in a paper by Zaidi, which recommends the Livermore photon and electron data libraries as the most up-to-date and accurate databases available for Monte Carlo modeling. Rayleigh interaction coefficient EPDL 97 NIST Rayleigh interaction coefficient in Au

15. Maria Grazia Pia, INFN Genova Electron Stopping Power Experimental set-up p-value stability study H 0 REJECTION AREA Geant4 LowE Penelope Geant4 Standard Geant4 LowE Livermore NIST - ESTAR Electrons are generated with random direction at the center of the box and stop inside the box Maximum step allowed in tracking particles was set about1/10 of the expected range value, to ensure the accuracy of the calculation Geant4 models: Standard Low Energy – EEDL Low Energy – Penelope Reference data: NIST – ESTAR (ICRU 37) Results All Geant4 models compatible with NIST and equivalent

16. Maria Grazia Pia, INFN Genova Electron CSDA Range CSDA : particle range without energy loss fluctuations and multiple scattering Geant4 LowE Penelope Geant4 Standard Geant4 LowE Livermore NIST - ESTAR CSDA range in U p-value stability study H 0 REJECTION AREA Geant4 models: Standard Low Energy – EEDL Low Energy – Penelope Reference data: NIST – ESTAR (ICRU 37) Results All Geant4 models compatible with NIST and equivalent

17. Maria Grazia Pia, INFN Genova Proton stopping power - range Proton stopping power - range Stopping power in Al Geant4 LowE Ziegler 1985 Geant4 LowE Ziegler 2000 Geant4 Standard Geant4 LowE ICRU 49 NIST - PSTAR + H 0 REJECTION AREA Stopping power: p-value stability study H 0 REJECTION AREA CSDA range: p-value stability study Results Ziegler parameterisations are as authoritative as ICRU 49 ones Comparison rather than validation

18. Maria Grazia Pia, INFN Genova  stopping power and range H 0 REJECTION AREA Stopping power: p-value stability study CSDA range in Si Geant4 LowE Ziegler 1977 Geant4 Standard Geant4 LowE ICRU 49 NIST - ASTAR The complex physics modeling of ion interactions in the low energy range is addressed by the Geant4 Low Energy package and it represented one of the main motivations for the developing of this package. The complex physics modeling of ion interactions in the low energy range is addressed by the Geant4 Low Energy package and it represented one of the main motivations for developing this package

19. Maria Grazia Pia, INFN Genova Quantitative process evaluation 2 conference presentations with preliminary results (CHEP, NSS 2003) Final results presented at NSS 2004, October 2004 ~ 10 months to put test infrastructure in place and get first results ~ 1 year to get publication-quality results ~ 3 months to write paper and produce final plots Publication –submitted to IEEE TNS March 2005 –accepted April 2005 –published August 2005 Workers/authors –13 authors –Barbara, Koichi, Sandra, Susanna + unnamed person did all the work –other people provided significant support to the project (funds, management, infrastructure etc.)

20. Maria Grazia Pia, INFN Genova Bremsstrahlung 3 sets of models available Standard: G4eBremsstrahlung Low Energy EEDL: G4LowEnergyBremsstrahlung Low Energy Penelope: G4PenelopeBremsstrahlung 3 angular distributions: Tsai, 2BS, 2BN Angular distribution of photons is strongly model-dependent Penelope Standard Low Energy EEDL (default) Penelope LowE-EEDL TSAI (def) LOWE-EEDL 2BS LOWE-EEDL 2BN Angle (deg) Susanna Guatelli Barbara Mascialino Luciano Pandola MG Pia

21. Maria Grazia Pia, INFN Genova Reference data The absolute Bremsstrahlung cross section can be tested Transmitted energy spectrum at two different emission angles for four materials (Al, Pb, W, Ag) Absolute yields are reported (= photons/primary), though with an “odd” normalization R. Ambrose et al., Nucl. Instr. Meth. B 56/57 (1991) 327

22. Maria Grazia Pia, INFN Genova Relative comparison... Relative comparison (45 degree direction) Shapes of the spectra are in good agreement Work in progress, will be published Intensity/Z (eV/sr keV) Photon energy (keV) LowE- Penelope Low E EEDL - TSAI Intensity/Z (eV/sr keV)

23. Maria Grazia Pia, INFN Genova Proton Bragg Peak Geant4 models: electromagnetic Standard Low Energy ICRU 49 Low Energy Ziegler 1977 Low Energy Ziegler 1985 Low Energy Ziegler 2000 Geant4 models: hadronic Precompound + default de-excitation Precompound + GEM evaporation with/without Fermi Break-up Binary Cascade (including Precompound + de-excitation options as above) Bertini Cascade Bertini Cascade + Bertini elastic scattering (when available) Parameterised Geant4 “educated guess” Medical Dosimetry Physics List Reference data from CATANA (INFN-LNS Hadrontherapy Group) Systematic test in progress Lot of work… Preliminary results Pablo Cirrone Giacomo Cuttone Francesco Di Rosa Susanna Guatelli Barbara Mascialino MG Pia Giorgio Russo

24. Maria Grazia Pia, INFN Genova EM only – Standard

25. Maria Grazia Pia, INFN Genova EM only – ICRU49 ENTIRE PEAK ExpG4 S2.892.43 T3.263.83 GoF testCVM-AD ENTIRE PEAK ExpG4 S2.892.43 T3.263.83 GoF testCVM-AD LEFT TAIL x<=30mm ExpG4 S9.7711.89 T2.663.16 GoF testCVM-AD RIGHT TAIL X>30mm ExpG4 S3.8912.24 T1.031.00 GoF testKS-AD

26. Maria Grazia Pia, INFN Genova EM only – ICRU49 – GoF results ALL (N 1 =149 N 2 =67) CVMAD Test statistics0.1129380.853737 p-value0.5250950.443831 LEFT TAIL x<=30mm (N 1 =140 N 2 =61) CVMAD Test statistics0.07015840.645422 p-value0.7505930.606120 RIGHT TAIL X>30mm (N 1 =9 N 2 =6) KSAD Test statistics0.3333330.816534 p-value0.7248710.469251

27. Maria Grazia Pia, INFN Genova LowE + precompound default ALLExpG4 S2.89 T3.26 GoF testCVM-AD LEFT TAIL x<=30mm ExpG4 S9.77 T2.66 GoF testAD RIGHT TAIL X>30mm ExpG4 S3.89 T1.03 GoF testKS-AD

28. Maria Grazia Pia, INFN Genova ICRU49 + precompound – GoF results ENTIRE PEAK (N 1 =149 N 2 =66) CVMAD Test statistics0.060.499375 p-value0.790.747452 LEFT TAIL x<=30mm (N 1 =140 N 2 =60) CVMAD Test statistics0.030.232255 p-value0.970.978972 RIGHT TAIL X>30mm (N 1 =9 N 2 =6) KSAD Test statistics0.330.901787 p-value0.730.413129 0.5250950.443831 ICRU 49 only 0.7505930.606120 0.7248710.469251

29. Maria Grazia Pia, INFN Genova Nuclear de-excitation alternative models Work in progress, more to come…

30. Maria Grazia Pia, INFN Genova Studies of environmental radioactivity from rocks and sands at the Gran Sasso Laboratory Geant4 can reproduce the result of a calibration with a 60 Co source (in the presence of the sample) very well detector sample source simulation data simulation data Radioactive spectrum Lower part of the histogram is not meaningful Anderson-Darling test Anderson-Darling test (for binned data) Lower E peak –A2 = 0.45 –p-value = 0.80 Higher E peak –A2 = 1.05 –p-value = 0.33 Both peaks –A2 = 0.80 –p-value = 0.48 Barbara Mascialino Luciano Pandola MG Pia

31. Maria Grazia Pia, INFN Genova Backscattering Transmission etc. Tests 1 st set of tests set-up for NSS 2003 BUT No time yet to bring them to publication quality Collaboration for systematic work desirable References in part from Standard EM tests

32. Maria Grazia Pia, INFN Genova Backscattering coefficient – E=100keV G4 LowE Lockwood et al. (1981) Angle of incidence (with respect to the normal to the sample surface) = 0°

33. Maria Grazia Pia, INFN Genova Comments Feedback from users is a helpful contribution to Geant4 validation systematic –but the systematic work needed to document Geant4 validity quantitatively can only come from the Geant4 Collaboration Better communication and collaboration is desirable –internal competition in the Geant4 Collaboration is not constructive –competition against Geant4 students and post-docs is not constructive –slow progress is due to limited womanpower fruitful Dedicated validation workshops are necessary and fruitful –not easy to organize (and expensive to attend) Regular VRVS Validation meetings will be organized in 2006 The key to success for publication-quality results is a rigorous software process

34. Maria Grazia Pia, INFN Genova Geant4 Physics Validation Workshop, Genova, July 2005 http://www.ge.infn.it/geant4/events/july2005/

35. Maria Grazia Pia, INFN Genova Conclusion More results available, no time to show them all… –multiple scattering? –atomic relaxation (A. Mantero, B. Mascialino, M.G. Pia, V. Zampichelli) –validation for specific use cases –etc. Systematic, quantitative validation of Geant4 EM physics in progress –all available models –rigorous statistical analysis A lot of work! –first paper published –several on-going projects –limited resources available The validation work provides valuable feedback for the improvement of Geant4 physics models

36. Maria Grazia Pia, INFN Genova Geant4 validation is not an easy job… experimental data often exhibit large differences!

37. Maria Grazia Pia, INFN Genova Publications It would be desirable to agree a publication plan for Geant4 physics validation –electromagnetic + hadronic IEEE Trans. Nucl. Sci. –major scientific journal about nuclear technologies and instrumentation –many Geant4-related papers published or currently in the review process –please consider publishing your results concerning Geant4 –copyright compatible with Geant4 Physics Book