Barbara MascialinoMonte Carlo 2005Chattanooga, April 19 th 2005 Monte Carlo Chattanooga, April 2005 B. Mascialino, A. Pfeiffer, M. G. Pia, A. Ribon, P. Viarengo

Barbara MascialinoMonte Carlo 2005Chattanooga, April 19 th 2005 Provide tools for the statistical comparison of distributions  equivalent reference distributions  experimental measurements  data from reference sources  functions deriving from theoretical calculations or fits Detector monitoring Simulation validation Reconstruction vs. expectation Regression testing Physics analysis Data analysis

Barbara MascialinoMonte Carlo 2005Chattanooga, April 19 th 2005 Qualitative evaluation Quantitative evaluation GoF statistical toolkit A project to develop a statistical comparison system A project to develop a statistical comparison system Comparison of distributions Goodness of fit testing

Barbara MascialinoMonte Carlo 2005Chattanooga, April 19 th 2005 United Software Development Process tailoredUnited Software Development Process, specifically tailored to the project RUP –practical guidance and tools from the RUP –both rigorous and lightweight –mapping onto ISO ISO 15504Guidance from ISO Incremental and iterative life cycle model Software process guidelines SPIRAL APPROACH

Barbara MascialinoMonte Carlo 2005Chattanooga, April 19 th 2005 solid architectural approachThe project adopts a solid architectural approach functionalityquality –to offer the functionality and the quality needed by the users maintainable –to be maintainable over a large time scale extensible –to be extensible, to accommodate future evolutions of the requirements Component-based approachComponent-based approach –to facilitate re-use and integration in different frameworks AIDAAIDA –adopt a (HEP) standard –no dependence on any specific analysis tool Architectural guidelines

Barbara MascialinoMonte Carlo 2005Chattanooga, April 19 th 2005

Barbara MascialinoMonte Carlo 2005Chattanooga, April 19 th 2005 The tests are specialised on the kind of distribution (binned/unbinned)

Barbara MascialinoMonte Carlo 2005Chattanooga, April 19 th 2005 G.A.P Cirrone, S. Donadio, S. Guatelli, A. Mantero, B. Mascialino, S. Parlati, M.G. Pia, A. Pfeiffer, A. Ribon, P. Viarengo “A Goodness-of-Fit Statistical Toolkit” IEEE- Transactions on Nuclear Science (2004), 51 (5): Release StatisticsTesting-V downloadable from the web:

Barbara MascialinoMonte Carlo 2005Chattanooga, April 19 th 2005 binnedApplies to binned distributions It can be useful also in case of unbinned distributions, but the data must be grouped into classes Cannot be applied if the counting of the theoretical frequencies in each class is < 5 –When this is not the case, one could try to unify contiguous classes until the minimum theoretical frequency is reached –Otherwise one could use Yates’ correction Chi-squared test

Barbara MascialinoMonte Carlo 2005Chattanooga, April 19 th 2005 EMPIRICAL DISTRIBUTION FUNCTION ORIGINAL DISTRIBUTIONS Kolmogorov-Smirnov test Goodman approximation of KS test Kuiper test D mn Tests based on maximum distance unbinned distributions SUPREMUMSTATISTICS

Barbara MascialinoMonte Carlo 2005Chattanooga, April 19 th 2005 Fisz-Cramer-von Mises test Approx Anderson-Darling test Tests containing a weighting function binned/unbinned distributions EMPIRICAL DISTRIBUTION FUNCTION ORIGINAL DISTRIBUTIONS QUADRATICSTATISTICS+ WEIGHTING FUNCTION Sum/integral of all the distances

Barbara MascialinoMonte Carlo 2005Chattanooga, April 19 th 2005 shielded The user is completely shielded from both statistical and computing complexity. USER EXTRACTS THE ALGORITHM WRITING ONE LINE OF CODE TOOLKIT STATISTICALRESULT User’s point of view Simple user layer Simple user layer AIDA objects comparison algorithm Only deal with AIDA objects and choice of comparison algorithm

Barbara MascialinoMonte Carlo 2005Chattanooga, April 19 th 2005 Software testing Rigorous software process adopted Test process Unit tests Integration tests System tests qualitycorrectness robustness Testing focuses primarily on the evaluation or assessment of quality of the software product, guaranteeing its correctness and robustness. finding and documenting defects in software quality validating software product functions as designed validating that the requirements have been implemented appropriately Test result summaries are included as part of the documentation of the Toolkit release and are available on the web.

Barbara MascialinoMonte Carlo 2005Chattanooga, April 19 th 2005 Weighted KS tests Weighted CVM tests CVM approximation to a  2 (Tiku test) Exact Anderson-Darling test Watson test Watson approximation to a  2 (Tiku test) With these tests the GoF Statistical Toolkit will be the most complete toolkit for the two-sample problem in physics as well as in the statistics domain. Work in progress: new tests unbinned distributions binned/unbinneddistributions supremum statistics supremum statistics quadratic statistics quadratic statistics

Barbara MascialinoMonte Carlo 2005Chattanooga, April 19 th 2005  2 loses information in a test for unbinned distribution by grouping the data into cells Kac, Kiefer and Wolfowitz (1955) showed that Kolmogorov-Smirnov test requires n 4/5 observations compared to n observations for  2 to attain the same power Cramer-von Mises and Anderson-Darling statistics are expected to be superior to Kolmogorov-Smirnov’s, since they make a comparison of the two distributions all along the range of x, rather than looking for a marked difference at one point 2222 2222 Supremum statistics tests Tests containing a weight function < < In terms of power: Is Is  2 the most powerful algorithm? The power of a test is the probability of rejecting the null hypothesis correctly

Barbara MascialinoMonte Carlo 2005Chattanooga, April 19 th 2005 Examples of practical applications

Barbara MascialinoMonte Carlo 2005Chattanooga, April 19 th 2005 Mi Microscopic validation of physics p-value H 0 REJECTION AREA The three Geant4 models are equivalent Physics models under test: Geant4 Standard Geant4 Low Energy – Livermore Geant4 Low Energy – Penelope Reference data: NIST ESTAR - ICRU 37 Z p-value stability study Geant4 LowE Penelope Geant4 Standard Geant4 LowE EEDL NIST - XCOM Geant4 LowE Penelope Geant4 Standard Geant4 LowE EEDL K. Amako, S. Guatelli, V. Ivanchenko, M. Maire, B. Mascialino, K. Murakami, P. Nieminen, L. Pandola, S. Parlati, A. Pfeiffer, M. G. Pia, M. Piergentili, T. Sasaki, L. Urban Precision validation of Geant4 electromagnetic physics

Barbara MascialinoMonte Carlo 2005Chattanooga, April 19 th 2005 Radioprotection Radioprotection applications in manned space missions different shielding materials shielding thickness E.m.hadronic interactions -Comparison of inflatable and conventional rigid habitat concepts: Effect of different shielding materials Effect of shielding thickness E.m. and hadronic interactions S. Guatelli, B. Mascialino, P. Nieminen, M. G. Pia Radioprotection for interplanetary manned missions inflatable habitat GCR vacuum air phantom Al structure/ inflatable structure + shielding 2.15 cm Al Inflatable habitat + 10 cm water Inflatable habitat + 5 cm water 4 cm Al Energy deposit in the phantom by GCR p S. Guatelli, B. Mascialino, P. Nieminen, M. G. Pia Radioprotection for interplanetary manned missions thanks to Susanna Guatelli KS TEST

Barbara MascialinoMonte Carlo 2005Chattanooga, April 19 th 2005  2 not appropriate (< 5 entries in some bins, physical information would be lost if rebinned) Anderson-Darling A c (95%) =0.752 A. Mantero, B. Mascialino, P. Nieminen, M. G. Pia, A. Owens, M. Bavdaz, A. Peacock A library for simulated X-ray emission from planetary surfaces Test beam at Bessy Bepi-Colombo mission Energy (keV) Counts X-ray fluorescence spectrum in Iceand basalt (E IN =6.5 keV) Very complex distributions thanks to Alfonso Mantero

Barbara MascialinoMonte Carlo 2005Chattanooga, April 19 th 2005 Medical applications-IMRT RangeDp-value -84  -60 mm  -48 mm  47 mm  59 mm  84 mm RangeDp-value -56  -35 mm  -22 mm  21 mm  32 mm  36 mm Kolmogorov-Smirnov test Distance (mm) % dose F. Foppiano, B. Mascialino, M. G. Pia, M. Piergentili Geant4 simulation of an accelerator head for intensity modulated radiotherapy thanks to Michela Piergentili

Barbara MascialinoMonte Carlo 2005Chattanooga, April 19 th 2005 newup-to-dateeasy to handlepowerful This is a new up-to-date easy to handle and powerful tool for statistical comparison in particle physics. sophisticated and powerful statistical tests It the first tool supplying such a variety of sophisticated and powerful statistical tests in HEP. Releaseddownloadable from the web Released and downloadable from the web. AIDA AIDA interfaces allow its integration in any other data analysis tool. Applications in: HEP, astrophysics, medical physics, … Conclusions