Comparison of data distributions: the power of Goodness-of-Fit Tests B. Mascialino1, A. Pfeiffer2, M.G. Pia1, A. Ribon2, P. Viarengo3 1INFN Genova, Italy 2CERN, Geneva, Switzerland 3IST – National Institute for Cancer Research, Genova, Italy IEEE – NSS 2006 San Diego, October 29-November 5, 2006

Goodness of Fit testing Goodness-of-fit testing is the mathematical foundation for the comparison of data distributions Regression testing Throughout the software life-cycle Online DAQ Monitoring detector behaviour w.r.t. a reference Simulation validation Comparison with experimental data Reconstruction Comparison of reconstructed vs. expected distributions Physics analysis Comparisons of experimental distributions Comparison with theoretical distributions Use cases in experimental physics THEORETICAL DISTRIBUTION SAMPLE ONE-SAMPLE PROBLEM SAMPLE 2 SAMPLE 1 TWO-SAMPLE PROBLEM

“A Goodness-of-Fit Statistical Toolkit” G.A.P Cirrone, S. Donadio, S. Guatelli, A. Mantero, B. Mascialino, S. Parlati, M.G. Pia, A. Pfeiffer, A. Ribon, P. Viarengo “A Goodness-of-Fit Statistical Toolkit” IEEE- Transactions on Nuclear Science (2004), 51 (5): 2056-2063. B. Mascialino, M.G. Pia, A. Pfeiffer, A. Ribon, P. Viarengo “New developments of the Goodness-of-Fit Statistical Toolkit” IEEE- Transactions on Nuclear Science (2006), 53 (6), to be published http://www.ge.infn.it/statisticaltoolkit/

GoF algorithms in the Statistical Toolkit TWO-SAMPLE PROBLEM GoF algorithms in the Statistical Toolkit Binned distributions Anderson-Darling test Chi-squared test Fisz-Cramer-von Mises test Tiku test (Cramer-von Mises test in chi-squared approximation) Unbinned distributions Anderson-Darling test Anderson-Darling approximated test Cramer-von Mises test Generalised Girone test Goodman test (Kolmogorov-Smirnov test in chi-squared approximation) Kolmogorov-Smirnov test Kuiper test Tiku test (Cramer-von Mises test in chi-squared approximation) Weighted Kolmogorov-Smirnov test Weighted Cramer-von Mises test

Unbinned Distributions Performance of the GoF tests AVERAGE CPU TIME Binned Distributions Unbinned Distributions Anderson-Darling (0.69±0.01) ms (16.9±0.2) ms Anderson-Darling (approximated) (0.60±0.01) ms (16.1±0.2) ms Chi-squared (0.55±0.01) ms Cramer-von Mises (0.44±0.01) ms (16.3±0.2) ms Generalised Girone (15.9±0.2) ms Goodman (11.9±0.1) ms Kolmogorov-Smirnov (8.9±0.1) ms Kuiper (12.1±0.1) ms Tiku (16.7±0.2) ms Watson (14.2±0.1) ms Weighted Kolmogorov-Smirnov (AD) (14.0±0.1) ms Weighted Kolmogorov-Smirnov (Buning) Weighted Cramer-von Mises

Power of GoF tests Which test to use? Do we really need such a wide collection of GoF tests? Why? Which is the most appropriate test to compare two distributions? How “good” is a test at recognizing real equivalent distributions and rejecting fake ones? No comprehensive study of the relative power of GoF tests exists in literature novel research in statistics (not only in physics data analysis!) Systematic study of all existing GoF tests in progress made possible by the extensive collection of tests in the Statistical Toolkit

two parent distributions Method for the evaluation of power The power of a test is the probability of rejecting the null hypothesis correctly Parent distribution 1 Parent distribution 2 Pseudo-experiment: a random drawing of two samples from two parent distributions GoF test Sample 1 n Sample 2 n N=10000 Monte Carlo replicas Confidence Level = 0.05 Power = # pseudo-experiments with p-value < (1-CL) # pseudo-experiments

Is there any recipe to identify the best test to use? Analysis cases Data samples drawn from different parent distributions Data samples drawn from the same parent distribution Applying a scale factor Applying a shift Use cases in experimental physics Signal over background “Hot channel”, dead channel etc. Power analysis on a set of reference mathematical distributions Power analysis on some typical physics applications Is there any recipe to identify the best test to use?

Parent reference distributions Uniform Gaussian Double Exponential Cauchy Exponential Contaminated Normal Distribution 2 Contaminated Normal Distribution 1 Left Tailed Pareto α= 1.0 α= 2.0 α= 3.0 α= 4.0

Parent Distribution SKEWNESS TAILWEIGHT f12(x) Pareto 4 0.037 1.647 0.076 1.488 f10(x) Pareto 2 0.151 1.351 f9(x) Pareto 1 0.294 1.245 f1(x) Uniform 1.000 1.267 f2(x) Gaussian 1.704 f6(x) Contamined Normal 1 1.991 f3(x) Double Exponential 2.161 f4(x) Cauchy 5.263 f7(x) Contamined Normal 2 1.769 1.693 f5(x) Exponential 4.486 1.883 f8(x) Exponential left tailed

Compare different distributions Parent1 ≠ Parent2 Unbinned distributions Compare different distributions Parent1 ≠ Parent2

The power increases as a function of the sample size Empirical power (%) Symmetric vs Skewed Medium tailed CN1 CN2 K W skewed GAUSSIAN AD CvM PARETO1 PARETO2 Short tailed WCvM EXPONENTIAL LEFT TAILED KS WKSB WKSAD No clear winner

The power varies as a function of the parent distributions’ characteristics Empirical power (%) Samples size = 50 EXPONENTIAL vs PARETO S1 – S2 Samples size = 15 GAUSSIAN OTHER DISTRIBUTIONS POWER CORRELATION COEFFICIENTS S1 – S2 T1 – T2 N 0.409 0.091 0.181 p<0.0001 General recipe p<0.0001

Quantitative evaluation of GoF tests power We propose a quantitative method to evaluate the power of various GoF tests. LINEAR MULTIPLE REGRESSION INCLUDE BOTH PARENT DISTRIBUTIONS’ CHARACTERISATION INCLUDE SAMPLES SIZE p<0.0001 < Standardised coefficients analysis:

Compare different distributions Parent1 ≠ Parent2 Binned distributions Compare different distributions Parent1 ≠ Parent2

Preliminary results CvM test More powerful Faster (CPU time) CAUCHY CN1 Samples size = 1000 Number of bins = 20 GAUSSIAN DOUBLE EXPONENTIAL CN2 χ2 = (38.91±0.49) CvM = (92.9 ± 0.26) χ2 = (98.67±0.12) CvM = (100.0 ± 0.0) χ2 = (50.32±0.50) CvM = (99.79 ± 0.05) χ2 = (100.0±0.0) χ2 = (77.72±0.42) CvM = (99.98 ± 0.02) χ2 = (65.04±0.48) CvM = (79.55 ± 0.40) χ2 = (33.23±0.47) CvM = (88.57 ± 0.32) χ2 = (92.83±0.26) CvM = (99.97 ± 0.02) χ2 = (99.95±0.02) CvM test More powerful Faster (CPU time)

Physics use case REAL PHYSICS DISTRIBUTIONS ARE MORE COMPLEX THAN THEORETICAL FUNCTIONS λ = 0.25 μ = 1.0 θ = 1.0 λ = 0.75 μ = 2.0 λ = 0.50 μ = 1.5

λ=0.25 μ=0.5 λ=0.25 μ=2.0 AD K KS CvM W WKSAD λ=0.75 μ=3.5 AD CvM λ=0.25 μ=0.5 λ=0.25 μ=2.0 AD K 40 KS CvM Empirical power (%) Empirical power (%) W WKSAD Samples size Samples size λ=0.75 μ=3.5 KS WKSB WKSAD CvM WCvM AD W K AD 40 Empirical power (%) CvM WKSAD Samples size

Conclusions No clear winner for all the considered distributions in general the performance of a test depends on its intrinsic features as well as on the features of the distributions to be compared Practical recommendations first classify the type of the distributions in terms of skewness and tailweight choose the most appropriate test given the type of distributions evaluating the best test by means of the quantitative model proposed Systematic study of the power in progress for both binned and unbinned distributions Topic still subject to research activity in the domain of statistics