Statistical Significance & Its Systematic Uncertainties Shan JIN Institute of High Energy Physics (IHEP) jins@mail.ihep.ac.cn November 26, 2007

How to quantify the possibility of a new discovery? HEP Experiments have always been at the frontier of searching for and discovering new signals – new particles and new physics phenomena. How to quantify the possibility of a new discovery? How can your results be accepted?

International Convention in HEP Community 3 sigma – evidence of a possible signal 5 sigma – discovery of a signal  What is statistical significance? How to calculate/obtain RELIABLE or RIGOROUS statistical significance? We need common “statistical language” to understand each other!

Outline Essence of Statistical Significance and its expressions 3 reliable/rigorous methods calculating statistical significance and their systematic uncertainties

Essence of Statistical Significance and Its Expressions Essence：The PROBABILITY of a statistical test on the consistency with background hypothesis Expressions: It can be directly expressed as a Prob., of course. It is more often intuitively to “be translated into n  ” according to a Gaussian prob. distribution.

Method I: Frequentist Method – 1- CLb (Widely used by LEP HiggsWG) When we can obtain full knowledge of background from MC simulation -----Statistical Estimator Test Statistic

Example: Simple Event Counting: We expext: B=10000 events, We observe: N0=10500 events (1-sided probability of 5 ), This is the probability that we observe Nb larger than N0 in the pure background distribution. Or we can understand it is as 5  deviation from no signal.

Systematic Uncertainties for Method I In this method, all possible factors causing the uncertainty of b should be taken into account. Example: ALEPH’s observation of “3 golden Higgs candidate events” with ~3.0  significance: Likelihood function includes number of events, mass and b-tagging distributions, etc

ALEPH Collaboration, PLB526 (2002) 191

Method II: Goodness of fit -- 2 tests (with known background shape) d.o.f = Nbin – Npara In statistics books, it reads “p-value it is the probability, under the assumption of a hypothesis H0, of obtaining data at least as incompatible with H0 as the data actually observed.” Sometimes，we can also” translate” this probability as “n ” deviation from hypothesis H0.

Some examples

Observation of an anomalous enhancement near the threshold of mass spectrum at BES II J/ygpp acceptance weighted BW +3 +5 -10 -25 M=1859 MeV/c2 G < 30 MeV/c2 (90% CL) c2/dof=56/56 0.1 0.2 0.3 Phys. Rev. Lett. 91, 022001 (2003)    M(pp)-2mp (GeV) 3-body phase space acceptance

Could it be a tail of a known resonance? 0-+ resonances in PDG tables: h(1760) M=1760 G = 60 MeV p(1800) M=1801 G = 210 MeV 2/dof=323/58 c2/dof=412/58

Pure FSI disfavored I=0 S-wave FSI CANNOT fit the BES data. FSI curve from A.Sirbirtsev et al. ( Phys.Rev.D71:054010, 2005 ) in the fit (I=0) FSI * PS * eff + bck

Systematic Uncertainties for Method II Only those that may change the background shape need to be taken into account: Some systematic errors, such as tracking efficiency, photon efficiency and 4c-fit, which have very small impact on the background shapes and the shape of acceptance curve, can be ignored, since they have littile contribution in the 2 calculation.

Method III: Likelihood Ratio Tests This method can be applied to background shapes obtained from sideband fit. It is widely used by many experiments. Two fits: with signal: fit1  L1 ; without signal: fit0  L0 Rigorous statistical theorem tell us that follow the 2 distribution with So, we have:

Using TOY MC experiments, it can also be easily shown that follows the 2 distribution with . So, when we apply likelihood ratio test method, the number of d.o.f. must be taken into account. For a BW-like new signal, usually we have at least 3 parameters for the signal (mass, width and amplitude), so using to estimate signal significance is incorrect and it over estimates the significance by 0.7  when claiming a 5  discovery, i.e., the actual significance is only 4.3 . ( The probability is more than 10 times larger  BE CAREFUL! )

BESII Observation of X(1835) in Statistical Significance 7.7  BESII The +- mass spectrum for  decaying into +- and  

Systematic Uncertainties for Method III The factors cause the change of should be taken into account into the systematic uncertainties. Since we obtain the background shape from sideband information, so the systematic uncertainties are mainly from the uncertainties of different choice of fitting functions and fitting range. Some systematic errors, such as tracking efficiency, photon efficiency and 4c-fit, which have very small impact on the background shapes, can be ignored.

Example on Systematic Uncertainty of Significance: Observation of Y(2175) in J/  f0(980) at BESII

BESII preliminary Fit with one resonance BG shape is fixed to , f0 sideband BG BESII preliminary 5.5  M =2.186±0.010 0.006 GeV/c2 =0.065±0.023  0.017 GeV/c2 N events= 5212 M(f0(980)) GeV/c2

BESII preliminary Fit with one resonance BG is represented by a 3-order polynomial BESII preliminary 4.9  M =2.182±0.010 GeV/c2 =0.073±0.024 GeV/c2 N events= 6114

BESII preliminary Fit with two resonances BG shape is fixed to , f0 sideband BG the mass and width of the second peak are fixed to those of from BaBar. BESII preliminary 5.8  2.5  M =2.186±0.010GeV/c2 = 0.065±0.022GeV/c2 N1 events= 4714 N2 events= 2211

Summary The Essence of the statistical significance is the PROBABILITY of a statistical test on the consistency with background hypothesis. It can be expressed as “n” according to a Gaussian p.d.f. Three RELIABLE/RIGOROUS methods are recommended and discussed. The systematic uncertainty considerations on the significance depend on different information used.

or is NOT recommended in the significance calculation.

Comments on statistical significance （Plenary Talk by S Comments on statistical significance （Plenary Talk by S.Jin at ICHEP04) Using to estimate statistical significance seems too optimistic. Even if we have firm knowledge on the background  CLb as LEP Higgs used is recommended. When the background is estimated from the fit of sideband, the likelihood ratio with D.O.F. taken into consideration is a better estimator of statistical significance. In this case, the uncertainty of all possible background shapes should be included in the uncertainty of significance. Do not optimize/tune the cuts on the data! Determine the cuts based on MC optimization before looking at data. The sys. uncertainty on significance from “bias” cut is hard to estimate.

谢 谢！ Thanks！

A peak around 2175 MeV/c2 is observed in J/  f0(980) phase space efficiency curve BESII preliminary M(f0(980)) GeV/c2 Backgrounds from sideband estimation