1. Statistical Significance & Its Systematic Uncertainties
Shan JIN
Institute of High Energy Physics (IHEP)
November 26, 2007

2. How to quantify the possibility of a new discovery?
HEP Experiments have always been at the frontier of searching for and discovering new signals – new particles and new physics phenomena.
How to quantify the possibility of a new discovery?
How can your results be accepted?

3. International Convention in HEP Community
3 sigma – evidence of a possible signal
5 sigma – discovery of a signal
 What is statistical significance?
How to calculate/obtain RELIABLE or RIGOROUS statistical significance?
We need common “statistical language” to understand each other!

4. Outline Essence of Statistical Significance and its expressions
3 reliable/rigorous methods calculating statistical significance and their systematic uncertainties

5. Essence of Statistical Significance and Its Expressions
Essence：The PROBABILITY of a statistical test on the consistency with background hypothesis
Expressions:
It can be directly expressed as a Prob., of course.
It is more often intuitively to “be translated into n  ” according to a Gaussian prob. distribution.

6. Method I: Frequentist Method – 1- CLb (Widely used by LEP HiggsWG)
When we can obtain full knowledge
of background from MC simulation
-----Statistical Estimator
Test Statistic

7. Example: Simple Event Counting:
We expext: B=10000 events,
We observe: N0=10500 events
(1-sided probability of 5 ),
This is the probability that we observe Nb larger
than N0 in the pure background distribution.
Or we can understand it is as 5  deviation from
no signal.

8. Systematic Uncertainties for Method I
In this method, all possible factors causing the uncertainty of b should be taken into account.
Example: ALEPH’s observation of “3 golden Higgs candidate events” with ~3.0  significance:
Likelihood function includes number of events, mass and b-tagging distributions, etc

9. ALEPH Collaboration, PLB526 (2002) 191

10. Method II: Goodness of fit -- 2 tests (with known background shape)
d.o.f = Nbin – Npara
In statistics books, it reads “p-value it is the probability,
under the assumption of a hypothesis H0, of obtaining
data at least as incompatible with H0 as the data actually
observed.”
Sometimes，we can also” translate” this probability as
“n ” deviation from hypothesis H0.

11. Some examples

12. Observation of an anomalous enhancement near the threshold of mass spectrum at BES II
J/ygpp
acceptance weighted BW
M= MeV/c2
G < 30 MeV/c2 (90% CL)
c2/dof=56/56
0.1
0.2
0.3
Phys. Rev. Lett. 91, (2003)   
M(pp)-2mp (GeV)
3-body phase space
acceptance

13. Could it be a tail of a known resonance?
0-+ resonances in PDG tables:
h(1760) M= G = 60 MeV
p(1800) M= G = 210 MeV
2/dof=323/58
c2/dof=412/58

14. Pure FSI disfavored I=0 S-wave FSI CANNOT fit the BES data.
FSI curve from A.Sirbirtsev et al. ( Phys.Rev.D71:054010, 2005 ) in the fit (I=0)
FSI * PS * eff + bck

15. Systematic Uncertainties for Method II
Only those that may change the background shape need to be taken into account:
Some systematic errors, such as tracking efficiency, photon efficiency and 4c-fit, which have very small impact on the background shapes and the shape of acceptance curve, can be ignored, since they have littile contribution in the 2 calculation.

16. Method III: Likelihood Ratio Tests
This method can be applied to background shapes obtained from sideband fit. It is widely used by many experiments.
Two fits:
with signal: fit1  L1 ;
without signal: fit0  L0
Rigorous statistical theorem tell us that
follow the 2 distribution with
So, we have:

17. Using TOY MC experiments, it can also be easily shown that follows the 2 distribution with .
So, when we apply likelihood ratio test method, the number of d.o.f must be taken into account.
For a BW-like new signal, usually we have at least 3 parameters for the signal (mass, width and amplitude), so using to estimate signal significance is incorrect and it over estimates the significance by 0.7  when claiming a 5  discovery, i.e., the actual significance is only 4.3 .
( The probability is more than 10 times larger  BE CAREFUL! )

18. BESII Observation of X(1835) in
Statistical Significance 7.7 
BESII
The +- mass spectrum for  decaying into +- and  

19. Systematic Uncertainties for Method III
The factors cause the change of should be taken into account into the systematic uncertainties.
Since we obtain the background shape from sideband information, so the systematic uncertainties are mainly from the uncertainties of different choice of fitting functions and fitting range.
Some systematic errors, such as tracking efficiency, photon efficiency and 4c-fit, which have very small impact on the background shapes, can be ignored.

20. Example on Systematic Uncertainty of Significance: Observation of Y(2175) in J/  f0(980) at BESII

21. BESII preliminary Fit with one resonance
BG shape is fixed to , f0 sideband BG
BESII preliminary
5.5 
M =2.186±0.010 0.006 GeV/c2
=0.065±0.023  GeV/c2
N events= 5212
M(f0(980)) GeV/c2

22. BESII preliminary Fit with one resonance
BG is represented by a 3-order polynomial
BESII preliminary
4.9 
M =2.182±0.010 GeV/c2
=0.073±0.024 GeV/c2
N events= 6114

23. BESII preliminary Fit with two resonances
BG shape is fixed to , f0 sideband BG
the mass and width of the second peak
are fixed to those of from BaBar.
BESII preliminary
5.8 
2.5 
M =2.186±0.010GeV/c2
= 0.065±0.022GeV/c2
N1 events= 4714
N2 events= 2211

24. Summary
The Essence of the statistical significance is the PROBABILITY of a statistical test on the consistency with background hypothesis It can be expressed as “n” according to a Gaussian p.d.f.
Three RELIABLE/RIGOROUS methods are recommended and discussed.
The systematic uncertainty considerations on the significance depend on different information used.

25. or is NOT recommended in the significance calculation.

26. Comments on statistical significance （Plenary Talk by S
Comments on statistical significance （Plenary Talk by S.Jin at ICHEP04)
Using to estimate statistical significance seems too optimistic. Even if we have firm knowledge on the background  CLb as LEP Higgs used is recommended.
When the background is estimated from the fit of sideband, the likelihood ratio with D.O.F. taken into consideration is a better estimator of statistical significance.
In this case, the uncertainty of all possible background shapes should be included in the uncertainty of significance.
Do not optimize/tune the cuts on the data! Determine the cuts based on MC optimization before looking at data.
The sys. uncertainty on significance from “bias” cut is hard to estimate.

27. 谢 谢！ Thanks！

28. A peak around 2175 MeV/c2 is observed
in J/  f0(980)
phase space
efficiency curve
BESII preliminary
M(f0(980)) GeV/c2
Backgrounds from
sideband estimation