1. THE CLs METHOD
Statistica per l'analisi dei dati – Dottorato in Fisica – XXIX Ciclo
Alessandro Pistone

2. Introduction
The ultimate goal of an experimental search for a new particle is to state whether
or not a statistically significant observation of the signal has been made
Is there a signal hidden in this data?

3. Statistical approach Define a test-statistic in order to separate
the signal+background hypothesis and
the background-only hypothesis
Compute from the observations
the observed value of the test-statistic
Decide to either fail to reject the null
hypothesis or reject it in favor of
an alternative hypothesis

4. Sensitive and insensitive experiments
CL s+b <0.05
with 1−CL b ≈0
CL s+b <0.05
with CL b ≈0
The experiment is not sensitive to the signal
The experiment is sensitive to the signal
Using this experiment to exclude the signal makes no sense!

5. The CLs method
The modified frequentist confidence level CL s ≡ CL s+b CL b
Normalizing CL s+b with CL b removes the dependence on background modelling:
more conservative limits on signal+background hypothesis
lower false exclusion rate than nominal 1−CL
This presentation!
A signal model is excluded at
95% confidence level if CL s <0.05

6. The model

7. The test-statistic: Likelihood-Ratio
Neyman-Pearson lemma most powerful test is the likelihood-ratio Q= L x|s+b L x|b with s b x L Q= i channels j bins s ij + b ij x ij e − s ij + b ij x ij ! b ij x ij e − b ij x ij ! −2ln Q = i channels j bins − s ij + x ij ln 1+ s ij b ij
signal
background
data
likelihood
Expected to converge to χ s+b 2 − χ b 2
in the high-statistics limit

8. The Likelihood-Ratio distributions
Distributions evaluated running O 100k pseudo-experiments for different signal hypothesis

9. Coverage: p.d.f. mode
Find Q ′ ⇔ CL s =0.05 and evaluate the corresponding CL s+b , then plot 1− CL s+b

10. Coverage: fit mode - i
Find Q ′ ⇔ CL s =0.05 and Q ′′ ⇔ CL s+b =0.05 for different signal hypothesis
2nd generation of pseudo-experiments O 10k÷50k for different signal hypothesis
Fit with theoretical distribution
and evaluation of Q obs
Exclusion if Q obs < Q ′ and Q obs < Q ′′

11. No sensible differences between χ 2 ≈1 and χ 2 ≈10
Coverage: fit mode - ii
More sensible to fluctuations w/r/t the p.d.f. mode, robustness tested against:
number of pseudo-experiments k sufficient
seed of pseudo-random number generator ‰ on coverage
errors used during the fit procedure
No sensible differences between χ 2 ≈1 and χ 2 ≈10

12. Conclusion
The correct behaviour of the coverage for the CL s method is reproduced