THE CLs METHOD Statistica per l'analisi dei dati – Dottorato in Fisica – XXIX Ciclo Alessandro Pistone

Introduction The ultimate goal of an experimental search for a new particle is to state whether or not a statistically significant observation of the signal has been made Is there a signal hidden in this data?

Statistical approach Define a test-statistic in order to separate the signal+background hypothesis and the background-only hypothesis Compute from the observations the observed value of the test-statistic Decide to either fail to reject the null hypothesis or reject it in favor of an alternative hypothesis

Sensitive and insensitive experiments CL s+b <0.05 with 1−CL b ≈0 CL s+b <0.05 with CL b ≈0 The experiment is not sensitive to the signal The experiment is sensitive to the signal Using this experiment to exclude the signal makes no sense!

The CLs method The modified frequentist confidence level CL s ≡ CL s+b CL b Normalizing CL s+b with CL b removes the dependence on background modelling: more conservative limits on signal+background hypothesis lower false exclusion rate than nominal 1−CL This presentation! A signal model is excluded at 95% confidence level if CL s <0.05

The model

The test-statistic: Likelihood-Ratio Neyman-Pearson lemma most powerful test is the likelihood-ratio Q= L x|s+b L x|b with s b x L Q= i channels j bins s ij + b ij x ij e − s ij + b ij x ij ! b ij x ij e − b ij x ij ! −2ln Q = i channels j bins − s ij + x ij ln 1+ s ij b ij signal background data likelihood Expected to converge to χ s+b 2 − χ b 2 in the high-statistics limit

The Likelihood-Ratio distributions Distributions evaluated running O 100k pseudo-experiments for different signal hypothesis

Coverage: p.d.f. mode Find Q ′ ⇔ CL s =0.05 and evaluate the corresponding CL s+b , then plot 1− CL s+b

Coverage: fit mode - i Find Q ′ ⇔ CL s =0.05 and Q ′′ ⇔ CL s+b =0.05 for different signal hypothesis 2nd generation of pseudo-experiments O 10k÷50k for different signal hypothesis Fit with theoretical distribution and evaluation of Q obs Exclusion if Q obs < Q ′ and Q obs < Q ′′

No sensible differences between χ 2 ≈1 and χ 2 ≈10 Coverage: fit mode - ii More sensible to fluctuations w/r/t the p.d.f. mode, robustness tested against: number of pseudo-experiments 10k sufficient seed of pseudo-random number generator 2‰ on coverage errors used during the fit procedure No sensible differences between χ 2 ≈1 and χ 2 ≈10

Conclusion The correct behaviour of the coverage for the CL s method is reproduced