Confidence Intervals Lecture 2 First ICFA Instrumentation School/Workshop At Morelia, Mexico, November 18-29, 2002 Harrison B. Prosper Florida State University

November 21, 2002First ICFA Instrumentation School/Workshop Harrison B. Prosper2 Recap of Lecture 1 To interpret a confidence level as a relative frequency requires the concept of a set of ensembles of experiments. To interpret a confidence level as a relative frequency requires the concept of a set of ensembles of experiments. Each ensemble has a coverage probability, which is simply the fraction of experiments in the ensemble with intervals that contain the value of the parameter pertaining to that ensemble. Each ensemble has a coverage probability, which is simply the fraction of experiments in the ensemble with intervals that contain the value of the parameter pertaining to that ensemble. The confidence level is the minimum coverage probability over the set of ensembles. The confidence level is the minimum coverage probability over the set of ensembles.

November 21, 2002First ICFA Instrumentation School/Workshop Harrison B. Prosper3 Count Parameter space Confidence Interval – General Algorithm Probability to be here is ≥ β Probability to be here is ≥ β )(Nl 

November 21, 2002First ICFA Instrumentation School/Workshop Harrison B. Prosper4 Coverage Probabilities θ Central Feldman-Cousins Mode-Centered N±√N

November 21, 2002First ICFA Instrumentation School/Workshop Harrison B. Prosper5 What’s Wrong With Ensembles? Nothing, if they are objectively real, such as: Nothing, if they are objectively real, such as: –The people in Morelia between the ages of 16 and 26 –Daily temperature data in Morelia during the last decade But the ensembles used in data analysis typically are not But the ensembles used in data analysis typically are not frequency interpretation repetition –There was only a single instance of Run I of DØ and CDF. But to determine confidence levels with a frequency interpretation we must embed the two experiments into ensembles, that is, we must decide what constitutes a repetition of the experiments. The problem is that reasonable people sometimes disagree about the choice of ensembles, but, because the ensembles are not real, there is generally no simple way to resolve disagreements. The problem is that reasonable people sometimes disagree about the choice of ensembles, but, because the ensembles are not real, there is generally no simple way to resolve disagreements.

November 21, 2002First ICFA Instrumentation School/Workshop Harrison B. Prosper Outline Deductive Reasoning Deductive Reasoning Inductive Reasoning Inductive Reasoning –Probability –Bayes’ Theorem Example Example Summary Summary

November 21, 2002First ICFA Instrumentation School/Workshop Harrison B. Prosper propositionsA Consider the propositions: A = (this is a baby) B B = (she cries a lot) Major premiseATRUEBTRUE Major premise: If A is TRUE, then B is TRUE Minor premiseATRUE Minor premise: A is TRUE Conclusion: BTRUE Conclusion: Therefore, B is TRUE Major premiseATRUEBTRUE Major premise: If A is TRUE, then B is TRUE Minor premiseBFALSE Minor premise: B is FALSE Conclusion: AFALSE Conclusion: Therefore, A is FALSE Aristotle, ~ 350 BC AB = A Deductive Reasoning

November 21, 2002First ICFA Instrumentation School/Workshop Harrison B. Prosper A A = (this is a baby) B B = (she cries a lot) Major premiseATRUEBTRUE Major premise: If A is TRUE, then B is TRUE Minor premiseAFALSE Minor premise: A is FALSE Conclusion: B? Conclusion: Therefore, B is ? Major premiseATRUEBTRUE Major premise: If A is TRUE, then B is TRUE Minor premiseBTRUE Minor premise: B is TRUE Conclusion: A? Conclusion: Therefore, A is ? AB = A Deductive Reasoning - ii

November 21, 2002First ICFA Instrumentation School/Workshop Harrison B. Prosper Major premiseATRUEBTRUE Major premise: If A is TRUE, then B is TRUE Minor premiseBTRUE Minor premise: B is TRUE Conclusion: Amore plausible Conclusion: Therefore, A is more plausible Major premiseATRUEBTRUE Major premise: If A is TRUE, then B is TRUE Minor premiseAFALSE Minor premise: A is FALSE Conclusion: Bless plausible Conclusion: Therefore, B is less plausible plausible Can “plausible” be made precise? Inductive Reasoning A A = (this is a baby) B B = (she cries a lot)

November 21, 2002First ICFA Instrumentation School/Workshop Harrison B. Prosper Bayes(1763), Laplace(1774), Boole(1854), Cox(1946) Jeffreys(1939), Cox(1946), Polya(1946), Jaynes(1957) Yes! In 1946, the physicist Richard Cox showed that inductive reasoning follows rules that are isomorphic to those of probability theory

November 21, 2002First ICFA Instrumentation School/Workshop Harrison B. Prosper Boolean Algebra The Rules of Inductive Reasoning: Boolean Algebra + Probability

November 21, 2002First ICFA Instrumentation School/Workshop Harrison B. Prosper12 Probability B A AB Conditional Probability A theorem

November 21, 2002First ICFA Instrumentation School/Workshop Harrison B. Prosper These rules together with Boolean algebra Bayesian Probability Theory are the foundation of Bayesian Probability Theory Product Rule Sum Rule Probability - ii Bayes’ Theorem

November 21, 2002First ICFA Instrumentation School/Workshop Harrison B. Prosper Bayes’ Theorem marginalization We can sum over propositions that are of no interest

November 21, 2002First ICFA Instrumentation School/Workshop Harrison B. Prosper15 Bayes’ Theorem: Example 1 Signal/Background Discrimination Signal/Background Discrimination –S = Signal –B = Background The probability P(S|Data), of an event being a signal given some event Data, can be approximated in several ways, for example, with a feed-forward neural network The probability P(S|Data), of an event being a signal given some event Data, can be approximated in several ways, for example, with a feed-forward neural network

November 21, 2002First ICFA Instrumentation School/Workshop Harrison B. Prosper posteriorprior marginalization likelihood I is the prior information Bayes’ Theorem: Continuous Propositions

November 21, 2002First ICFA Instrumentation School/Workshop Harrison B. Prosper17 Bayes’ Theorem: Model Comparison Standard Model SUSY Models M For each model M, we integrate over the parameters of the theory. M P(M|x,I) is the probability of M model M given data x.

November 21, 2002First ICFA Instrumentation School/Workshop Harrison B. Prosper Variance BayesianConfidenceInterval where Bayesian measures of uncertainty Also known as a Credible Interval

November 21, 2002First ICFA Instrumentation School/Workshop Harrison B. Prosper19 Example 1: Counting Experiment Experiment: Experiment: θ –To measure the mean rate θ of UHECRs above eV per unit solid angle. Assume the probability of N events to be a given by a Poisson distribution Assume the probability of N events to be a given by a Poisson distribution

November 21, 2002First ICFA Instrumentation School/Workshop Harrison B. Prosper20 Example 1: Counting Experiment - ii Apply Bayes’ Theorem Apply Bayes’ Theorem What should we take for the prior probability P(θ|I)? What should we take for the prior probability P(θ|I)? How about How about –P(θ|I) = dθ?

November 21, 2002First ICFA Instrumentation School/Workshop Harrison B. Prosper21 But why choose P(θ|I) = dθ? Why not P(θ 2 |I) = dθ? Why not P(θ 2 |I) = dθ? Or P(tan(θ)|I) = dθ? Or P(tan(θ)|I) = dθ? Choosing a prior probability in the absence of useful prior information is a difficult and controversial problem. Choosing a prior probability in the absence of useful prior information is a difficult and controversial problem. –The difficulty is to determine the variable in terms of which the prior probability density is constant. –In practice one considers a class of prior probabilities and uses it to assess the sensitivity of the results to the choice of prior.

November 21, 2002First ICFA Instrumentation School/Workshop Harrison B. Prosper22 Credible Intervals With P(θ|I) = dθ θ Bayesian N±√N

November 21, 2002First ICFA Instrumentation School/Workshop Harrison B. Prosper23 Credible Interval Widths Bayesian N±√N

November 21, 2002First ICFA Instrumentation School/Workshop Harrison B. Prosper24 Coverage Probabilities: Credible Intervals θ Bayesian Even though these are Bayesian intervals nothing prevents us from studying their frequency behavior with respect to some ensemble!

November 21, 2002First ICFA Instrumentation School/Workshop Harrison B. Prosper Model Data Likelihood l is the efficiency times branching fraction times integrated luminosity Prior information is the mean background count estimates are estimates Bayes’ Theorem: Measuring a Cross-section

November 21, 2002First ICFA Instrumentation School/Workshop Harrison B. Prosper Apply Bayes’ Theorem: nuisance parameters Then marginalize: that is, integrate over nuisance parameters priorlikelihood posterior Measuring a Cross-section - ii

November 21, 2002First ICFA Instrumentation School/Workshop Harrison B. Prosper27 What Prior? We can usually write down something sensible for the luminosity and background priors. But, again, what to write for the cross-section prior is controversial. In DØ and CDF, P(σ|I) = d σ as a matter of convention, we set P(σ|I) = d σ Rule: Always report the prior you have used!

November 21, 2002First ICFA Instrumentation School/Workshop Harrison B. Prosper28 Summary Confidence levels are probabilities; as such, their interpretation depends on the interpretation of probability adopted Confidence levels are probabilities; as such, their interpretation depends on the interpretation of probability adopted If one adopts a frequency interpretation the confidence level is tied to the set of ensembles one uses and is a property of that set. Typically, however, these ensembles do not objectively exist. If one adopts a frequency interpretation the confidence level is tied to the set of ensembles one uses and is a property of that set. Typically, however, these ensembles do not objectively exist. If one adopts a degree of belief interpretation the confidence level is a property of the interval calculated. A set of ensembles is not required for interpretation. However, the results necessarily depend on the choice of prior. If one adopts a degree of belief interpretation the confidence level is a property of the interval calculated. A set of ensembles is not required for interpretation. However, the results necessarily depend on the choice of prior.