1. Page 1 Charles Plager LJ+MET, March 23, 2009 Charles Plager UCLA LJ+MET Meeting March 23, 2008 “Throwing PEs” and More

2. Page 2 Charles Plager LJ+MET, March 23, 2009 “PE? What’s a PE?” Pseudo-experiment (PE) is the process of generating a set of random numbers that simulates the given quantities needed to perform an analysis.Pseudo-experiment (PE) is the process of generating a set of random numbers that simulates the given quantities needed to perform an analysis. –This can be as simple of generating a number of events for a counting experiment. –This can be as complicated as picking a subset of “events” needed to perform correlated analyses. In Charles-speak:In Charles-speak: –PE ) One pseudo-experiment. –PEs ) Many pseudo-experiments. –Throw PEs ) Generate many pseudo-experiments “Why throw PEs?”“Why throw PEs?” –Throwing and “fitting” PEs is a very useful way of: Debugging an analysisDebugging an analysis Checking correlations between analysesChecking correlations between analyses Measuring sensitivity of a given analysisMeasuring sensitivity of a given analysis

3. Page 3 Charles Plager LJ+MET, March 23, 2009 Simple Counting Experiment PE –You expect N back background events and N sig signal events. –For each PE, your number of observed events: nObs= random.Poisson (N back ) + random.Poisson (N sig ) = random.Poisson (N back + N sig ) Your analysis is a simple counting experiment with no systematic uncertainties.Your analysis is a simple counting experiment with no systematic uncertainties.

4. Page 4 Charles Plager LJ+MET, March 23, 2009 PEs from Templates Same idea as “Simple Counting PEs,” except that we can’t merge signal and background:Same idea as “Simple Counting PEs,” except that we can’t merge signal and background: –N obsSig = random.Poisson (N sig ) –N obsBack = random.Poisson (N back ) For each signal event, a value is drawn from the templateFor each signal event, a value is drawn from the template Fit the resulting PE as if it were data.Fit the resulting PE as if it were data.

5. Page 5 Charles Plager LJ+MET, March 23, 2009 Aside: PEs with a Fixed Number of Events It may sometimes happen that one wants to generate PEs with a fixed number of events, N total.It may sometimes happen that one wants to generate PEs with a fixed number of events, N total. –E.g., one wants to match the total number of events in a particular data sample. If you are assuming, for example, a small background N back and a large signal N sig, you can not:If you are assuming, for example, a small background N back and a large signal N sig, you can not: –Poisson fluctuate background N obsBack = Poisson (N back ), and then –N obsSig = N total – N back You need to fluctuate both:You need to fluctuate both: –N obsBack = Poisson (N back ), –N obsSig = Poisson (N sig ), –Keep only if N obsBack + N obsSig == N total Note: If you have N obsBack + N obsSig > N total,Note: If you have N obsBack + N obsSig > N total, –Pick random event to get rid of (signal or background) –Repeat until N obsBack + N obsSig == N total

6. Page 6 Charles Plager LJ+MET, March 23, 2009 PEs with Systematic Uncertainties “Blur, then throw”“Blur, then throw” You have a whole bunch of estimates described by Gaussian uncertainties: x §  x.You have a whole bunch of estimates described by Gaussian uncertainties: x §  x. For each PE, generate a value for each estimate from the correct Gaussian distribution.For each PE, generate a value for each estimate from the correct Gaussian distribution. Make sure all estimates are physical. If not, repeat previous step *.Make sure all estimates are physical. If not, repeat previous step *. Given this set of estimates, generate PE as we have done in previous steps.Given this set of estimates, generate PE as we have done in previous steps.

7. Page 7 Charles Plager LJ+MET, March 23, 2009 Correlated PEs “We have two analyses that use overlapping data samples. What do we do?”“We have two analyses that use overlapping data samples. What do we do?” Instead of picking values from a template, we pick whole “events” from MC.Instead of picking values from a template, we pick whole “events” from MC. –All analyses then take what they need from each event. Using weighted MC is more complicated here:Using weighted MC is more complicated here: –Pick events –Generate random number to see if event is kept –Repeat until total number of events is reached in sub-sample –Important: Reweighting is very powerful technique and throwing PEs is a solved problem (see me).

8. Page 8 Charles Plager LJ+MET, March 23, 2009 Your Analysis Checkup I’ve now thrown many PEs. What do I look at?I’ve now thrown many PEs. What do I look at? For each PE, take the results and treat it like you would with data:For each PE, take the results and treat it like you would with data: –Fit, etc. Look at different distributions:Look at different distributions: –Uncertainties ) Sensitivity –Pull distributions (more later) Looking for coverage, bias, etc.Looking for coverage, bias, etc. Hint: Store PE results in a TTreeHint: Store PE results in a TTree Easier to debug problems.Easier to debug problems.

9. Page 9 Charles Plager LJ+MET, March 23, 2009 Pull Distributions A pull distribution is a very useful way to make sure your analysis machinery is doing what it is supposed to.A pull distribution is a very useful way to make sure your analysis machinery is doing what it is supposed to. –Checks for biases as well as under and over coverage. –For each event, calculate: –Resulting distribution should be a unit Gaussian.

10. Page 10 Charles Plager LJ+MET, March 23, 2009 More with Pull Distributions If you measure asymmetric uncertainties, simply use the “right one:”If you measure asymmetric uncertainties, simply use the “right one:” –  upper : measurement < true value –  lower : measurement > true value If the parameter of interest has a Gaussian constraint in your fit:If the parameter of interest has a Gaussian constraint in your fit: –If you have Gaussian constraints in your fit, it is important to throw the PEs using the same distribution. ) Throw Consistently. Remember: checking pull distribution is aRemember: checking pull distribution is a necessary, but not sufficient check. MeasLowerUpper True

11. Page 11 Charles Plager LJ+MET, March 23, 2009 Hints Small statistics can sometimes “mess” with pull distributions:Small statistics can sometimes “mess” with pull distributions: –If you aren’t already, try varying the true value of the variable when throwing the PE. –E.g., When throwing pull distributions for top pair cross sections, vary the theoretical cross section using the theory errors. Root lets you generate random numbers from a histogram:Root lets you generate random numbers from a histogram: ) value = myHistPtr->Random() ) value = myHistPtr->Random()

12. Page 12 Charles Plager LJ+MET, March 23, 2009 Root and Random Number Generators For all modern versions of Root (≥ 5.18), gRandom is an instance of TRandom3.For all modern versions of Root (≥ 5.18), gRandom is an instance of TRandom3. –This is a decent random number generator. If you are using an older version of Root, this may not be the default.If you are using an older version of Root, this may not be the default. –Change it: delete gRandom; gRandom = new TRandom3; If you want to run PE jobs in parallel,If you want to run PE jobs in parallel, make sure you set the jobs to have different random seeds: –gRandom->SetSeed (seed); where seed is an integer not equal to 0. If you don’t set different seeds, yourIf you don’t set different seeds, your different jobs will not generate different PEs. Hardware Random Number Generator

13. Page 13 Charles Plager LJ+MET, March 23, 2009 Warning: The concept of ensembles (e.g., PEs) is the heart of frequentist statistical thought.The concept of ensembles (e.g., PEs) is the heart of frequentist statistical thought. –When I talk about how to deal with systematic uncertainty, I will be effectively talking about integrating over systematic priors. ) Bayesian statistics. Most of this will be a (reasonable) mix of the two.Most of this will be a (reasonable) mix of the two. –These methods will be sufficient for almost all of your needs, but: –These methods, however, will not always lead to proper frequentist coverage. ?

14. Page 14 Charles Plager LJ+MET, March 23, 2009 Summary Throwing PEs is fun, easy, and important.Throwing PEs is fun, easy, and important. –Weights and reweighting is cool and powerful. Pull distributions should be standard operating procedure.Pull distributions should be standard operating procedure. Using these simple ideas can really pay off in the end!Using these simple ideas can really pay off in the end!