Max Baak (CERN) 1 Summary of experiences with HistFactory and RooStats Max Baak (CERN) (on behalf of list of people) RooFit / RooStats meeting January 19 th 2012

Max Baak (CERN) 2 Overview of this talk  In past months, have actively started using HistFactory and RooStats code for model/workspace creation and limit setting. In context of Atlas SUSY group Previously, had own RooFit-based code for doing this.  This talk: some feedback on experiences with using this code.  Suggestions in several areas for possible additions and improvements to the code. Additions indicated in Red.  (Some of these suggestions I may already have told or mailed in private.)

Max Baak (CERN) 3 Our ‘HistFitter’ package  Big fan of HistFactory idea: very transparent way of creating one (complex) fit model. Easy for non-RooFit-experts to get started.  Our physics situation: need to deal with a (2D) grid of (O(100)) signal models to test.  ‘HistFitter’ package: built around HistFactory code, with purpose of creating/managing/testing many workspaces in flexible way. Used by group of people, not necessarily RooFit (= model-building) or (= limit-setting) RooStats experts. Possibly useful code for adding to RooStats:  Several classes to program xml input files for HistFactory, through python.  Machinery for bookkeeping many workspaces (one workspace for each signal model) and HypoTest results. Examples follow.

Max Baak (CERN) 4 Example signal model grid  Signal model parameters on x and y axes.  Every point is a grid point with certain (mass) parameter values.

Max Baak (CERN) 5 HypoTestTool  Though many examples available, setting up RooStats tools for limit setting not entirely straight-forward for newcomer.  Suggest that one-line wrapper functions are added with reasonable default settings for doing. This is how my non-RooStats-expert colleagues tend to use the code.  Eg. Hypothesis test at mu=0 or 1 (CL S ) 95% CL upper limit Nice upper limit plot

Max Baak (CERN) 6 HypoTestTool  HypoTestTool: wrapper class to setup several RooStats HypoTestInverters  Class originally written by Gena. Taken from: $ROOTSYS/tutorials/roostats/StandardHypoTestInvDemo.C  Several modifications/additions to HypoTestTool ensure the class: Works stand-alone, Can do both hypothesis test and hypothesis-test inversion, … both for exclusion and discovery  Had to patch ProfileLikelihoodTestStat.cxx to be able to evaluate discovery test statistic (!). (I.e. do fit at both mu=0 and mu=free.) Consistent generation of both background-only and signal+background toys Reasonable default settings (e.g. for upper limit plots)  Usual set of configurable options still available …

Max Baak (CERN) 7 Wrapper functions calling HypoTestTool  Stand-alone one-line functions that should exist in RooStats Perhaps they do, but I’m simply not aware of them Ie. I wrote these, calling HypoTestTool …  Double get_Freq_CLs( RooWorkspace* w )  Double get_Asymp_CLs( RooWorkspace* w )  Double get_Freq_pValue( RooWorkspace* w )  HypoTestResult* DoHypoTest( RooWorkspace* w )  HypoTestInverterResult* DoHypoTestInversion( RooWorkspace* w ) I imagine this is how most non-experts will like to run RooStats...

Max Baak (CERN) 8 ToyMCSampler  Slightly black-box feeling to it. (For me at least.)  Would be great of ToyMCSampler class can return set of fit results for set of toys that are generated and fit.  … ala existing RooMCStudy functionality.  Vital for every analysis to understand toy MC behaviour. Especially if unconditional ensemble of toys is non-trivial to generate.  (E.g. what is happening with failed fits in case of limit setting?)  Wrote some simple bit of code to do this externally. (Slow.)  CERN has technical student in next months who can help implement this in ToyMCSampler …

Max Baak (CERN) 9 HypoTestResult  Some more info that could (imo should) be added to this class:  RooFitResult(s!) of evaluated data test statistic  CLs expected, CLs ±1 and ±2 sigma uncertainty values Right now I had to get those from HypoTestInverterResult, evaluated at just one mu value.  Fractions of failed toy fits.  Is it possible to add up 2 HypoTestResults, resulting in more precise summed HypoTestResult ? Have my own LimitResult class that contains / can do all this. Useful to add?

Max Baak (CERN) 10 Conversion HypoTestInverterResult  Eventually, need to turn grid of hypothesis tests in contour (exclusion) plot on right-hand side. Observed, expected, ±1 sigma uncertainty band contours …  Tool for converting set of HypoTestInverterResults into TTree, which can then be used for plotting purposes. Input file contains many hypotest results with specific name format.  TString outputfile = CollectAndWriteHypoTestResults( inputfile, “hypo_SU_%f_%f_0_10”, “m0:m12”, “m12>600” ) ;

Max Baak (CERN) 11 Minor fixes and annoyances  HistFactory: Gauss sigma settable (not a constant)  HistFactory: stat error pdf only included when rel. error > X  HistFactory: needs to have POI set – crashes otherwise.  HistFactory: always add luminosity pdf, even  To make PLL test statistic work in discovery mode: initial_mu > 0