1. 1 Dipak K. Dey University of Connecticut Some parts joint with: Junfeng Liu Case Western Reserve University Prior Elicitation from Expert Opinion

2. 2 Elicitation Elicitation is the process of extracting expert knowledge about some unknown quantity of interest, or the probability of some future event, which can then be used to supplement any numerical data that we may have. If the expert in question does not have a statistical background, as is often the case, translating their beliefs into a statistical form suitable for use in our analyses can be a challenging task.

3. 3 Introduction Prior elicitation is an important and yet under researched component of Bayesian statistics. In any statistical analysis there will typically be some form of background knowledge available in addition to the data at hand. For example, suppose we are investigating the average lifetime of a component. We can do tests on a sample of components to learn about their average lifetime, but the designer/ engineer of the component may have their own expectations about its performance.

4. 4 Introduction If we can represent the expert's uncertainty about the lifetime through a probability distribution, then this additional (prior) knowledge can be utilized within the Bayesian framework. With a large quantity of data, prior knowledge tends to have less of an effect on final inferences. Given this fact, and the various techniques available for representing prior ignorance, practitioners of Bayesian statistics are frequently spared the effort of thinking about the available prior knowledge.

5. 5 Introduction It will not always be the case that we will have sufficient data to be able to ignore prior knowledge, and one example of this would be in the uncertainty in computer models application or modeling extreme events. Uncertain model input parameters are often assigned probability distributions entirely on the basis of expert judgments. In addition, certain parameters in statistical models can be hard to make inferences about, even with a reasonable amount of data.

6. 6 Introduction The amount of research in eliciting prior knowledge is relatively low, and various proposed techniques are often targeted at specific applications. At the same time, recent advances in Bayesian computation have allowed far greater flexibility in modeling prior knowledge. In general, elicitation can be made difficult by the fact that we cannot expect the expert to provide probability distributions for quantities of interest directly.

7. 7 Introduction The challenge is then to find appropriate questions to ask the expert in order to extract their knowledge, and then to determine a suitable probabilistic description of the variables we are interested in based on the information we have learned from them.

8. 8 Motivation Three approaches : [1] Direct Prior Elicitation: Berger (1985) Relative frequency, and quantile based elicitation. [2] Predictive prior probability space, which requires simple priors and may be burdened with additional uncertainties arising from the response model. (Kadane, et al, 1980; Garthwaite and Dickey, 1988, Al-Awadhi and Garthwaite, 1998, etc.). [3] Nonparametric Elicitation: (Oakley and OHagan, 2002)

9. 9 Symmetric Prior Elicitation Double bisection method: Expert provides q(.25), q(.5) and q(.75), the three quantiles IQR = q(.75)-q(.25) Normal prior: Z(q)= IQR of std. normal, then, prior mean and std. dev. are, q(.5) and IQR/ Z(q) respectively.

10. 10 Students t Prior Three non redundant quantiles are required to estimate the df ν. Kadane et.al. (1980) suggested obtaining q(.5), q(.75) and q(.9375) a(x) = (q(.9375)-q(.5))/(q(.75)-q(.5)) depends on df ν only Df is determined from look up table of a(x) vs df ν.

11. 11 Students t Prior After elicitation of df obtain t ν,0.75 Calculate S(q) = (q(.75)-q(.5)) 2 / t 2 ν,0.75 for elicitation of scale parameter σ. This idea can be applied to any general location-scale family.

12. 12 Lognormal Prior Garthwaite (1989) used split-normal distribution, OHagan (1998) used 1/6, 3/6 and 5/6 quantiles. Proposition: If X has a log-normal distribution, i.e.,, then the variance and the mean,where is the median of is the IQR for standard normal distribution.

13. 13 Direct Prior Elicitation (1)Simple and limited prior family with only location and scale parameters (normal, exponential, etc.) (2) Location-scale-shape (µ- - ) parameter joint elicitation (gamma, skew-normal, Students t, etc.)

14. 14 Normal Students t Log-normal Skew-normal Normal-exponential Skew-Students t Symmetric and Asymmetric Priors Location-scale, symmetric No location scale but shape, symmetric Location-scale, asymmetric Location-scale-shape, asymmetric

15. 15 Shape Parameter Elicitation This is most challenging. This is most challenging. Presumably, the Interquantile-Range-ratio (IQRR= [q(.75)- q(.5)]/[q(.5)-q(.25)] is a monotone function of shape parameter. We have two cases: (1) Shape-parameter is in the non-sensitive region, absolute value larger than 1. (2) Shape-parameter is in the sensitive region, absolute value smaller than 1.

16. 16 Nonsensitive and sensitive regions (Skew-normal) IQRR (interquantile range ratio) vs. shape parameter Non-sensitive Sensitive

17. 17 Shape Parameter Sensitive Region: Gamma Case

18. 18 Parameter Elicitation Guideline: We prefer a moderate sensitivity index (SI): Hyperparameter change / elicitation input change SI= (IQRR)/ (l) We look for SI close to 1. Sensitive region: shape parameter is small in magnitude. The elicitation input is IQRR and the hyperparameter is the shape parameter.

19. 19 Parameter Elicitation on Shape Parameter Non- Sensitive Region (1) Elicit shape parameter from plot of IQRR( ) vs. (2) Scale parameter = IQR/IQR( ) where, IQR is the interquantile range from expert, IQR( ) is the standardized IQR with elicited from (1), =1 and µ=0. (3) The location parameter is Q(0.75)- Q(0.75, ) where, Q(0.75) is.75 quantile from expert, comes from (2), and Q(0.75, ) is the standardized.75 quantile with elicited from (1), =1 and µ=0.

20. 20 The sensitivity index in IQR( ) vs. and Q(0.75, ) vs. is usually moderate. Note:

21. 21 Approximate Scale Parameter Elicitation from Taylors Expansion (1: Basics) [1] g(*) is the characteristic point of density f(x|µ,, ), say mean, median, mode, etc. [2] g(*) = µ+ g( ), where g( ) is the standardized characteristic point. [3] f(g(*)|µ,, ) = (1/ )f(g( )|0,1, ). General approach for any location, scale and shape Family:

22. 22 Approximate Scale Parameter Elicitation from Taylors Expansion (2: Method) Letting (1)-(2) and only keeping first 2 terms on the right hand side, we get We get the approximate scale parameter without considering any consequences as

23. 23 Relative Error in Students t Prior Elicitation (1: Values) From Taylors expansion, we have approximate The exact Where, [1] v is degrees of freedom [2] IQR is interquantile range from expert [3] p = 0.5 [4] is.75 quantile of Students t distribution with v degrees of freedom

24. 24 Approximate Scale Parameter Elicitation from Taylors Expansion (3: Relative Error) Now (1)-(2) Denote (Only related to ) The relative error is

25. 25 Relative Error in Students t Prior Elicitation (2: Plot) (1) ``approximate represents Taylor expansion value: (2) ``exact represents Taylor expansion value: (3) ``normal represents, with as interquantile range for standardized normal distribution. (1) : (2) approaches 1.0763 as v goes to infinity.

26. 26 An Important Observation When shape parameter is highly sensitive to IQRR, the approximate scale parameter elicitation by Taylors expansion will be very stable in terms of relative error.

27. 27 Elicitation of Shape Parameter on Sensitive Region (Skew-normal, Iteration on characteristic points) Iteration based on Taylors expansion at median, mode or mean. (1) Start with current l, from high-proportional- fidelity by Taylor expansion, we have (2) The skew(shape) parameter can be obtained by plotting (3) Go to (1) until convergence (complete and ) (4) Location parameter

28. 28 Elicitation on Shape Parameter Sensitive Region (Skew-normal, Iteration on IQRs) Iteration based on IQRs (1) Start with current, we look up, then (2) The skew (shape) parameter can be obtained by plot (3) Go to (1) until convergence (complete and ) (4) Location parameter Since

29. 29 Graphical Comparison 1 ( reference: IQR based iteration )

30. 30 Graphical Comparison 2 ( reference: median based iteration )

31. 31 Graphical Comparison 3 ( reference: mean based iteration )

32. 32 Graphical Comparison 4 ( reference: mode based iteration )

33. 33 Another Important Observation The IQR based iteration is close to mean based iteration for skew-normal case, since mean is explicit, other than numerically solved.

34. 34

35. 35

36. 36 Non-Parametric Prior Elicitation To estimate prior density directly such that, Suppose, where = parametric family of distributions, = vector of hyper parameters = underlying parameters in

37. 37 Non-Parametric Prior Elicitation =(correlation function) = 1 if decreasing function of ensures that prior variance covariance matrix of any set of observation otherwise. or functional of is positive semi-definite.

38. 38 Choice of Covariance function specifies the true density function. controls smoothness of the density. b large impliesis large.

39. 39 Hierarchical prior (Gaussian Process Prior) Special Case : then Then Prior:

40. 40 Let D = elicited summaries relating to = {data} H is a function of A and is a function of

41. 41 This implies, with

42. 42 Posterior n = # of elements in D use MCMC to obtain samples from

43. 43 Other Choices of Centering a) b) c) d) Gamma or Log-normal etc.

44. 44 Side Conditions Given Derivatives or quantiles D will be appropriately changed. In fact D can incorporate all the constraints specified in the prior, e.g., moments.

45. 45 Psychological Perspective of Imprecise Subjective Probabilities Numerical probabilty estimates (N) Ranges of numerical values (R) Verbal phrases (V) Objective: Translate the triplate (N,R,V) to a decision makers model

46. 46 Imprecisely Assessed Distributions Contamination: Class of Bi-modal distribution

47. 47 Future problems Prior elicitation in Extreme Value Modeling Quantile and graphical approaches for GEV model, Coles and Powel(1996) Prior elicitation for short and long tailed distribution Spatial modeling High dimensional modeling

48. 48 References 1. Daneshkhah, A. (2004). Psychological Aspects Influencing Elicitation of Subjective Probability. BEEP working paper. 2. Dey, D.K. and Liu, J. (2007). A quantitative study of quantile based direct prior elicitation from expert opinion. Bayesian Analysis, 2, 137-166. 3. Garthwaite, P. H., Kadane, J. B., and O'Hagan, A. (2005). Statistical methods for eliciting probability distributions. Journal of the American Statistical Association, 100, 680-701. 4. Jenkinson, D. (2005). The Elicitation of Probabilities-A Review of the Statistical Literature. BEEP working paper. 5. Kadane, J.B.,Dickey,J.M., Winkler, R.L., Smith, W.S. and Peters, S.C.(1980). Interactive elicitation of opinion for a normal linear model. JASA, 75, 845-854.

49. 49 6. Oakley, J., and O'Hagan, A. (2005). Uncertainty in prior elicitations: a non-parametric approach. Revised version of research report No. 521/02 Department of Probability and Statistics, University of Sheffield. 7. O'Hagan, A. (2005). Research in elicitation. Research Report No.557/05, Department of Probability and Statistics, University of Sheffield. Invited article for a volume entitled Bayesian Statistics and its Applications. 8. O' Hagan, A., Buck, C. E., Daneshkhah, A., Eiser, J. E., Garthwaite, P. H., Jenkinson, D. J., Oakley, J. E. and Rakow, T. (2006). Uncertain Judgements: Eliciting Expert Probabilities. This book Will be published by John Wiley and Sons in July 2006.

50. 50 THANK YOU