Using imprecise estimates for weights Alan Jessop Durham Business School

Motivation In a weighted value function model weights are inferred from judgements. Judgements are imprecise and so, therefore, weight estimates must be imprecise. Probabilistic weight estimates enable the usual inferential methods, such as confidence intervals, to be used to decide whether weights or alternatives may justifiably be differentiated.

Testing & sensitivity Single parameter. Results easily shown and understood. But partial. Multi parameter. Examine pairs (or more) to get a feel for interaction. Global (eg. Monte Carlo). Comprehensive but results may be hard to show simply. Using some familiar methods uncertainty can be inferred from judgements and the effects of global imprecision can be shown. An analytical approach rather than a simulation.

Sources of imprecision Statements made by the judge are inexact. This is imprecise articulation: variance = σ a ² The same judgements may be made in different circumstances, using different methods or at different times, for instance. This is circumstantial imprecision: variance = σ c ²

articulation single pointthree-point degrees of freedom implicit in questions 0none available a2a2 >0 c2c2  c 2 +  a 2 Sources of imprecision Redundancy e.g. ask at different times reciprocal matrix No redundancy e.g. simple rating related?

3 point estimate:  a 2 μ = aM + (1-a)(L+H)/2 σ a = b(H-L) Previous studies for PERT analyses. Generalise as a = 1.800x c b = c Beta distribution But because  w = 1 variances will be inconsistent. Solution: fit a Dirichlet distribution.

Dirichlet f(W) = k  i w i u i -1 ; 0<w i <1,  i w i = 1, u i  0,  i wherek =  (  i u i ) /  i  (u i ) which has Beta marginal distributions with mean  i = u i / v variance  i ² = u i (v- u i ) / v²(v+1) =  i (1-  i ) / (v+1) and covariance  ij = -u i u j / v²(v+1) = -  i  j / (v+1) ; i≠j wherev =  i u i Relative values of parameters u i determine means. Absolute values determine variance via their sum, v.

Dirichletset parametersweight values Dirichlet judgements: marginal characteristics mean e i and variance s i 2 consistent variances  i ² Usually used by specifying parameters (eg in Monte Carlo simulation) But can also be used to ensure compatibility: Put  i = e i Then get least squares best fit to minimise S =  (  i ² - s i 2 ) 2  S/  v = 0 → v+1 =  [e i (1- e i )] 2 /  e i (1- e i )s i 2 so  i ² = e i (1-e i ) / (v+1) ( NOTE: only have to know mean values and v ) Sum over available estimates s i ² so can tolerate missing values

Experiment: FT full-time MBA ranking 7 variables used in experiment

Experiment: 3 point estimate →  a 2 3 point judgementscaled from which, mean and standard deviation Dirichlet consistent missing value tolerated

Summarising discrimination between programmes y =  i w i x i var(y) =  i  j σ ij x i x j = [  i w i (1-w i )x i ² - 2  i  j w i w j x i x j ] / (v+1) j>i For two alternatives replace x values with differences (x a -x b )

Northern Europe: UK, France, Belgium, Netherlands, Ireland

Summarising discrimination between programmes Summaries (v+1) = Proportion of all pairwise differences significantly different at p = 0.1: discrimination = 81%

Lines show indistinguishable pairs; p = 10%

Lines show indistinguishable pairs; p = 1%

Lines show indistinguishable pairs; p = 25%

Weights. discrimination = 71%

Weights; p = 10%

Different circumstances →  c 2. Reciprocal matrix.

 a 2 and  c 2. Reciprocal matrix. Give each judgement in a reciprocal matrix as a 3-point evaluation. Then treat each column as a separate 3-point evaluation and find Dirichlet compatible  a 2 as before. For each weight the mean of these variances is the value of  a 2 as in aggregating expert judgements (Clemen & Winkler, 2007). The mean of the column means is the weight and the variance of the means is  c 2.

Results from 10 MBA students standard deviations σ = [  a 2 +  c 2 ] ½

Are the two sources of uncertainty related? mean r = 0.70 taken together r = 0.33 consistently σ c > σ a

Student G is representative

Scores. (v+1) = discrimination = 30%

Lines show indistinguishable pairs; p = 10%

Lines show indistinguishable pairs; p = 50% decide that 1 & 5 can be distinguished

A possible form of interaction Assume that new discrimination is due to increased precision rather than difference in scores: statistical significance rather than material significance. So, change precision by changing (v+1) and leave weights unaltered. z is directly proportional to √(v+1). In this case (v+1) = 8.54 → z 1,5 = 0.55 p = 50% → z* = 0.67 (v+1) new = (z*/z 1,5 )² × (v+1) = (0.67 / 0.55)² × 8.54 = and so...

Weights. discrimination = 14%

Group aggregation Do this for all ten assessors.

Scores. (v+1) = discrimination = 16%

Lines show indistinguishable pairs; p = 10%

Lines show indistinguishable pairs; p = 50%

Weights. discrimination = 0%

Tentative conclusions Even though results are imprecise there may still exist enough discrimination to be useful, as in forming a short list. May give ordering of clusters. Makes explicit what may justifiably be discriminated. Choosing confidence levels and significance values is, as ever, sensible but arbitrary. Explore different values. Once a short list is identified, further analysis needed, probably using some form of what-if interaction to see the effect of greater precision. Variation between circumstances seems to be consistently greater than self- assessed uncertainty. Does this matter? Do we want to justify one decision now or address circumstantial (temporal?) variation?

end