1. 1 Chapter 10 Methods for Eliciting Probabilities

2. 2 Preparing for probability assessment Motivating Structuring Conditioning

3. 3 Motivating This phase is designed to introduce the decision maker to the task of assessing probabilities and to explain the importance and purpose of the task. Sensitivity analysis should be used by the analyst to identify those probabilities which need to be assessed with precision.

4. 4 Structuring In the structuring phase the quantity to be assessed should be clearly defined and structured. It is also important at this stage to agree on a scale of measurement which the decision maker feels comfortable with. When the decision maker thinks that the quantity to be assessed depends on other factors it may be simpler to restructure the assessment task, possibly by making use of a probability tree.

5. 5 Conditioning The objective of this phase is to identify and thereby avoid the biases which might otherwise distort the decision maker's probability assessments. It involves an exploration of how the decision maker approaches the task of judging probabilities. For example, are last year's sales figures being used as a basis for this year's estimates? If they are, there may be an anchoring effect.

6. 6 Assessment methods Assessment methods for individual probabilities Assessment methods for probability distributions

7. 7 Direct assessment methods Posing a direct question e.g. ‘ What is the probability that the product will achieve break-even sales next month? ’

8. 8 Unfortunately, many people would feel uncomfortable with this sort of approach, and they might be tempted to give a response without sufficient thought. Asking the individual to mark a point on a scale which runs from 0 to 1 might be preferred because at least the scale enables the probability to be envisaged.

9. 9 The probability wheel A probability wheel is a device like that shown in Figure 10.1, and it consists of a disk with two different colored sectors, whose size can be adjusted, and a fixed pointer.

10. 10 The probability wheel

11. 11 Bet One: If the rival launches the product within the next week you will win $100000. If the rival does not launch the product you will win nothing. Bet Two: If, after spinning the wheel once, the pointer is in the white sector you will win $100 000. If it is pointing toward the black sector you will win nothing.

12. 12 If the manager says that she would choose Bet Two then this implies that she thinks that the probability of the rival launching the product in the next week is less than 80%. The size of the white sector could then be reduced and the question posed again. Eventually, the manager should reach a point where she is indifferent between the two bets. If this is achieved when the white sector takes up 30% of the wheel's area, this clearly implies that she estimates that the required probability is 0.3.

13. 13 The wheel has the advantage that it enables the decision maker to visualize the chance of an event occurring. However, because it is difficult to differentiate between the sizes of small sectors, the probability wheel is not recommended for the assessment of events which have either a very low or very high probability of occurrence.

14. 14 The analyst should also ensure that the rewards of the two bets are regarded as being equivalent by the decision maker. A number of devices similar to the probability wheel have also been used in probability assessment. For example, the decision maker may be asked to imagine an urn filled with 1000 colored balls (400 red and 600 blue).

15. 15 Assessment methods for probability distributions The probability method There is evidence that, when assessing probability distributions, individuals tend to be overconfident, so that they quote too narrow a range within which they think the uncertain quantity will lie. The following procedure is recommended.

16. 16 Step 1: Establish the range of values within which the decision maker thinks that the uncertain quantity will lie. Step 2: Ask the decision maker to imagine scenarios that could lead to the true value lying outside the range. Step 3: Revise the range in the light of the responses in Step 2. Step 4: Divide the range into six or seven roughly equal intervals.

17. 17 Step 5: Ask the decision maker for the cumulative probability at each interval. This can either be a cumulative 'less than' distribution (e.g. what is the probability that the uncertain quantity will fall below each of these values?) or a cumulative 'greater than' (e.g. what is the probability that the uncertain quantity will exceed each of these values?), depending on which approach is easiest for the decision maker.

18. 18 Step 6: Fit a curve, by hand, through the assessed points. Step 7: Carry out checks as follows. (i) Split the possible range into three equally likely intervals and find out if the decision maker would be equally happy to place a bet on the uncertain quantity falling in each interval. If he is not, then make appropriate revisions to the distribution. (ii) Check the modality of the elicited distribution (a mode is a value where the probability distribution has a peak). For example, if the elicited probability distribution has a single mode (this can usually be recognized by examining the cumulative curve and seeing if it has a single inflection), ask the decision maker if he does have a single best guess as to the value the uncertain quantity will assume. Again revise the distribution, if necessary.

19. 19 Assessment methods for probability distributions The probability method Graph drawing …

20. 20 Graph drawing Graphs can be used in a number of ways to elicit probability distributions. The method of relative heights is one well- known graphical technique that is designed to elicit a probability density function.

21. 21 First, the decision maker is asked to identify the most likely value of the variable under consideration and a vertical line is drawn on a graph to represent this likelihood. Shorter lines are then drawn for other possible values to show how their likelihoods compare with that of the most likely value.

22. 22 The method of relative heights

23. 23 To convert the line lengths to probabilities they need to be normalized so that they sum to one. This can be achieved by dividing the length of each line by the sum of the line lengths, which is 36, as shown below

24. 24

25. 25 Assessing the validity of subjective probabilities

26. 26 Consistency checks are, of course, a crucial element of probability assessment. The use of different assessment methods will often reveal inconsistencies that can then be fed back to the decision maker.

27. 27 If probability estimates derived by different methods for the same event are inconsistent, which method should be taken as the true index of degree of belief? One way to answer this question is to use a single method of assessing subjective probability that is most consistent with itself. (+)

28. 28 In other words, there should be high agreement between the subjective probabilities, assessed at different times by a single assessor for the same event, given that the assessor's knowledge of the event is unchanged. It was concluded that most of the subjects in all experiments were very consistent when using a single assessment method.

29. 29 One useful coherence check is to elicit from the decision maker not only the probability that an event will occur but also the probability that it will not occur. The two probabilities should, of course, sum to one. If the events are seen by the probability assessor as mutually exclusive then the addition rule can be applied to evaluate the coherence of the assessments.

30. 30 A major measure of the validity of subjective probability forecasts is known as calibration. By calibration we mean the extent to which the assessed probability is equivalent to proportion correct over a number of assessments of equal probability.

31. 31 Assessing probabilities for very rare events Assessment techniques that differ from those we have so far discussed are generally required when probabilities for very rare events have to be assessed. Because of the rarity of such events, there is usually little or no reliable past data which can support a relative frequency approach to the probability assessment.

32. 32 Decision makers are also likely to have problems in conceiving the magnitudes involved in the probability assessment. It is difficult to distinguish between probabilities such as 0.0001 and 0.000001, yet the first probability is a hundred times greater than the second.

33. 33 Event trees and fault trees allow the problem to be decomposed so that the combinations of factors which may cause the rare event to occur can be identified. Each of the individual factors may have a relatively high (and therefore more easily assessed) probability of occurrence. A log-odds scale allows the individual to discriminate more clearly between very low probabilities.

34. 34 Figure 10.4 shows a simplified tree for a catastrophic failure at an industrial plant. Each stage of the tree represents a factor which might, in combination with others, lead to the catastrophe. By using the multiplication and addition rules of probability, the overall probability of failure can be calculated.

35. 35 An event tree

36. 36 Fault trees In contrast to event trees, fault trees start with the failure or accident and then depict the possible causes of that failure.

37. 37 A fault tree

38. 38 Using a log-odds scale