1. Copyright © 2004 David M. Hassenzahl Monte Carlo Analysis David M. Hassenzahl

2. Copyright © 2004 David M. Hassenzahl Purpose of lecture Introduce Monte Carlo Analysis as a tool for managing uncertainty Demonstrate how it can be used in the policy setting Discuss its uses and shortcomings, and how they are relevant to policy making processes

3. Copyright © 2004 David M. Hassenzahl What is Monte Carlo Analysis? It is a tool for combining distributions, and thereby propagating more than just summary statistics It uses random number generation, rather than analytic calculations It is increasingly popular due to high speed personal computers

4. Copyright © 2004 David M. Hassenzahl Background/History “Monte Carlo” from the gambling town of the same name (no surprise) First applied in 1947 to model diffusion of neutrons through fissile materials Limited use because time consuming Much more common since late 80’s Too easy now? Name…is EPA “gambling” with people’s lives (anecdotal, but reasonable).

5. Copyright © 2004 David M. Hassenzahl Why Perform Monte Carlo Analysis? Combining distributions With more than two distributions, solving analytically is very difficult Simple calculations lose information – Mean  mean = mean –95% %ile  95%ile  95%ile! –Gets “worse” with 3 or more distributions

6. Copyright © 2004 David M. Hassenzahl Monte Carlo Analysis Takes an equation –example: Risk = probability  consequence Instead of simple numbers, draws randomly from defined distributions Multiplies the two, stores the answer Repeats this over and over and over… Then the set of results is displayed as a new, combined distribution

7. Copyright © 2004 David M. Hassenzahl Simple (hypothetical) example Skin cream additive is an irritant Many samples of cream provide information on concentration: –mean 0.02 mg chemical –standard dev. 0.005 mg chemical Two tests show probability of irritation given application –low freq of effect per mg exposure = 5/100/mg –high freq of effect per mg exposure = 10/100/mg

8. Copyright © 2004 David M. Hassenzahl Analytical results Risk = exposure  potency –Mean risk = 0.02 mg  0.075 / mg = 0.0015 or 15 out of 10,000 applications will result in irritation

9. Copyright © 2004 David M. Hassenzahl Analytical results “Conservative estimate” –Use upper 95 th %ile Risk = 0.03 mg  0.0975 / mg = 0.0029

10. Copyright © 2004 David M. Hassenzahl Monte Carlo: Visual example Exposure = normal(mean 0.02 mg, s.d. = 0.005 mg) potency = uniform (range 0.05 / mg to 0.10 / mg)

11. Copyright © 2004 David M. Hassenzahl Random draw one p(irritate) = 0.0165 mg × 0.063/mg = 0.0010

12. Copyright © 2004 David M. Hassenzahl Random draw two p(irritate) = 0.0175 mg × 0.089 /mg = 0.0016 Summary: {0.0010, 0.0016}

13. Copyright © 2004 David M. Hassenzahl Random draw three p(irritate) = 0.152 mg × 0.057 /mg = 0.0087 Summary: {0.0010, 0.0016, 0.00087}

14. Copyright © 2004 David M. Hassenzahl Random draw four p(irritate) = 0.0238 mg × 0.085 /mg = 0.0020 Summary: {0.0010, 0.0016, 0.00087, 0.0020}

15. Copyright © 2004 David M. Hassenzahl After ten random draws Summary {0.0010, 0.0016, 0.00087, 0.0020, 0.0011, 0.0018, 0.0024, 0.0016, 0.0015, 0.00062} mean 0.0014 standard deviation (0.00055)

16. Copyright © 2004 David M. Hassenzahl Using software Could write this program using a random number generator But, several software packages out there. I use Crystal Ball –user friendly –customizable –r.n.g. good up to about 10,000 iterations

17. Copyright © 2004 David M. Hassenzahl 100 iterations (about two seconds) Monte Carlo results –Mean0.0016 –Standard Deviation0.00048 –“Conservative” estimate0.0026 Compare to analytical results –Mean 0.0015 –standard deviationn/a –“Conservative” estimate0.0029

18. Copyright © 2004 David M. Hassenzahl Summary chart - 100 trials

19. Copyright © 2004 David M. Hassenzahl Summary - 10,000 trials Monte Carlo results –Mean0.0015 –Standard Deviation0.000472 –“Conservative” estimate0.0024 Compare to analytical results –Mean 0.0015 –standard deviationn/a –“Conservative” estimate0.0029

20. Copyright © 2004 David M. Hassenzahl Summary chart - 10,000 trials About 1.5 minutes run time

21. Copyright © 2004 David M. Hassenzahl Policy applications When there are many distributional inputs Concern about “excessive conservatism” –multiplying 95 th percentiles –multiple exposures Because we can Bayesian calculations

22. Copyright © 2004 David M. Hassenzahl Issues: Sensitivity Analysis Sensitivity analysis looks at which input distributions have the greatest effect on the eventual distribution Helps to understand which parameters can both be influenced by policy and reduce risks Helps understand when better data can be most valuable (information isn’t free…nor even cheap)

23. Copyright © 2004 David M. Hassenzahl Issues: Correlation Two distributions are correlated when a change in one causes a change in another Example: People who eat lots of peas may eat less broccoli (or may eat more…) Usually doesn’t have much effect unless significant correlation (|  |>0.75)

24. Copyright © 2004 David M. Hassenzahl Generating Distributions Invalid distributions create invalid results, which leads to inappropriate policies Two options –empirical –theoretical

25. Copyright © 2004 David M. Hassenzahl Empirical Distributions Most appropriate when developed for the issue at hand. Example: local fish consumption –survey individuals or otherwise estimate –data from individuals elsewhere may be very misleading A number of very large data sets have been developed and published

26. Copyright © 2004 David M. Hassenzahl Empirical Distributions Challenge: when there’s very little data Example of two data points –uniform distribution? –triangular distribution? –not a hypothetical issue…is an ongoing debate in the literature Key is to state clearly your assumptions Better yet…do it both ways!

27. Copyright © 2004 David M. Hassenzahl Which Distribution?

28. Copyright © 2004 David M. Hassenzahl Random number generation Shouldn’t be an issue…@Risk and Crystal Ball are both good to at least 10,000 iterations 10,000 iterations is typically enough, even with many input distributions

29. Copyright © 2004 David M. Hassenzahl Theoretical Distributions Appropriate when there’s some mechanistic or probabilistic basis Example: small sample (say 50 test animals) establishes a binomial distribution Lognormal distributions show up often in nature

30. Copyright © 2004 David M. Hassenzahl Some Caveats Beware believing that you’ve really “understood” uncertainty Beware: misapplication –ignorance at best –fraudulent at worst…porcine hoof blister

31. Copyright © 2004 David M. Hassenzahl Example (after Finkel) Alar “versus” aflatoxin Exposure has two elements Peanut butter consumption aflatoxin residue Juice consumption Alar/UDMH residue Potency has one element aflatoxin potencyUDMH potency Risk = (consumption  residue  potency)/body weight

32. Copyright © 2004 David M. Hassenzahl Inputs for Alar & aflatoxin

33. Copyright © 2004 David M. Hassenzahl Alar and aflatoxin point estimates aflatoxin estimates: –Mean = 0.028 –Conservative = 0.29 Alar (UDMH) estimates: –Mean = 0.046 –Conservative = 0.77

34. Copyright © 2004 David M. Hassenzahl Alar and aflatoxin Monte Carlo 10,000 runs Generate distributions –(don’t allow 0) Don’t expect correlation

35. Copyright © 2004 David M. Hassenzahl Aflatoxin and Alar Monte Carlo results (point values)

36. Copyright © 2004 David M. Hassenzahl Aflatoxin and Alar Monte Carlo results (distributions)

37. Copyright © 2004 David M. Hassenzahl Aflatoxin and Alar Monte Carlo results (distributions)

38. Copyright © 2004 David M. Hassenzahl Aflatoxin and Alar Monte Carlo results (distributions)

39. Copyright © 2004 David M. Hassenzahl Aflatoxin and Alar Monte Carlo results (distributions)

40. Copyright © 2004 David M. Hassenzahl Aflatoxin and Alar Monte Carlo results (distributions)

41. Copyright © 2004 David M. Hassenzahl Aflatoxin and Alar Monte Carlo results (distributions)

42. Copyright © 2004 David M. Hassenzahl References and Further Reading Burmaster, D.E and Anderson, P.D. (1994). “Principles of good practice for the use of Monte Carlo techniques in human health and ecological risk assessments.” Risk Analysis 14(4):447-81 Finkel, A (1995). “Towards less misleading comparisons of uncertain risks: the example of aflatoxin and Alar.” Environmental Health Perspectives 103(4):376-85. Kammen, D.M and Hassenzahl D.M. (1999). Should We Risk It? Exploring Environmental, Health and Technological Problem Solving. Princeton University Press, Princeton, NJ. Thompson, K. M., D. E. Burmaster, et al. (1992). "Monte Carlo techniques for uncertainty analysis in public health risk assessments." Risk Analysis 12(1): 53-63. Vose, David (1997) “Monte Carlo Risk Analysis Modeling” in Molak, Ed., Fundamentals of Risk Analysis and Risk Management.