Copyright © 2004 David M. Hassenzahl Monte Carlo Analysis David M. Hassenzahl
Copyright © 2004 David M. Hassenzahl Purpose of lecture Introduce Monte Carlo Analysis as a tool for managing uncertainty Demonstrate how it can be used in the policy setting Discuss its uses and shortcomings, and how they are relevant to policy making processes
Copyright © 2004 David M. Hassenzahl What is Monte Carlo Analysis? It is a tool for combining distributions, and thereby propagating more than just summary statistics It uses random number generation, rather than analytic calculations It is increasingly popular due to high speed personal computers
Copyright © 2004 David M. Hassenzahl Background/History “Monte Carlo” from the gambling town of the same name (no surprise) First applied in 1947 to model diffusion of neutrons through fissile materials Limited use because time consuming Much more common since late 80’s Too easy now? Name…is EPA “gambling” with people’s lives (anecdotal, but reasonable).
Copyright © 2004 David M. Hassenzahl Why Perform Monte Carlo Analysis? Combining distributions With more than two distributions, solving analytically is very difficult Simple calculations lose information – Mean mean = mean –95% %ile 95%ile 95%ile! –Gets “worse” with 3 or more distributions
Copyright © 2004 David M. Hassenzahl Monte Carlo Analysis Takes an equation –example: Risk = probability consequence Instead of simple numbers, draws randomly from defined distributions Multiplies the two, stores the answer Repeats this over and over and over… Then the set of results is displayed as a new, combined distribution
Copyright © 2004 David M. Hassenzahl Simple (hypothetical) example Skin cream additive is an irritant Many samples of cream provide information on concentration: –mean 0.02 mg chemical –standard dev mg chemical Two tests show probability of irritation given application –low freq of effect per mg exposure = 5/100/mg –high freq of effect per mg exposure = 10/100/mg
Copyright © 2004 David M. Hassenzahl Analytical results Risk = exposure potency –Mean risk = 0.02 mg / mg = or 15 out of 10,000 applications will result in irritation
Copyright © 2004 David M. Hassenzahl Analytical results “Conservative estimate” –Use upper 95 th %ile Risk = 0.03 mg / mg =
Copyright © 2004 David M. Hassenzahl Monte Carlo: Visual example Exposure = normal(mean 0.02 mg, s.d. = mg) potency = uniform (range 0.05 / mg to 0.10 / mg)
Copyright © 2004 David M. Hassenzahl Random draw one p(irritate) = mg × 0.063/mg =
Copyright © 2004 David M. Hassenzahl Random draw two p(irritate) = mg × /mg = Summary: {0.0010, }
Copyright © 2004 David M. Hassenzahl Random draw three p(irritate) = mg × /mg = Summary: {0.0010, , }
Copyright © 2004 David M. Hassenzahl Random draw four p(irritate) = mg × /mg = Summary: {0.0010, , , }
Copyright © 2004 David M. Hassenzahl After ten random draws Summary {0.0010, , , , , , , , , } mean standard deviation ( )
Copyright © 2004 David M. Hassenzahl Using software Could write this program using a random number generator But, several software packages out there. I use Crystal Ball –user friendly –customizable –r.n.g. good up to about 10,000 iterations
Copyright © 2004 David M. Hassenzahl 100 iterations (about two seconds) Monte Carlo results –Mean –Standard Deviation –“Conservative” estimate Compare to analytical results –Mean –standard deviationn/a –“Conservative” estimate0.0029
Copyright © 2004 David M. Hassenzahl Summary chart trials
Copyright © 2004 David M. Hassenzahl Summary - 10,000 trials Monte Carlo results –Mean –Standard Deviation –“Conservative” estimate Compare to analytical results –Mean –standard deviationn/a –“Conservative” estimate0.0029
Copyright © 2004 David M. Hassenzahl Summary chart - 10,000 trials About 1.5 minutes run time
Copyright © 2004 David M. Hassenzahl Policy applications When there are many distributional inputs Concern about “excessive conservatism” –multiplying 95 th percentiles –multiple exposures Because we can Bayesian calculations
Copyright © 2004 David M. Hassenzahl Issues: Sensitivity Analysis Sensitivity analysis looks at which input distributions have the greatest effect on the eventual distribution Helps to understand which parameters can both be influenced by policy and reduce risks Helps understand when better data can be most valuable (information isn’t free…nor even cheap)
Copyright © 2004 David M. Hassenzahl Issues: Correlation Two distributions are correlated when a change in one causes a change in another Example: People who eat lots of peas may eat less broccoli (or may eat more…) Usually doesn’t have much effect unless significant correlation (| |>0.75)
Copyright © 2004 David M. Hassenzahl Generating Distributions Invalid distributions create invalid results, which leads to inappropriate policies Two options –empirical –theoretical
Copyright © 2004 David M. Hassenzahl Empirical Distributions Most appropriate when developed for the issue at hand. Example: local fish consumption –survey individuals or otherwise estimate –data from individuals elsewhere may be very misleading A number of very large data sets have been developed and published
Copyright © 2004 David M. Hassenzahl Empirical Distributions Challenge: when there’s very little data Example of two data points –uniform distribution? –triangular distribution? –not a hypothetical issue…is an ongoing debate in the literature Key is to state clearly your assumptions Better yet…do it both ways!
Copyright © 2004 David M. Hassenzahl Which Distribution?
Copyright © 2004 David M. Hassenzahl Random number generation Shouldn’t be an and Crystal Ball are both good to at least 10,000 iterations 10,000 iterations is typically enough, even with many input distributions
Copyright © 2004 David M. Hassenzahl Theoretical Distributions Appropriate when there’s some mechanistic or probabilistic basis Example: small sample (say 50 test animals) establishes a binomial distribution Lognormal distributions show up often in nature
Copyright © 2004 David M. Hassenzahl Some Caveats Beware believing that you’ve really “understood” uncertainty Beware: misapplication –ignorance at best –fraudulent at worst…porcine hoof blister
Copyright © 2004 David M. Hassenzahl Example (after Finkel) Alar “versus” aflatoxin Exposure has two elements Peanut butter consumption aflatoxin residue Juice consumption Alar/UDMH residue Potency has one element aflatoxin potencyUDMH potency Risk = (consumption residue potency)/body weight
Copyright © 2004 David M. Hassenzahl Inputs for Alar & aflatoxin
Copyright © 2004 David M. Hassenzahl Alar and aflatoxin point estimates aflatoxin estimates: –Mean = –Conservative = 0.29 Alar (UDMH) estimates: –Mean = –Conservative = 0.77
Copyright © 2004 David M. Hassenzahl Alar and aflatoxin Monte Carlo 10,000 runs Generate distributions –(don’t allow 0) Don’t expect correlation
Copyright © 2004 David M. Hassenzahl Aflatoxin and Alar Monte Carlo results (point values)
Copyright © 2004 David M. Hassenzahl Aflatoxin and Alar Monte Carlo results (distributions)
Copyright © 2004 David M. Hassenzahl Aflatoxin and Alar Monte Carlo results (distributions)
Copyright © 2004 David M. Hassenzahl Aflatoxin and Alar Monte Carlo results (distributions)
Copyright © 2004 David M. Hassenzahl Aflatoxin and Alar Monte Carlo results (distributions)
Copyright © 2004 David M. Hassenzahl Aflatoxin and Alar Monte Carlo results (distributions)
Copyright © 2004 David M. Hassenzahl Aflatoxin and Alar Monte Carlo results (distributions)
Copyright © 2004 David M. Hassenzahl References and Further Reading Burmaster, D.E and Anderson, P.D. (1994). “Principles of good practice for the use of Monte Carlo techniques in human health and ecological risk assessments.” Risk Analysis 14(4): Finkel, A (1995). “Towards less misleading comparisons of uncertain risks: the example of aflatoxin and Alar.” Environmental Health Perspectives 103(4): Kammen, D.M and Hassenzahl D.M. (1999). Should We Risk It? Exploring Environmental, Health and Technological Problem Solving. Princeton University Press, Princeton, NJ. Thompson, K. M., D. E. Burmaster, et al. (1992). "Monte Carlo techniques for uncertainty analysis in public health risk assessments." Risk Analysis 12(1): Vose, David (1997) “Monte Carlo Risk Analysis Modeling” in Molak, Ed., Fundamentals of Risk Analysis and Risk Management.