ACCURACY IN PERCENTILES

ACCURACY OF PERCENTILES Sometimes, we may want accuracy in the percentiles of a distribution estimated using simulation: Rather than the mean value of the distribution 2

EXAMPLES X is the profit or loss from a new business venture: What is probability that a loss causes bankruptcy? Y is the temperature of the fuel in a nuclear power plant: What is probability that Y exceeds the melting point? We estimate that the probability of falling into this region is 3.7%. How can we measure the accuracy of that estimate? 3

STANDARD DEVIATION OF PERCENTILES If we do N simulations, and K = number of simulation points beyond some cutoff: Such as Y = 550 Then K is binomially distributed with parameters N and So, we have 4

STANDARD DEVIATION OF PERCENTILES The binomial distribution has: Let our estimate of the true probability p be given by: Then we have: 5

NORMAL APPROXIMATION TO BINOMIAL Our estimator has: By the normal approximation to the binomial distribution: Normal estimation gives good results if: 6

CONFIDENCE INTERVALS To get a 90% confidence interval for p: 7

EXAMPLE (1% ABSOLUTE ERROR) Estimate percentiles to within Number of points needed to achieve 1% error: p Confidence Interval N .5 .49-.51 6765 .9 .89-.91 2436 .95 .94-.96 1286 .99 .98-1.0 267500 Because we need N (1 – p) ≥ 5 8

EXAMPLE (10% RELATIVE ERROR) Estimate percentiles to within p Confidence Interval N .5 .45-.55 271 .9 .89-.91 2,436 .95 .945-.955 5,142 .99 .989-.991 26,790 .999 .9989-.9991 270,332 .9999 .99989-.99991 2,705,754 May not need ±10% relative error, but at a minimum for p=.9999 (1 - p=10-4), would need 50,000 points to get any accuracy at all! 9

EXAMPLE For a fixed sample size (N=10,000): p Percent Error, E/(1-p) Range .5 5% .475-.525 .9 16% .884-.916 .95 23% .939-.961 .99 27% .987-.993 .999 164% ??? 10

ACCURACY OF PERCENTILES Say we want to estimate P(core melt) by simulating peak fuel temperature, and then taking: To demonstrate that P(core melt) ≤ 10-4: Must estimate 99.99th percentile of the distribution Would need a minimum of 50,000 simulation runs! Even one simulation run may be extremely expensive: E.g., a detailed finite-element calculation Besides, we probably don’t believe the tails of the input distributions anyway! 11

WHAT CAN WE DO? In conventional probabilistic risk analysis: We estimate P(core melt) directly As a function of component-failure probabilities So, we are no longer estimating a rare tail probability: Tails of the input distributions are not so important May be able to get acceptable accuracy with a reasonable number of simulation runs 12

WHAT ELSE CAN WE DO? Buy a supercomputer Use a simpler calculation, so can do 50,000 samples: But what approximation is introduced? Extrapolate the tails: Based on distribution fitting from fewer samples Use variance-reduction methods—e.g.: Importance sampling, to sample from tails Latin hypercube sampling Find alternative approaches—e.g.: Greater reliance on expert opinion or experiments 13

IMPORTANCE SAMPLING Sample disproportionately from tails of the distribution: Then correct for the oversampling in the analysis Unfortunately, this is difficult to do properly: Can lead to increased rather than decreased variance! 14

LATIN HYPERCUBE SAMPLING Divide all input distributions into M equal-probability bins: For example, this shows a case with M = 8 bins Then ensure that in each batch of M samples: One sample is chosen from each bin So tails are sampled systematically, not randomly 15

LATIN HYPERCUBE SAMPLING To estimate the mean value of the output distribution: Compute the mean value of each batch of M samples These can be used to estimate accuracy of the mean Because the various “batch means” are independent Can also estimate percentiles of the output distribution: Based on the individual samples, not the batch means However, the properties of Latin hypercube sampling for estimating the percentiles of the distribution are not known: Samples from the same batch are not independent! 16

EXAMPLE: PASSIVE SAFETY In theory, passively safe reactors are supposed to provide greater safety, with less need for regulation: However, it’s actually difficult to demonstrate this using probabilistic risk analysis P(core melt) is no longer a function of component-failure probabilities: But depends on whether natural convection fails for some rare combination of parameter values This requires analysis of tail probabilities! In practice, it may even be necessary to require the addition of active systems (subject to regulation)! Links and Notes: Mackay, F. J., et al. (2007), “Incorporating Reliability Analysis into the Design of Passive Cooling Systems with an Application to a Gas-cooled Reactor.” Nuclear Engineering and Design, Vol. 238, No. 1, pp. 217-228. 17