1. ACCURACY IN PERCENTILES

2. ACCURACY OF PERCENTILES
Sometimes, we may want accuracy in the percentiles of a distribution estimated using simulation:
Rather than the mean value of the distribution 2

3. EXAMPLES X is the profit or loss from a new business venture:
What is probability that a loss causes bankruptcy?
Y is the temperature of the fuel in a nuclear power plant:
What is probability that Y exceeds the melting point?
We estimate that the probability of falling into this region is 3.7%.
How can we measure the accuracy of that estimate? 3

4. STANDARD DEVIATION OF PERCENTILES
If we do N simulations, and
K = number of simulation points beyond some cutoff:
Such as Y = 550
Then K is binomially distributed with parameters N and
So, we have 4

5. STANDARD DEVIATION OF PERCENTILES
The binomial distribution has:
Let our estimate of the true probability p be given by:
Then we have: 5

6. NORMAL APPROXIMATION TO BINOMIAL
Our estimator has:
By the normal approximation to the binomial distribution:
Normal estimation gives good results if: 6

7. CONFIDENCE INTERVALS To get a 90% confidence interval for p:7

8. EXAMPLE (1% ABSOLUTE ERROR)
Estimate percentiles to within
Number of points needed to achieve 1% error: p
Confidence Interval N .5
6765 .9
2436
.95
1286
.99
267500
Because we need
N (1 – p) ≥ 5 8

9. EXAMPLE (10% RELATIVE ERROR)
Estimate percentiles to within p
Confidence Interval N .5
271 .9
2,436
.95
5,142
.99
26,790
.999
270,332
.9999
2,705,754
May not need ±10% relative error, but at a minimum for p=.9999
(1 - p=10-4), would need 50,000 points to get any accuracy at all! 9

10. EXAMPLE For a fixed sample size (N=10,000): p Percent Error, E/(1-p)
Range .5 5% .9
16%
.95
23%
.99
27%
.999
164%
??? 10

11. ACCURACY OF PERCENTILES
Say we want to estimate P(core melt) by simulating peak fuel temperature, and then taking:
To demonstrate that P(core melt) ≤ 10-4:
Must estimate 99.99th percentile of the distribution
Would need a minimum of 50,000 simulation runs!
Even one simulation run may be extremely expensive:
E.g., a detailed finite-element calculation
Besides, we probably don’t believe the tails of the input distributions anyway! 11

12. WHAT CAN WE DO? In conventional probabilistic risk analysis:
We estimate P(core melt) directly
As a function of component-failure probabilities
So, we are no longer estimating a rare tail probability:
Tails of the input distributions are not so important
May be able to get acceptable accuracy with a reasonable number of simulation runs 12

13. WHAT ELSE CAN WE DO? Buy a supercomputer
Use a simpler calculation, so can do 50,000 samples:
But what approximation is introduced?
Extrapolate the tails:
Based on distribution fitting from fewer samples
Use variance-reduction methods—e.g.:
Importance sampling, to sample from tails
Latin hypercube sampling
Find alternative approaches—e.g.:
Greater reliance on expert opinion or experiments 13

14. IMPORTANCE SAMPLING
Sample disproportionately from tails of the distribution:
Then correct for the oversampling in the analysis
Unfortunately, this is difficult to do properly:
Can lead to increased rather than decreased variance! 14

15. LATIN HYPERCUBE SAMPLING
Divide all input distributions into M equal-probability bins:
For example, this shows a case with M = 8 bins
Then ensure that in each batch of M samples:
One sample is chosen from each bin
So tails are sampled systematically, not randomly 15

16. LATIN HYPERCUBE SAMPLING
To estimate the mean value of the output distribution:
Compute the mean value of each batch of M samples
These can be used to estimate accuracy of the mean
Because the various “batch means” are independent
Can also estimate percentiles of the output distribution:
Based on the individual samples, not the batch means
However, the properties of Latin hypercube sampling for estimating the percentiles of the distribution are not known:
Samples from the same batch are not independent! 16

17. EXAMPLE: PASSIVE SAFETY
In theory, passively safe reactors are supposed to provide greater safety, with less need for regulation:
However, it’s actually difficult to demonstrate this using probabilistic risk analysis
P(core melt) is no longer a function of component-failure probabilities:
But depends on whether natural convection fails for some rare combination of parameter values
This requires analysis of tail probabilities!
In practice, it may even be necessary to require the addition of active systems (subject to regulation)!
Links and Notes: Mackay, F. J., et al. (2007), “Incorporating Reliability Analysis into the Design of Passive Cooling Systems with an Application to a Gas-cooled Reactor.” Nuclear Engineering and Design, Vol. 238, No. 1, pp 17