ACCURACY IN PERCENTILES

Slides:



Advertisements
Similar presentations
Chapter 7 Hypothesis Testing
Advertisements

“Students” t-test.
Hypothesis Testing. To define a statistical Test we 1.Choose a statistic (called the test statistic) 2.Divide the range of possible values for the test.
Estimation in Sampling
Sampling: Final and Initial Sample Size Determination
POINT ESTIMATION AND INTERVAL ESTIMATION
Point estimation, interval estimation
Lec 6, Ch.5, pp90-105: Statistics (Objectives) Understand basic principles of statistics through reading these pages, especially… Know well about the normal.
Fundamentals of Sampling Method
Probability & Statistics for Engineers & Scientists, by Walpole, Myers, Myers & Ye ~ Chapter 10 Notes Class notes for ISE 201 San Jose State University.
Chapter Topics Confidence Interval Estimation for the Mean (s Known)
8-2 Basics of Hypothesis Testing
Copyright © 2010, 2007, 2004 Pearson Education, Inc. Lecture Slides Elementary Statistics Eleventh Edition and the Triola Statistics Series by.
BCOR 1020 Business Statistics
Understanding sample survey data
Quiz 6 Confidence intervals z Distribution t Distribution.
Computer Vision Lecture 8 Performance Evaluation.
Confidence Intervals and Hypothesis Testing - II
The paired sample experiment The paired t test. Frequently one is interested in comparing the effects of two treatments (drugs, etc…) on a response variable.
Statistical Decision Making. Almost all problems in statistics can be formulated as a problem of making a decision. That is given some data observed from.
Learning Objectives In this chapter you will learn about the t-test and its distribution t-test for related samples t-test for independent samples hypothesis.
2.6 Confidence Intervals and Margins of Error. What you often see in reports about studies… These results are accurate to within +/- 3.7%, 19 times out.
Statistical Hypotheses & Hypothesis Testing. Statistical Hypotheses There are two types of statistical hypotheses. Null Hypothesis The null hypothesis,
Statistical Inference Statistical Inference is the process of making judgments about a population based on properties of the sample Statistical Inference.
Estimation Chapter 8. Estimating µ When σ Is Known.
Chapter 5 Parameter estimation. What is sample inference? Distinguish between managerial & financial accounting. Understand how managers can use accounting.
1 Chapter 8 Hypothesis Testing 8.2 Basics of Hypothesis Testing 8.3 Testing about a Proportion p 8.4 Testing about a Mean µ (σ known) 8.5 Testing about.
Copyright © 2010, 2007, 2004 Pearson Education, Inc. All Rights Reserved. Section 7-1 Review and Preview.
Sampling and Statistical Analysis for Decision Making A. A. Elimam College of Business San Francisco State University.
Lesoon Statistics for Management Confidence Interval Estimation.
Statistical Inference for the Mean Objectives: (Chapter 8&9, DeCoursey) -To understand the terms variance and standard error of a sample mean, Null Hypothesis,
Copyright (c) 2004 Brooks/Cole, a division of Thomson Learning, Inc. Chapter 7 Inferences Concerning Means.
Statistical Decision Making. Almost all problems in statistics can be formulated as a problem of making a decision. That is given some data observed from.
Lecture Slides Elementary Statistics Twelfth Edition
Chapter Nine Hypothesis Testing.
Chapter 8: Inferences Based on a Single Sample: Tests of Hypotheses
Chapter 8: Estimating with Confidence
Chapter 7 Review.
Chapter 8: Estimating with Confidence
Confidence Intervals and Sample Size
Chapter 6 Inferences Based on a Single Sample: Estimation with Confidence Intervals Slides for Optional Sections Section 7.5 Finite Population Correction.
ESTIMATION.
MSA / Gage Capability (GR&R)
Inference and Tests of Hypotheses
STAT 312 Chapter 7 - Statistical Intervals Based on a Single Sample
Lecture Slides Essentials of Statistics 5th Edition
Math 4030 – 10a Tests for Population Mean(s)
Hypothesis Testing and Confidence Intervals (Part 1): Using the Standard Normal Lecture 8 Justin Kern October 10 and 12, 2017.
Chapter 8: Inference for Proportions
CONCEPTS OF HYPOTHESIS TESTING
Introduction to Instrumentation Engineering
Estimation Goal: Use sample data to make predictions regarding unknown population parameters Point Estimate - Single value that is best guess of true parameter.
LESSON 18: CONFIDENCE INTERVAL ESTIMATION
Chapter 8: Estimating with Confidence
Introduction to Estimation
OMGT LECTURE 10: Elements of Hypothesis Testing
Chapter 8: Estimating with Confidence
One-Sample Tests of Hypothesis
Chapter 8: Estimating with Confidence
Chapter 8: Estimating with Confidence
Determining Which Method to use
Chapter 8: Estimating with Confidence
Chapter 8: Estimating with Confidence
Chapter 8: Estimating with Confidence
Chapter 8: Estimating with Confidence
Chapter 8: Estimating with Confidence
Chapter 8: Estimating with Confidence
Chapter 8: Estimating with Confidence
Simulation Berlin Chen
Business Statistics For Contemporary Decision Making 9th Edition
Presentation transcript:

ACCURACY IN PERCENTILES

ACCURACY OF PERCENTILES Sometimes, we may want accuracy in the percentiles of a distribution estimated using simulation: Rather than the mean value of the distribution 2

EXAMPLES X is the profit or loss from a new business venture: What is probability that a loss causes bankruptcy? Y is the temperature of the fuel in a nuclear power plant: What is probability that Y exceeds the melting point? We estimate that the probability of falling into this region is 3.7%. How can we measure the accuracy of that estimate? 3

STANDARD DEVIATION OF PERCENTILES If we do N simulations, and K = number of simulation points beyond some cutoff: Such as Y = 550 Then K is binomially distributed with parameters N and So, we have 4

STANDARD DEVIATION OF PERCENTILES The binomial distribution has: Let our estimate of the true probability p be given by: Then we have: 5

NORMAL APPROXIMATION TO BINOMIAL Our estimator has: By the normal approximation to the binomial distribution: Normal estimation gives good results if: 6

CONFIDENCE INTERVALS To get a 90% confidence interval for p: 7

EXAMPLE (1% ABSOLUTE ERROR) Estimate percentiles to within Number of points needed to achieve 1% error: p Confidence Interval N .5 .49-.51 6765 .9 .89-.91 2436 .95 .94-.96 1286 .99 .98-1.0 267500 Because we need N (1 – p) ≥ 5 8

EXAMPLE (10% RELATIVE ERROR) Estimate percentiles to within p Confidence Interval N .5 .45-.55 271 .9 .89-.91 2,436 .95 .945-.955 5,142 .99 .989-.991 26,790 .999 .9989-.9991 270,332 .9999 .99989-.99991 2,705,754 May not need ±10% relative error, but at a minimum for p=.9999 (1 - p=10-4), would need 50,000 points to get any accuracy at all! 9

EXAMPLE For a fixed sample size (N=10,000): p Percent Error, E/(1-p) Range .5 5% .475-.525 .9 16% .884-.916 .95 23% .939-.961 .99 27% .987-.993 .999 164% ??? 10

ACCURACY OF PERCENTILES Say we want to estimate P(core melt) by simulating peak fuel temperature, and then taking: To demonstrate that P(core melt) ≤ 10-4: Must estimate 99.99th percentile of the distribution Would need a minimum of 50,000 simulation runs! Even one simulation run may be extremely expensive: E.g., a detailed finite-element calculation Besides, we probably don’t believe the tails of the input distributions anyway! 11

WHAT CAN WE DO? In conventional probabilistic risk analysis: We estimate P(core melt) directly As a function of component-failure probabilities So, we are no longer estimating a rare tail probability: Tails of the input distributions are not so important May be able to get acceptable accuracy with a reasonable number of simulation runs 12

WHAT ELSE CAN WE DO? Buy a supercomputer Use a simpler calculation, so can do 50,000 samples: But what approximation is introduced? Extrapolate the tails: Based on distribution fitting from fewer samples Use variance-reduction methods—e.g.: Importance sampling, to sample from tails Latin hypercube sampling Find alternative approaches—e.g.: Greater reliance on expert opinion or experiments 13

IMPORTANCE SAMPLING Sample disproportionately from tails of the distribution: Then correct for the oversampling in the analysis Unfortunately, this is difficult to do properly: Can lead to increased rather than decreased variance! 14

LATIN HYPERCUBE SAMPLING Divide all input distributions into M equal-probability bins: For example, this shows a case with M = 8 bins Then ensure that in each batch of M samples: One sample is chosen from each bin So tails are sampled systematically, not randomly 15

LATIN HYPERCUBE SAMPLING To estimate the mean value of the output distribution: Compute the mean value of each batch of M samples These can be used to estimate accuracy of the mean Because the various “batch means” are independent Can also estimate percentiles of the output distribution: Based on the individual samples, not the batch means However, the properties of Latin hypercube sampling for estimating the percentiles of the distribution are not known: Samples from the same batch are not independent! 16

EXAMPLE: PASSIVE SAFETY In theory, passively safe reactors are supposed to provide greater safety, with less need for regulation: However, it’s actually difficult to demonstrate this using probabilistic risk analysis P(core melt) is no longer a function of component-failure probabilities: But depends on whether natural convection fails for some rare combination of parameter values This requires analysis of tail probabilities! In practice, it may even be necessary to require the addition of active systems (subject to regulation)! Links and Notes: Mackay, F. J., et al. (2007), “Incorporating Reliability Analysis into the Design of Passive Cooling Systems with an Application to a Gas-cooled Reactor.” Nuclear Engineering and Design, Vol. 238, No. 1, pp. 217-228. 17