1 of 31 The EPA 7-Step DQO Process Step 6 - Specify Error Tolerances 60 minutes (15 minute Morning Break) Presenter: Sebastian Tindall DQO Training Course Day 3 Module 15

2 of 31 Step Objective: n To specify the decision makers’ tolerable limits on decision errors, which are used for limiting uncertainty in the data –Since analytical data can only provide an estimate the true condition of a site, decisions that are based on such data could potentially be in error Step 6: Specify Error Tolerances Step 4: Specify Boundaries Step 2: Identify Decisions Step 3: Identify Inputs Step 1: State the Problem Step 5: Define Decision Rules Step 6 : Specify Error Tolerances Step 7 : Optimize Sample Design

3 of 31 Objectives To be able to define, for a specific project: 1. The variability for each COPC, 2. the decision errors, 3. the consequences of the errors, 4. the null hypothesis, 5. the error rates (Alpha and Beta) 6. the Lower Bound of the Gray Region & Delta

4 of 31 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Step 6- Specify Error Tolerances Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region

5 of 31 Decision Error Tolerances n The goal of the planning team is to develop a data collection design that reduces the chance of making a decision error to a tolerable level n Step 6 provides a mechanism for allowing the decision maker to define tolerable limits on the probability of making a decision error

6 of 31 Two Reasons Why Decision Makers Make Decision Errors n Sampling error occurs because the sampling design is unable to capture and control the complete extent of heterogeneity that exists in the true state of the environment n Measurement error occurs because analytical methods and instruments are not absolutely perfect

7 of 31 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region In order to calculate the number of samples needed (in DQO Step 7), an estimate of the population standard deviation is needed for each environmental variable. Compile a list of the “driver” COPCs Use existing data (must pass Step 3 data assessments) Establish the range based on historical information – Existing data – Process knowledge – Professional judgment Estimate of the population standard deviation – Reference source – Method of calculating Step 6- Specify Error Tolerances

8 of 31 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region In order to calculate the number of samples needed (in DQO Step 7), an estimate of the population standard deviation is needed for each environmental variable. Compile a list of the “driver” COPCs Use existing data (must pass Step 3 data assessments) Establish the range based on historical information – Existing data – Process knowledge – Professional judgment Estimate of the population standard deviation – Reference source – Method of calculating Estimate the standard deviation by using the Deming approach of dividing the range by 2 or 3, depending on the frequency distribution. Step 6- Specify Error Tolerances

9 of 31 Estimated Standard Deviations The choice of an estimate of a standard deviation has a large impact on the number of samples required. Avoid underestimating the standard deviation. Always be conservative when estimating the standard deviation.

10 of 31 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region Define both types of decision error: Determine which one occurs above and which one occurs below the action level. Two Types of Decision Error: Cleaning up a clean site Walking away from a dirty site Step 6- Specify Error Tolerances

11 of 31 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region For each Alternative Action: Create a list of possible decision error(s) that may occur if an action is incorrectly taken Discuss the consequences of making each decision error Rate the severity of the consequences of a decision error (i.e., low, moderate, severe) at a point: –Far below the action level –Below but near the action level –Above but near the action level –Far above the action level Indicate which decision error has the most severe consequence near the action level Step 6- Specify Error Tolerances

12 of 31 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region Provide rationale for rating the severity of consequences as low or severe Step 6- Specify Error Tolerances

13 of 31 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region Define the null hypothesis (baseline condition) and the alternative hypothesis: The decision error that has the most adverse potential consequences should be defined as the null hypothesis. The null hypothesis should state the OPPOSITE of what the project hopes to demonstrate. Site is assumed to be contaminated until shown to be clean Site is assumed to be clean until shown to be contaminated Step 6- Specify Error Tolerances

14 of 31 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region The gray region is a range of possible parameter values within which the consequences of a decision error are relatively minor. Step 6- Specify Error Tolerances

15 of 31 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region The gray region is bounded on one side by the action level, and on the other side by the parameter value where the consequences of decision error begins to be significant. This point is labeled LBGR, which stands for lower bound of the gray region. Step 6- Specify Error Tolerances

16 of 31 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the possible range of the parameter of interest Choose the null hypothesis. Identify the decision errors. Specify the boundaries of the gray region Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors It is necessary to specify the gray region because variability in the population and unavoidable imprecision in the measurement system combine to produce variability in the data such that a decision may be “too close to call” when the true parameter value is very near the action level. Step 6- Specify Error Tolerances

17 of 31 Width of the Gray Region (  ) : UBGR - LBGR or AL - LBGR n  = Analytical + Sampling Error –Estimated based on past data and general knowledge n  = 1/2 of the AL –For each COPC, calculate and set LBGR n  = % of the AL –For each COPC, calculate and set LBGR n  = PDF method –Use PDF for worst COPC to set LBGR

18 of 31 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region Present the rationale of how the LBGR was calculated or determined. Step 6- Specify Error Tolerances

19 of 31 Lower Bound of the Gray Region n Because the null hypothesis is that the site is contaminated, the upper bound of the gray region is set equal to the action level n The LBGR should be set at a value where the consequences of the decision error begin to be significant

20 of 31 How to Set the LBGR n LBGR set by the Analytical + Sampling Error n LBGR set to 1/2 Action Level n LBGR set to ~ 50 to 90% of AL (Decision- Maker “whim”) n LBGR set by the Probability Density Function (PDF) method UBGR - GR = LBGR AL -  = LBGR

21 of 31 n The LBGR is often based on unavoidable variability in the concentration data –The GR may be estimated based on the precision that the analytical methods allow plus an estimate as to the sampling variance –LBGR = AL – GR (Analytical + Sampling Error) n 100 ppm – (10 ppm + 31 ppm) = 59 ppm n MARSSIM suggests the LBGR be set as: –LBGR = AL – GR (1/2 AL) n 100 ppm – 50 ppm = 50 ppm How to Set the LBGR (cont.)

22 of 31 n The LBGR is often set at some other value –This is based on the decision makers’ choice and is not scientifically based –LBGR = AL – GR (20% of AL); n 100 ppm – 20 ppm = 80 ppm How to Set the LBGR (cont.)

23 of 31 n Use the Probability Density Function (PDF) method –The LBGR may be estimated based the Probability Density Function –Place the Action Level on the mean of the PDF –Ask: “Does a substantial amount of contaminant concentration values exceed the Action Level?” –Ask: “Is there a reasonably high probability that data from a small sample size will result in declaring the site clean?” –If “yes” to the first question and “no” to the second question, begin moving the PDF backwards along thex-axis towards zero concentration –Pause and ask both questions again –When the answer is “no” to the 1 st and “yes” to the 2nd, you have set the LBGR (e.g., where the mean of the PDF lies on the x-axis is now the LBGR) Use probability theory to show this How to Set the LBGR (cont.)

24 of 31 Methods for Evaluating the Attainment of Cleanup Standards - Volume 1: Soils and Solid Media EPA, February 1989 PB How to set the LBGR  1 is a hypothetical “true mean concentration where the site should be declared clean with a high probability”. (  1 = LBGR)

25 of 31 Show Probability Density Function Distribution Demonstration Show VDT File: Moving Gray Region

26 of 31 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region Assign probability values that reflect the decision maker’s tolerable limits for making an incorrect decision. At the action level (Alpha error) At the other bound of the gray region (Beta error) At a point far below the action level At a point far above the action level Step 6- Specify Error Tolerances Note: EPA QA/G9 recommends that you set both Alpha and Beta error rates to 1% to start.

27 of 31 Site is dirtySite is clean 100 True State of Site Alternative Action Walk away from siteClean up site 75 Probability of deciding that the site is dirty Lower Bound of Gray Region Four Decision-Maker Error Tolerance Locations Null Hypothesis: Site is dirty. True mean COPC Concentration Action Level The Gray Region

28 of 31 Step 6 Summary n Determine the variability for each COPC n Define the two types of error –Incorrectly walking away from a dirty site, or –Incorrectly cleaning a clean site n Evaluate severity of the incorrect decisions both below, above, and near the action level n Select the null hypothesis

29 of 31 Step 6 Summary n Establish a LBGR based on one of the four methods shown previously n Provide the basis for selecting the LBGR n Remember the closer the LBGR is to the action level, the more samples are needed n Assign probability limits on either side of the gray region (Delta) –Specify the error rates (Alpha and Beta) decision makers are willing to accept and provide rational for the rates

30 of 31 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region Step 6- Specify Error Tolerances

31 of 31 End of Module 15 Thank you Questions? We will now take a 15 minute break. Please be back in 15 minutes.