1. 1 of 48 The EPA 7-Step DQO Process Step 6 - Specify Error Tolerances 3:00 PM - 3:30 PM (30 minutes) Presenter: Sebastian Tindall Day 2 DQO Training Course Module 8

2. 2 of 48 Terminal Course Objective To be able to define the decision errors, consequences of the errors, the null hypothesis, and the lower bound of the gray region for a specific project

3. 3 of 48 Step Objective: n To specify the decision makers’ tolerable limits on decision errors, which are used for limiting uncertainty in the data –Since analytical data can only provide an estimate the true condition of a site, decisions that are based on such data could potentially be in error Step 6: Specify Error Tolerances Step 4: Specify Boundaries Step 2: Identify Decisions Step 3: Identify Inputs Step 1: State the Problem Step 5: Define Decision Rules Step 6 : Specify Error Tolerances Step 7 : Optimize Sample Design

4. 4 of 48 Step Objective: n To specify the decision makers’ tolerable limits on decision errors, which are used for limiting uncertainty in the data –Since analytical data can only provide an estimate the true condition of a site, decisions that are based on such data could potentially be in error Step 6: Specify Error Tolerances Step 4: Specify Boundaries Step 2: Identify Decisions Step 3: Identify Inputs Step 1: State the Problem Step 5: Define Decision Rules Step 6 : Specify Error Tolerances Step 7 : Optimize Sample Design Estimation Error Inherent in the process of estimation is error (deviation from the true value). That’s why it’s called estimation. Error  Mistake Error = Deviation

5. 5 of 48 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Step 6- Specify Error Tolerances Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region

6. 6 of 48 Decision Error Tolerances n The goal of the planning team is to develop a data collection design that reduces the chance of making a decision error to a tolerable level n Step 6 provides a mechanism for allowing the decision maker to define tolerable limits on the probability of making a decision error

7. 7 of 48 Two Reasons Why Decision Makers Make Decision Errors n Sampling error occurs because the sampling design is unable to capture the complete extent of heterogeneity that exists in the true state of the environment n Measurement error occurs because analytical methods and instruments are not absolutely perfect

8. 8 of 48 Where do errors occur? Planning Sampling Analysis Data Vs Decision

9. 9 of 48 Types of Decision Errors n Before we can talk about acceptable limits for making decision errors, we must first understand what correct decisions and decision errors look like n There are two types of correct decisions and two types of decision errors that can be made

10. 10 of 48 Null Hypothesis: Site is dirty. Site is clean. 100 True State of Site Alternative Action Walk away from site.Clean up site. 75 Probability of deciding that the site is dirty 0.0 0.5 1.0 Action Level Lower Bound of Gray Region Typical Curve Decision Performance Goal Diagram Sample Mean UCL True Mean Sample Mean UCL Sample Mean UCL True Mean

11. 11 of 48 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region In order to calculate the number of samples needed (in DQO Step 7), an estimate of the population standard deviation is needed for each environmental variable. Compile a list of the “driver” COPCs Use existing data (must pass Step 3 data assessments) Establish the range based on historical information – Existing data – Process knowledge – Professional judgment Estimate of the population standard deviation – Reference source – Method of calculating Step 6- Specify Error Tolerances

12. 12 of 48 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region In order to calculate the number of samples needed (in DQO Step 7), an estimate of the population standard deviation is needed for each environmental variable. Compile a list of the “driver” COPCs Use existing data (must pass Step 3 data assessments) Establish the range based on historical information – Existing data – Process knowledge – Professional judgment Estimate of the population standard deviation – Reference source – Method of calculating Estimate the standard deviation by using the Deming approach of dividing the range by 2 or 3, depending on the frequency distribution. Step 6- Specify Error Tolerances

13. 13 of 48 Estimated Standard Deviations

14. 14 of 48 Estimated Standard Deviations Perimeter Side-Slope Soil CS

15. 15 of 48 Estimated Standard Deviations Trench Footprint Soil CS

16. 16 of 48 Analytical + Sub-sampling + Natural heterogeneity of the site = Total Uncertainty Uncertainty is Additive! Remember the uncertainty is additive for all steps in sampling and analysis

17. 17 of 48 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region Define both types of decision error: Determine which one occurs above and which one occurs below the action level. Two Types of Decision Error: Cleaning up a clean site Walking away from a dirty site Step 6- Specify Error Tolerances

18. 18 of 48 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region For each Alternative Action: Create a list of possible decision error(s) that may occur if an action is incorrectly taken Discuss the consequences of making each decision error Rate the severity of the consequences of a decision error (i.e., low, moderate, severe) at a point: –Far below the action level –Below but near the action level –Above but near the action level –Far above the action level Indicate which decision error has the most severe consequence near the action level Step 6- Specify Error Tolerances

19. 19 of 48 Decision Error Consequences CS

20. 20 of 48 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region Provide rationale for rating the severity of consequences as low or severe Step 6- Specify Error Tolerances

21. 21 of 48 Rationale for Error Consequence Ratings CS

22. 22 of 48 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region Define the null hypothesis (baseline condition) and the alternative hypothesis: The decision error that has the most adverse potential consequences should be defined as the null hypothesis. The null hypothesis should state the OPPOSITE of what the project hopes to demonstrate. Site is assumed to be contaminated until shown to be clean Site is assumed to be clean until shown to be contaminated Step 6- Specify Error Tolerances

23. 23 of 48 Null Hypothesis Contaminated: H 0 :  > Action Level Uncontaminated: H A :  < Action Level CS

24. 24 of 48 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region The gray region is a range of possible parameter values within which the consequences of a decision error are relatively minor. Step 6- Specify Error Tolerances

25. 25 of 48 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region The gray region is bounded on one side by the action level, and on the other side by the parameter value where the consequences of decision error begins to be significant. This point is labeled LBGR, which stands for lower bound of the gray region. Step 6- Specify Error Tolerances

26. 26 of 48 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the possible range of the parameter of interest Choose the null hypothesis. Identify the decision errors. Specify the boundaries of the gray region Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors It is necessary to specify the gray region because variability in the population and unavoidable imprecision in the measurement system combine to produce variability in the data such that a decision may be “too close to call” when the true parameter value is very near the action level. Step 6- Specify Error Tolerances

27. 27 of 48 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region Present the rationale of how the LBGR was calculated or determined. Step 6- Specify Error Tolerances

28. 28 of 48 Lower Bound of the Gray Region n Because the null hypothesis is that the site is contaminated, the upper bound of the gray region is set equal to the action level n The LBGR should be set at a value where the consequences of the decision error begin to be significant

29. 29 of 48 How to Set the LBGR n LBGR = AL - (Analytical + Sampling Error) n LBGR = AL - 1/2 Action Level n LBGR = Decision-Maker “whim” - AL - 0.2 AL n LBGR = Frequency Distribution method

30. 30 of 48 n The LBGR is often based on unavoidable variability in the concentration data –The LBGR may be estimated based on the precision that the analytical methods allow plus an estimate as to the sampling variance –LBGR = AL - (Analytical + Sampling Error) n MARSSIM suggests the LBGR be set as: –LBGR = AL - 1/2 AL How to Set the LBGR (cont.)

31. 31 of 48 n The LBGR is often set at some other value –This is based on the decision makers’ choice and is not scientifically based –LBGR = AL - (10 - 20% of AL) How to Set the LBGR (cont.)

32. 32 of 48 n Use the Frequency Distribution method –The LBGR may be estimated based the Probability Distribution Function (PDF) –Place the Action Level on the mean of the PDF –Ask: “Does a substantial amount of contaminant concentration values exceed the Action Level?” –If yes, begin moving the PDF backwards along the x-axis towards zero concentration –Pause and ask again –When the answer is no, you have set the LBGR (e.g., where the mean of the PDF lies on the x-axis is now the LBGR) Use probability theory to show/prove this How to Set the LBGR (cont.)

33. 33 of 48 Show Probability Density Function Distribution Demonstration

34. 34 of 48 Methods for Evaluating the Attainment of Cleanup Standards - Volume 1: Soils and Solid Media EPA, February 1989 PB89-234959 How to set the LBGR  1 is a hypothetical “mean concentration where the site should be declared clean with a high probability”

35. 35 of 48 n “Normal” FD n “Skewed” FD n Computer Simulations: “Badly skewed” or Any FD Evaluate  and  errors to select n Using the LBGR to Estimate n

36. 36 of 48 Example 1 n Balancing the width of the gray region controls the cost of sampling and analysis n The closer the LBGR lies to the action level, typically the greater the number of samples, and thus the cost increases Mean = LBGRAL concentration Width of the Gray Region Mean = LBGR AL concentration Width of the Gray Region

37. 37 of 48 Width of the Gray Region n GR = Analytical + Sampling Error –Estimated based on past data and general knowledge n GR = 1/2 of the AL –For each COPC, calculate and set LBGR n GR = 20% of the AL –For each COPC, calculate and set LBGR n GR = PDF method –Use PDF for worst COPC to set LBGR

38. 38 of 48 Setting the GR Based on the (Analytical + Sampling Error) CS

39. 39 of 48 Setting the GR Based on Regulator Input (20% of the AL) CS

40. 40 of 48 Setting the GR Based on ½ of the Action Level CS

41. 41 of 48 Setting the GR Based on the PDF Method CS

42. 42 of 48 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region Assign probability values that reflect the decision maker’s tolerable limits for making an incorrect decision. At the action level At the other bound of the gray region At a point far below the action level At a point far above the action level Step 6- Specify Error Tolerances

43. 43 of 48 Site is dirty.Site is clean. 100 True State of Site Alternative Action Walk away from site.Clean up site. 75 Probability of deciding that the site is dirty 0.0 0.5 1.0 Action Level Lower Bound of Gray Region Typical Curve Null Hypothesis: Site is dirty.

44. 44 of 48 Decision Error Consequences CS

45. 45 of 48 Step 6 Summary n Define the two types of error –Incorrectly cleaning a dirty site or –Incorrectly cleaning a clean site n Evaluate severity of the incorrect decisions both below, above, and near the action level n Select the null hypothesis n Specify the error rates decision makers are willing to accept and provide rational for the rates

46. 46 of 48 n Establish a LBGR based on one of the four methods shown previously Step 6 Summary n Provide the basis for selecting the LBGR n Remember the closer the LBGR is to the action level, the more samples are needed n More samples can mean real-time measurements or traditional laboratory measurements n Assign probability limits on either side of the gray region

47. 47 of 48 Decision Error Tolerances Bounds of the Gray Region Assign probability limits on either side of the gray region Information INActions Information OUT From Previous Step To Next Step Decision Rules Step 5 Determine the variability of the environmental variables Choose the null hypothesis Identify the decision errors Specify the boundaries of the gray region Step 6- Specify Error Tolerances

48. 48 of 48 End of Module 8 Thank you