Presentation is loading. Please wait.

Presentation is loading. Please wait.

THE WILD, WILD, WET! SETAC Expert Advisory Panel Performance Evaluation and Data Interpretation.

Similar presentations


Presentation on theme: "THE WILD, WILD, WET! SETAC Expert Advisory Panel Performance Evaluation and Data Interpretation."— Presentation transcript:

1

2 THE WILD, WILD, WET! SETAC Expert Advisory Panel Performance Evaluation and Data Interpretation

3 THE PERFECT WORLD

4

5 FOCUS ON DATA ANALYSIS STEP 1: GRAPH THE DATASTEP 1: GRAPH THE DATA STEP 2: Analyze the data by EPA flowchartsSTEP 2: Analyze the data by EPA flowcharts STEP 3: DO THE RESULTS MAKE SENSE?STEP 3: DO THE RESULTS MAKE SENSE?

6

7 SOFTWARE PROGRAMS Many software packages/programs are availableMany software packages/programs are available DO NOT assume they follow the EPA recommended analysisDO NOT assume they follow the EPA recommended analysis DO verify the software by running example datasets from the methods manualsDO verify the software by running example datasets from the methods manuals

8 STATISTICAL AND BIOLOGICAL SIGNIFICANCE SETAC Expert Advisory Panel Performance Evaluation and Data Interpretation

9

10 TOXIC VS. NON-TOXIC How are data from a WET test used to make a decision of toxicity?How are data from a WET test used to make a decision of toxicity? –Two paths: Decision based on the observed result Decision based on standard effect

11 WHO DECIDES WHICH PATH? The Permit WritersThe Permit Writers –BOTH approaches are supported by the TSD and the methods manual

12 OBSERVED RESULT Data from the test are used to determine if toxicity is present by hypothesis testingData from the test are used to determine if toxicity is present by hypothesis testing –H O : Effluent is not toxic –H a : Effluent is toxic

13 STANDARD EFFECT A preselected level of effect is considered toxicA preselected level of effect is considered toxic – Acute test:50 % effect –Chronic test:25 % effect

14

15

16

17 THERE ARE INHERENT STRENGTHS AND WEAKNESSES TO BOTH APPROACHES

18 COMPONENTS WHICH IMPACT THE NOEC

19

20 WHAT BIOLOGICAL CONCLUSIONS CAN BE MADE FROM THE STATISTICAL ANALYSIS OF A SINGLE TOXICITY TEST? The biological impact was significant in the beaker

21 THE LESS THAN PERFECT WORLD

22

23 INTRA- AND INTER-TEST VARIABILITY SETAC Expert Advisory Panel Performance Evaluation and Data Interpretation

24 TYPES OF VARIABILITY Intra-test : among and between concentrationsIntra-test : among and between concentrations Inter-test: within one lab, same methodInter-test: within one lab, same method Inter-lab: between labs, same methodInter-lab: between labs, same method Method specific: within limits of methodMethod specific: within limits of method

25 INTRA-TEST VARIABILITY

26 INTRA-TEST VARIABILITY AND ENDPT. UNCERTAINTY

27

28

29 SOURCES OF INTRA-TEST VARIABILITY Genetic variabilityGenetic variability Organism handling and feedingOrganism handling and feeding Toxicity among and between treatmentsToxicity among and between treatments Non-homogeneous sample sourceNon-homogeneous sample source

30 SOURCES OF INTRA-TEST VARIABILITY Abiotic conditionsAbiotic conditions Dilution schemeDilution scheme Number of organisms/treatmentNumber of organisms/treatment Dilution water pathogensDilution water pathogens

31 SOURCES OF INTER-TEST VARIABILITY Intra-test sourcesIntra-test sources Analyst experience and practiceAnalyst experience and practice Organism age and healthOrganism age and health AcclimationAcclimation Dilution waterDilution water

32 SOURCES OF INTER-TEST VARIABILITY Sample qualitySample quality Test chamber characteristicsTest chamber characteristics

33 SOURCES OF INTER-TEST VARIABILITY Replicate volumeReplicate volume ProceduresProcedures

34 VARIABILITY AND POINT ESTIMATE UNCERTAINTY

35 HIGH VARIABILITY - LOW STATISTICAL POWER

36 LOW VARIABILITY - HIGH STATISTICAL POWER

37 ACTIONS TO REDUCE VARIABILITY Increase number of reps/treatmentIncrease number of reps/treatment QA programQA program Establish and follow strict proceduresEstablish and follow strict procedures Maximize analyst skillMaximize analyst skill Contract lab selectionContract lab selection Additional QA/QC criteriaAdditional QA/QC criteria

38 EXAMPLES OF ADDITIONAL QC TEST CRITERIA Region IX: upper MSD limitsRegion IX: upper MSD limits Washington: upper MSD limits, change in alphaWashington: upper MSD limits, change in alpha N. Carolina: limit control CVs, C. dubia “PSC”N. Carolina: limit control CVs, C. dubia “PSC” Region VI: limit control CV, increase # replicates,Region VI: limit control CV, increase # replicates, biological significance

39

40 SUSPICIOUS DATA AND OUTLIER DETECTION SETAC Expert Advisory Panel Performance Evaluation and Data Interpretation

41 CONCERNS Outliers make interpretation of WET data difficult byOutliers make interpretation of WET data difficult by –Increasing the variability in test responses –Biasing mean responses

42 IDENTIFYING OUTLIERS Graph raw data, means and residualsGraph raw data, means and residuals

43

44 IDENTIFYING OUTLIERS Formal statistical test - Chauvenet’s CriterionFormal statistical test - Chauvenet’s Criterion –Using the previous mysid data, the critical values are: Mean =.80,Std. Dev. = 0.302, n = 8 –Chauvenet’s Criterion Value = n/2 = 4 –Z-score = 2.054 (two-tailed probability of 4 %) –The calculations are: Equation 1)(Z-score)(Std. Dev.) = (2.054)(0.302) = 0.620 Mean  Equation 1 = 0.80  0.620 = 1.42 - 0.18 Outlier Range is >1.42 or <0.18 – A value of 0.2 is not an outlier.

45 CAN A CAUSE BE ASSIGNED TO THE OUTLIER(S) ? Review analyst’s daily observationsReview analyst’s daily observations Check water chemistry dataCheck water chemistry data Check data entryCheck data entry Check calculationsCheck calculations If cause can be assigned to outlier, then reanalyze data without outlierIf cause can be assigned to outlier, then reanalyze data without outlier

46 DETERMINE EFFECT ON TEST INTERPRETATION Keep all data unless cause is foundKeep all data unless cause is found Analyze data with and without suspect dataAnalyze data with and without suspect data Determine effect of suspect data on test interpretationDetermine effect of suspect data on test interpretation Results reported will depend on effect of outlier(s) on test interpretationResults reported will depend on effect of outlier(s) on test interpretation

47 REPORTING OF RESULTS Insignificant EffectInsignificant Effect –With Outlier IC 25 = 131 (96.9-158) ppb NOEC = 100 ppb % MSD = 28.1 % –Without Outlier IC 25 = 124 (93.6-152) ppb NOEC = 100 ppb % MSD = 20.9 % Report results with suspect data includedReport results with suspect data included Significant Effect –With Outlier IC 25 = 131 (96.9-158) ppb NOEC = 100 ppb % MSD = 28.1 % –Without Outlier IC 25 = 106 (83.8-126) ppb NOEC = 50 ppb % MSD = 12.2 % Report results from both analyses

48

49 HORMESIS AND NON-MONOTONIC CONCENTRATION RESPONSES SETAC Expert Advisory Panel Performance Evaluation and Data Interpretation

50 WHAT IS HORMESIS ? Calabrese and Baldwin, 1998Calabrese and Baldwin, 1998 General conceptGeneral concept OccurrenceOccurrence Typical CharacteristicsTypical Characteristics

51 TYPICAL TRAITS OF HORMESIS Hormetic - concentration rangeHormetic - concentration range Magnitude of hormetic stimulationMagnitude of hormetic stimulation Range from maximum stimulation to NOELRange from maximum stimulation to NOEL

52 WHY IS HORMESIS DIFFICULT TO DETECT IN TOXICITY TESTS? Inadequate concentration seriesInadequate concentration series Inadequate description of concentration - responseInadequate description of concentration - response Inadequate statistical powerInadequate statistical power Hormesis is not the causeHormesis is not the cause

53 EFFECTS OF NON- MONOTONIC DATA NOEC >LOEC Limited replicates (4)Limited replicates (4) Control/low concentration variabilityControl/low concentration variability High Statistical PowerHigh Statistical Power NOEC > LOECNOEC > LOEC

54 EFFECTS OF NON- MONOTONIC DATA HETEROGENEITY IN PROBIT ANALYSIS Limited replicates (5)Limited replicates (5) Control/low concentration variabilityControl/low concentration variability Significant chi-squareSignificant chi-square Inflated confidence intervalsInflated confidence intervals Reanalyze with non- parametric modelsReanalyze with non- parametric models

55 EFFECTS OF NON- MONOTONIC DATA SMOOTHING IN ICP ANALYSIS Smoothing is used in all non-parametric models.Smoothing is used in all non-parametric models. Smoothing procedure averages treatment responsesSmoothing procedure averages treatment responses Increases observed toxicityIncreases observed toxicity

56 REMEDIES FOR PROBLEMS ASSOCIATED WITH NON- MONOTONIC DATA Better concentration series selectionBetter concentration series selection Increase number of replicatesIncrease number of replicates % MSD limits (NOEC’s)% MSD limits (NOEC’s) Concentration-response curve criterionConcentration-response curve criterion Use of more robust parametric models Bailer and Oris, 1997 Kerr and Meador, 1996 Baird et al., 1996Use of more robust parametric models Bailer and Oris, 1997 Kerr and Meador, 1996 Baird et al., 1996

57

58 ANALYSIS OF MULTIPLE CONTROL TOXICITY TESTS SETAC Expert Advisory Panel Performance Evaluation and Data Interpretation

59 WHEN ARE MULTIPLE CONTROLS USED? To compare “standard” and “alternative methods.To compare “standard” and “alternative methods. –Food, dilution water, sterilization, organism source, etc.. –Control response is often not sufficient to determine differences. –Use of reference toxicant tests is recommended.

60 EFFECT OF KELP STORAGE ON SENSITIVITY TO COPPER

61 WHY ARE MULTIPLE CONTROLS USED? When manipulations are made to SOME of the test concentrations.When manipulations are made to SOME of the test concentrations. –Primarily used for salinity adjustments. –First rule, avoid if at all possible. –Treat extra control as most manipulated concentration. –Purpose is to determine if adjustments affected test results.

62 BRINE ADDITION IN MARINE TESTS

63 ANALYSIS OF TWO-CONTROL TOXICITY TESTS WHEN SOME CONCENTRATIONS WERE MANIPULATED

64

65 MOST SENSITIVE SPECIES DETERMINATION SETAC Expert Advisory Panel Performance Evaluation and Data Interpretation

66 WHAT IS A MOST SENSITIVE SPECIES SCREEN (MSSS)? A group of toxicity tests used to determine the species/method most capable of characterizing the toxicity associated with a dischargeA group of toxicity tests used to determine the species/method most capable of characterizing the toxicity associated with a discharge

67 COMMON CONSIDERATIONS Test species selectionTest species selection Frequency/timing of initial and subsequent screensFrequency/timing of initial and subsequent screens Changing effluent characteristicsChanging effluent characteristics Selection of data analysis methodsSelection of data analysis methods

68 MULTIPLE BIOLOGICAL ENDPOINT ANALYSIS Evaluate each biological endpointEvaluate each biological endpoint Use most “toxic” endpointUse most “toxic” endpoint

69 METHODS OF COMBINING MSSS RESULTS Proportion (X times out of Y screens)Proportion (X times out of Y screens) AveragingAveraging

70 STATISTICAL ENDPOINTS FOR EVALUATING MSSS NOEC’sNOEC’s Point-estimatesPoint-estimates Effect at critical concentration (ECC)Effect at critical concentration (ECC) Probability of effect at critical concentration (pECC)Probability of effect at critical concentration (pECC)

71 NOEC’S Experimental Question Which method/species is most likely to identify a change from control response?Experimental Question Which method/species is most likely to identify a change from control response?

72 ADVANTAGES OF NOEC’S Common endpointCommon endpoint Integrates effect and intra-test variabilityIntegrates effect and intra-test variability

73 DISADVANTAGES OF NOEC’S Can not separate biological effect and statistical sensitivityCan not separate biological effect and statistical sensitivity Can not averageCan not average NOEC’s may not be environmentally relevantNOEC’s may not be environmentally relevant

74 POINT-ESTIMATES Experimental Question Which method/species shows the specified effect at the lowest concentration?Experimental Question Which method/species shows the specified effect at the lowest concentration?

75 ADVANTAGES OF POINT-ESTIMATES Evaluates a common effect levelEvaluates a common effect level Utilizes the entire concentration- response curve (parametric models)Utilizes the entire concentration- response curve (parametric models) Can use proportion or average analysisCan use proportion or average analysis

76 DISADVANTAGES OF POINT-ESTIMATES Effect level selectionEffect level selection Concentration- response requiredConcentration- response required SmoothingSmoothing No consideration of endpoint precisionNo consideration of endpoint precision EC values may not be environmentally relevantEC values may not be environmentally relevant

77 EFFECT AT CRITICAL CONCENTRATION (ECC) Experimental Question Which method/species shows the greatest effect at the concentration of environmental concern?Experimental Question Which method/species shows the greatest effect at the concentration of environmental concern?

78 ADVANTAGES OF ECC Can use proportion or average analysisCan use proportion or average analysis Environmental relevanceEnvironmental relevance No concentration- response requiredNo concentration- response required

79 DISADVANTAGES OF ECC Does not consider certainty of response estimateDoes not consider certainty of response estimate Ability to obtain effect estimate at IWC from point- estimate modelsAbility to obtain effect estimate at IWC from point- estimate models

80 PROBABILITY OF ECC (pECC) Experimental Question At the concentration of environmental concern, which method/species had the greatest effect at the lower 95 % confidence limit?Experimental Question At the concentration of environmental concern, which method/species had the greatest effect at the lower 95 % confidence limit?

81 ADVANTAGES OF pECC Considers precision of response estimateConsiders precision of response estimate Can use proportion or average analysisCan use proportion or average analysis Environmental relevanceEnvironmental relevance No concentration- response requiredNo concentration- response required

82 DISADVANTAGES OF pECC Zero replicate varianceZero replicate variance Boot-strappingBoot-strapping Obtaining 95% confidence intervals at IWCObtaining 95% confidence intervals at IWC

83 SUMMARY Discuss the MSSS procedure in detail during permit developmentDiscuss the MSSS procedure in detail during permit development Select variety of organism typesSelect variety of organism types Initially test for trends in toxicityInitially test for trends in toxicity Continue periodic screeningContinue periodic screening Select type of statistical analysis carefullySelect type of statistical analysis carefully Make sure that statistical analysis and the raw results “make sense”Make sure that statistical analysis and the raw results “make sense”

84

85 AGE-RELATED SENSITIVITY OF FISH IN ACUTE WET TESTS SETAC Expert Advisory Panel Performance Evaluation and Data Interpretation

86 REVISIONS TO FISH AGES IN EPA ACUTE TEST MANUALS From: 1-90 days old in the 3rd edition of the acute manual (1985; EPA/600/4- 85/013)From: 1-90 days old in the 3rd edition of the acute manual (1985; EPA/600/4- 85/013) To: 1-14 days old (or 9-14 days old for silversides) in the 4th edition of the acute manual (1993; EPA/600/4- 90/027F)To: 1-14 days old (or 9-14 days old for silversides) in the 4th edition of the acute manual (1993; EPA/600/4- 90/027F)

87 COMMONLY USED TEST SPECIES Fathead minnowsFathead minnows Sheepshead minnowsSheepshead minnows Silversides (inland, atlantic, and tidewater)Silversides (inland, atlantic, and tidewater)

88 RATIONALE Younger life stage is generally more sensitive than older life stageYounger life stage is generally more sensitive than older life stage Reduction in range of acceptable ages from 1-90 to 1-14 days will reduce variabilityReduction in range of acceptable ages from 1-90 to 1-14 days will reduce variability

89 CONCERN Use of younger fish in NPDES testing may show an increase in apparent toxicity, without any changes in effluent conditionsUse of younger fish in NPDES testing may show an increase in apparent toxicity, without any changes in effluent conditions

90 COMMON QUESTIONS Are <14-day old fish more sensitive than <90-day old fish to toxicants?Are <14-day old fish more sensitive than <90-day old fish to toxicants? Does the use of <14-day old fish reduce intertest variability when compared to <90 day-old fish?Does the use of <14-day old fish reduce intertest variability when compared to <90 day-old fish? How does the sensitivity and precision vary within the 1 to 14 day old age range?How does the sensitivity and precision vary within the 1 to 14 day old age range?

91 SENSITIVITY OF 14, 30, AND 90 DAY-OLD FATHEAD MINNOWS

92 INTER-TEST PRECISION OF 14, 30, AND 90-Day Old FATHEAD MINNOWS

93 SENSITIVITY OF 1-14 DAY-OLD FATHEAD MINNOWS

94 INTER-TEST PRECISION OF 1-14 DAY-OLD FATHEAD MINNOWS

95 SUMMARY 14-day old fathead minnow larvae are more sensitive to copper & ammonia than 90 day- old fish.14-day old fathead minnow larvae are more sensitive to copper & ammonia than 90 day- old fish. The inter-test precision of 90 day old fish is equal or better than 14 day-old fish for copper & ammonia.The inter-test precision of 90 day old fish is equal or better than 14 day-old fish for copper & ammonia.

96 SUMMARY (CONTINUED) Within the 1-14 day age range, 1 day- old larvae are less sensitive to several toxicants.Within the 1-14 day age range, 1 day- old larvae are less sensitive to several toxicants. The sensitivity of these toxicants becomes constant after 4-7 days of age.The sensitivity of these toxicants becomes constant after 4-7 days of age. Maximum inter-test precision for these toxicants is observed when the age range is limited to 7 -14 day old larvae.Maximum inter-test precision for these toxicants is observed when the age range is limited to 7 -14 day old larvae.

97

98 THE CHRONIC TEST GROWTH ENDPOINT SETAC Expert Advisory Panel Performance Evaluation and Data Interpretation

99 CHANGE IN GROWTH ENDPOINT CALCULATION Pre-Nov., 1995 Approach Growth = D.W. surviving organisms # surviving organisms # surviving organisms Post-Nov., 1995 Approach Growth = D.W. surviving organisms # initial organisms # initial organisms

100 EFFECT ON MEAN TREATMENT RESPONSES

101

102

103 EFFECTS ON HYPOTHESIS TEST ENDPOINTS

104

105 EFFECTS ON POINT ESTIMATE ENDPOINTS

106

107 NOEC/IC25 RELATIONSHIP

108 IMPACT ON TEST INTERPRETATION Hypothesis Test Results - most cases show little change, but not alwaysHypothesis Test Results - most cases show little change, but not always Point Estimate Results - usually increases predicted toxicityPoint Estimate Results - usually increases predicted toxicity

109 ISSUES RELATED TO CHANGE IN APPROACH Test growth or biomass?Test growth or biomass? Accurate representation of growth?Accurate representation of growth? Correlation between new results and instream responses?Correlation between new results and instream responses?

110 ISSUES RELATED TO CHANGE IN APPROACH Conflict between new results and unchanged effluent quality?Conflict between new results and unchanged effluent quality? Effect on reference toxicant control chartsEffect on reference toxicant control charts Relationship between NOEC and IC25Relationship between NOEC and IC25

111

112 ANOMALOUS PATTERNS OF SURVIVAL IN SHORT-TERM CHRONIC WET TESTS WITH FATHEAD MINNOWS SETAC Expert Advisory Panel Performance Evaluation and Data Interpretation

113 WHERE HAS THE “PROBLEM” BEEN SHOWN? Effluent toxicity tests where receiving water is used as test dilution water (diluent)Effluent toxicity tests where receiving water is used as test dilution water (diluent) “Once through” cooling waters, where receiving water is used for cooling and then discharged“Once through” cooling waters, where receiving water is used for cooling and then discharged Ambient toxicity testsAmbient toxicity tests

114 COMMON CHARACTERISTICS Observed in fathead minnow short-term chronic WET tests, but not in acute WET testsObserved in fathead minnow short-term chronic WET tests, but not in acute WET tests Not observed in concurrently performed ceriodaphnia short-term chronic testsNot observed in concurrently performed ceriodaphnia short-term chronic tests High variability in survival between replicates within a concentrationHigh variability in survival between replicates within a concentration Concentration-effects relationship is often non- monotonicConcentration-effects relationship is often non- monotonic Mortality is often first seen on day 4 of the test in RW controlsMortality is often first seen on day 4 of the test in RW controls

115 INDUSTRIAL EFFLUENT #1 SURVIVAL ENDPOINT (7 D)

116 INDUSTRIAL EFFLUENT #1 GROWTH ENDPOINT

117 INDUSTRIAL EFFLUENT #1 DAILY SURVIVAL (%) Majority of mortality occurred on Test Days 3 and 4 Final survival for four replicates in receiving water was 70, 30, 40, and 70%

118 INDUSTRIAL EFFLUENT #2 SURVIVAL ENDPOINT (7 D)

119 INDUSTRIAL EFFLUENT #2 GROWTH ENDPOINT

120 WISCONSIN DNR PROGRAM For chronic WET tests performed between 1988-1998, 26% showed unacceptable receiving water control survivalFor chronic WET tests performed between 1988-1998, 26% showed unacceptable receiving water control survival Only 2.9% of lab controls failed to meet survival criterionOnly 2.9% of lab controls failed to meet survival criterion Effects are not seasonalEffects are not seasonal

121 MICROBIOLOGICAL EXAMINATION OF FISH Aeromonas hydrophilaAeromonas hydrophila Flexibacter aurantic and F. columnarisFlexibacter aurantic and F. columnaris Flavobacterium sp.Flavobacterium sp. Saprolegnia sp.Saprolegnia sp.

122 WHAT WORKS TO ELIMINATE THE PROBLEM? Filtration (0.2 µ; some success with 0.45µ)Filtration (0.2 µ; some success with 0.45µ) AutoclavingAutoclaving UV disinfectionUV disinfection HeatingHeating AntibioticsAntibiotics

123 HOW ARE PEOPLE HANDLING THIS PROBLEM? Use laboratory water as test dilution and control water for all WET testingUse laboratory water as test dilution and control water for all WET testing Use laboratory water, after receiving water problems have been shownUse laboratory water, after receiving water problems have been shown Perform concurrent testing in both laboratory water and receiving waterPerform concurrent testing in both laboratory water and receiving water

124 HOW ARE PEOPLE HANDLING THE PROBLEM? (CONTINUED) Accept “problem” tests for compliance, but don’t use them to determine “pass/fail”Accept “problem” tests for compliance, but don’t use them to determine “pass/fail” Manipulate receiving water sample, before use in testManipulate receiving water sample, before use in test

125 SUMMARY Interpretation of fathead minnow short- term chronic tests may be complicated by presence of a “biological agent”Interpretation of fathead minnow short- term chronic tests may be complicated by presence of a “biological agent” This problem has been observed in many areas, but it is not known how widespread is its occurrenceThis problem has been observed in many areas, but it is not known how widespread is its occurrence Phenomenon is being studied by different investigators and may result in recommendations for test method modificationsPhenomenon is being studied by different investigators and may result in recommendations for test method modifications

126


Download ppt "THE WILD, WILD, WET! SETAC Expert Advisory Panel Performance Evaluation and Data Interpretation."

Similar presentations


Ads by Google