Steps 3 & 4: Evaluating types of evidence for the Truckee River case study

2 Define the Case List Candidate Causes Evaluate Data from Elsewhere Identify Probable Cause Detect or Suspect Biological Impairment As Necessary: Acquire Data and Iterate Process Identify and Apportion Sources Management Action: Eliminate or Control Sources, Monitor Results Biological Condition Restored or Protected Decision-maker and Stakeholder Involvement Stressor Identification Step 3: Evaluate Data from the Case Step 4: Evaluate Data from Elsewhere

3 Types of evidence using data from the case Spatial/temporal co-occurrence Evidence of exposure or biological mechanism Causal pathway Stressor-response relationships from the field Manipulation of exposure Laboratory tests of site media Temporal sequence Verified predictions Symptoms Types of evidence using data from elsewhere Stressor-response relationships from other field studies Stressor-response relationships from laboratory studies Stressor-response relationships from ecological simulation models Mechanistically plausible cause Manipulation of exposure at other sites Analogous stressors Use all available types of evidence to make an inferential assessment italics indicates commonly available types of evidence

4 Basic analysis strategy Develop as many types of evidence, for as many candidate causes, as you can – you won’t have all types of evidence, for all candidate causes – most effective when you can compare results across candidate causes Work through one type of evidence, then set it aside – avoid cognitive overload Show your work – make your process transparent & reproducible – make use of appendices

5 Let’s begin by figuring out what types of evidence we have for the Truckee…

6 General vs. specific causation General – Does C cause E? – Does smoking cause lung cancer? – Does increased water temperature reduce bull trout abundance in rivers? Specific – Did C cause E? – Did smoking cause lung cancer in Ronald Fisher? – Did increased water temperature reduce bull trout abundance in my stream?

7 Specific causation: using data from the case

8 Spatial/temporal co-occurrence SUPPORTS WEAKENS Want paired measurements of proximate stressors & biological impairments, at locations where impairments are & are not observed.

9 Causal pathway Want paired measurements of other steps in causal pathway & biological impairments, at locations where impairments are & are not observed.

10 Stressor-response relationships from the field Want paired measurements of proximate stressors (or other steps in causal pathway) & biological impairments, at varying levels of exposure.

11 Other types of evidence using data from case TYPE OF EVIDENCESUPPORTING EVIDENCE Manipulation of exposure Impairment improves after stressor is removed Laboratory tests of site media Exposure to site media in lab tests results in effects similar to impairment Evidence of exposure or biological mechanism Measurements of biota (e.g., biomarkers, tissue residues) show proposed mechanism of exposure has occurred Verified predictions Predictions based on stressor’s mode of action are made & confirmed at site Temporal sequenceExposure to stressor precedes impairment SymptomsOnly one stressor supports observed symptom

12 What types of evidence do we have, using data from the case?

13 General causation: using data from elsewhere?

Stressor-response relationships from other field studies

15 Stressor-response relationships from the lab

16 Other types of evidence using data from elsewhere TYPE OF EVIDENCESUPPORTING EVIDENCE Stressor-response relationships from ecological simulation models Stressor is at levels associated with impairment in mathematical models simulating ecological processes Manipulation of exposure Impairment improves after stressor is removed at another site Mechanistically plausible cause Relationship between stressor & impairment is consistent with current scientific knowledge Analogous stressors Stressor is structurally similar to other stressors known to cause impairment Verified predictions Predictions based on stressor’s mode of action are made & confirmed at other sites

17 What types of evidence do we have, using data from elsewhere?

18 Now that we know what data we have, how do we analyze it?

19 Spatial co-occurrence Do your impairment and your stressor co-occur in space? To Do: 1.Load relevant data file 2.Merge files 3.Make boxplots for each candidate cause Select ‘reference’ and impaired sites 4.Fill in worksheet

20 Causal Pathway Does your data support the steps in the causal path between the stressor and the impairment? To Do: 1.Return to the conceptual diagram 2.Identify the steps in the causal pathway 3.Construct table to show whether data supports the steps between the stressor and the impairment 4.Fill in worksheet

21 Verified Prediction - Traits Do data support predictions based on stressor’s mode of action? To Do: 1.Load relevant data file 2.Merge files 3.Make boxplot Select ‘reference’ and impaired sites 4.Fill in worksheet

22 Verified Prediction - PECBO Do data support predictions based on stressor’s mode of action? To Do: 1.Load relevant data file 2.Merge files 3.Run PECBO 4.Load PECBO results file into CADStat 5.Merge files 6.Make boxplot 1.Sed 2.STRMTEMP 7.Fill in worksheet

23 Stressor-response from elsewhere Does impairment decrease as exposure to the stressor decreases (or increases as exposure increases)? To Do: 1.Listen and ask lots of questions 2.Fill in the worksheet

24 Randomized, controlled experiments Key elements: Replication: use of multiple test units (e.g. tanks, sites) Controls: differ only by absence of the treatment Randomization: random assignment of test units to “control” or “treated” status Statistical analysis: estimate treatment effect (causal) The scientific standard for establishing cause and effect

25 Observational studies Key elements: Replication: collect data from multiple test units Controls: ? Randomization: ? Statistical analysis: identify associations among variables of interest (non-causal) Often the only option for large-scale field studies None

26 Trade offs: control vs. realism, scale Lab Experiment Field Experiment Observational Study control realism, scale

27 Biomonitoring = Observational Issues for causal analysis: Estimates of stressor effects are confounded by covarying factors Analyst can’t randomly assign treatments (stressors) to sites *Reference sites are not experimental controls

28 Analogous to clinical trials Does smoking cause lung cancer? Estimates of stressor effects are confounded by covarying factors Analyst can’t randomly assign treatments (stressors) to subjects * Non-smokers without lung cancer are not experimental controls

29 Example using western EMAP* Using propensity scores to infer cause-effect relationships in observational data –Analysis and slides by Lester Yuan (USEPA), Amina Pollard (USEPA), and Daren Carlisle (USGS) –Original presentation given at North American Benthological Society conference, May 2008 *EPA Environmental Monitoring and Assessment Program (EMAP)

30 EMAP-West Study Area Measurements Collected: Macroinvertebrates Substrate composition (SED) Stream temperature (STRMTEMP) N = 838 Data collected by the EPA Environmental Monitoring and Assessment Program (EMAP)

31 Total N vs. total taxon richness Data from EMAP Western Pilot SLOPE = -16.5

32 Total N covaries with many other factors

33 Multiple linear regression Include covariates in the regression model to control for their effect. SLOPE = -9.4 Regression model includes: %agriculture, %urban, grazing intensity, %sands/fines, stream temperature, and log conductivity. SLOPE = Correlation of Total Richness and Total N (ug/L)

34 Potential issues with multiple regression Must assume that linear relationships are appropriate for all covariates. Regression model may extrapolate. Inclusion of certain variables may “mask” true effect: –e.g., part of the effect of agriculture may be attributed to total N

35 Alternate approach: Stratify dataset r = -0.01r = 0.15r = 0.27 r = 0.64

36 Model richness vs. total N within strata How do we simultaneously stratify on many different covariates? SLOPE = -10.7SLOPE = -12.3SLOPE = -9.7

37 Propensity Score Matching Method developed in epidemiology to retroactively control for confounding effects in observational studies Sometimes called a quasi-experiment Intuitively: 1.Model the magnitude of treatment (e.g. nutrient concentration) as a function of the covariates. The predicted magnitude of treatment at each site is its propensity score. 2.Stratify the total set of observations by the propensity scores (i.e., group sites with similar scores). Six strata are typically used. 3.Within each stratum, sites having different treatment levels (e.g. high vs. low nutrients) may be considered to have been “randomly assigned” to those treatment levels, because covariates have effectively been controlled by propensity score matching of “treated” and “control” sites.

38 Propensity Score Model Total N = f(percent agriculture, percent urban, grazing intensity, percent sand/fines, stream temperature, log conductivity, elevation, log catchment area, canopy cover, sampling day)

39 Define 6 strata based on propensity scores

40 Covariate values within strata Grazing intensity Percent agriculture

41 Stratification by propensity score controls covariance of all modeled variables Original rMin rMax r Percent agriculture Percent urban Grazing intensity Percent sand/fines Stream temperature log conductivity Elevation log catchment area Canopy cover Sampling day After stratification

42 Total N vs. total taxon richness Data from EMAP Western Pilot SLOPE = -16.5

43 Total N vs. total richness: Stratified SLOPE = -3.3 (n.s.) SLOPE = -10.0*** SLOPE = -7.1* SLOPE = -10.5***SLOPE = -8.1***