Presentation on theme: "Western Ecology Division Web Page:"— Presentation transcript:
1 Western Ecology Division Web Page: http://www.epa.gov/NHEERL/ARM Statistical Perspective on the Design and Analysis of Natural Resource Monitoring ProgramsAnthony (Tony) R. OlsenUSEPA NHEERLWestern Ecology DivisionCorvallis, Oregon(541)Web Page:
2 National Water Quality Monitoring Council: Monitoring Framework Applies to all natural resource monitoringMonitoring pieces must be designed and implemented to fit togetherView as information systemNational monitoring requires consistent frameworkReference: Water Resources IMPACT, September 2003 issueIt is not perfect but it is good framework. May lack clarity on which activities belong to which cog. Note that the cogs were consciously selected to not have sharp boundaries – reflection of reality of designing a monitoring program.
3 Impact Article Contributors Data CollectionFranceska WildeHerbert J. BrassJerry DiamondData ManagementKaren S. KlimaKenneth J. LanfearEllen McCarronAssess and InterpretDennis R. HelselLindsay M. GriffithReport ResultsMary AmbroseAbby MarkowitzCharles JobFramework OverviewCharles A. PetersRobert C. WardThe Three C’sAbby MarkowitzLinda T. GreenJames LaineMonitoring ObjectivesCharles S. SpoonerGail E. MallardMonitoring DesignTony OlsenDale M. RobertsonSeveral of the authors are also members of the Monitoring Council. The framework is a product of the council; the IMPACT articles were based on work by the council.
4 Monitoring Program Weaknesses Monitoring results are not directly tied to management decision making (monitoring objectives)Results are not timely nor communicated to key audiences (convey results)Objectives for monitoring are not clearly, precisely stated and understood (monitoring objectives)Monitoring program not viewed/implemented as an information system (data management, overall)Monitoring measurement protocols, survey design, and statistical analysis become scientifically out-of-date (field/lab methods, monitoring design, data analysis/assessment)Why monitor if you are not making a difference.Failure may be reflected in monitoring program not being adequately funded.Monitoring programs must continually improve and so must plan for change.Note that only cog not represented is field operations!Individuals designing a monitoring program typically spend more time on the details of field/lab methods, monitoring design, and data assessment. These areas are not one of the major weaknesses – unless they are not updated to keep current.First three reasons are there because they are HARD and don’t involve just science.Now will address comments on statistical perspective cog by cog.
5 Communicate, Coordinate, Collaborate Communication: process of conveying information; can be one way or an exchange of thoughts, messages, or ideasCoordination: process in which two or more participants link, harmonize or synchronize interaction and activitiesCollaboration: process in which two or more participants work collectively to deal with issues that they cannot solve individually; partnerships, alliances, teamsTwo aspects: one internal to a monitoring program and one external.Internal: multiple disciplines are essential in the development of a monitoring program. Learning how to communicate across these disciplines is not always easy nor straightforward. Terminology differs, cultures differ.Disciplines: Decision-makers, management, natural resource specialists, statistical specialists, field operation specialists, laboratory specialists, data management specialists, communication specialistsIn some cases, a monitoring program must serve multiple “masters” or decision-makers. Need to make sure the Three C’s are effective with all of themRarely does a monitoring program exist in a vacuum. Other organizations may monitor the same resource: federal, state, local groups. Cost-effective monitoring across all these organizations places a premium on the Three C’s.A statistical perspective is part of each of the six cogs – consequently a statistical specialist has much to contribute to effective communication, coordination, and collaboration
6 Statistical perspective is key ConveyResultsandfindingsDevelopmonitoringobjectivesDesignmonitoringprogramKish (1965): “The survey objectives should determine the sample design; but the determination is actually a two-way process…”Initially objectives are stated in common sense statements – challenge is to transform them into quantitative questions that can be conveyed precisely to intended audience.Statistical perspective is keyKnow whether a monitoring design can answer the questionKnow when the question is not precise enough – multiple interpretationsUseful to think about what tables and graphics will be used to convey results and findings in a report. Do this assuming that can visit all sites (ie ignore the issue of site selection). Also useful to think about what the monitoring program would present to decision-makers when they only have minutes to make the presentation. Helps focus on high priority objectives. I recognize that data from a monitoring program will be summarized and used in many ways that was not anticipated – that is fine. What don’t want to do is lose sight of what key information is required by decision-makers and public.Developing monitoring objectives must be done in the context of institutional constraints. That is a major reason it is difficult. Major institutional constraint is funding. It makes a difference if are designing for a $1M, $10M, or $100M national monitoring program.
7 Identify Monitoring Objectives Objectives determine the monitoring design (yet monitoring design constrains objectives that can be met)Usual to have multiple objectivesPrecise statements are requiredObjectives must be prioritizedObjectives compete for samplesStatistical perspective helps identifyTarget populationSubpopulations that require estimatesElements of target populationPotential sample framesVariables to be measuredImpact of precision requiredAbstract concepts from survey design and experimental design are useful when discussing monitoring objectives.
8 Example: From Question to Objective What is the quality of waters in the United States?What is the quality of streams with flowing water during summer in the U.S.?What is the biological quality of streams with flowing water during summer in the U.S.?How many km of streams with flowing water during the summer are impaired, non-impaired, or marginally-impaired within the U.S.?How is impairment determined?What is meant by summer?Are constructed channels, canals, effluent-dominated streams included?Want the objective to require a quantitative answer that can be used by management.The process of making the objectives quantitative and specific identifies the target population is identified, the subpopulations of interest, the variables to be measured, and the types of statistical summaries required to convey results
9 Key components of monitoring design DevelopmonitoringobjectivesDesignmonitoringprogramCollectfield andlab dataKey components of monitoring designWhat will be monitored? (target population)What will be measured? (variables or indicators)When and how frequently will the measurements be taken? (temporal design)Where will the measurements be taken? (site selection)Statistical perspectiveSample frame and target populationSurvey design“What will be measured” is not considered in this presentation.
10 What is a Target Population? Target population denotes the ecological resource for which information is wantedRequires a clear, precise definitionMust be understandable to usersField crews must be able to determine if a particular site is in the target populationMore difficult to define than most expect.Includes definition of what the elements are that make up the target populationAll forests within the United States is a target population for the FIA monitoring program. How do define a forest? Does it include urban forested areas? Transition zones exist between what is clearly forested land and clearly rangeland. When are no longer in forest?When is a lake, a lake of interest? National Lake Fish Tissue survey gave what we thought was a very precise definition of a lake. One lake waterbody selected in the sample was a tertiary treatment pond associated with sewage treatment. It met all requirements of being a lake, including having a permanent fish population. Pond was fenced off from public access. Monitoring objectives was to estimate concentrations of contaminants in fish tissue. Results would be used not only for human consumption but also wildlife consumption perspective. Since it was known that wildlife ate fish from the pond, decision was made that it was part of the target population. In the course of discussion, also became known that humans also climbed the fence to catch fish!
11 Target Population, Sample Frame, Sampled Population We Live in an Imperfect World…Target population denotes the ecological resource about which estimates are neededDefined conceptually using written textMust be sufficiently specific so that it is clear if an aquatic resource is included or not.Must define what are the elements of the target population. Elements may be any location in an estuary, a lake, any point on a stream network, or a 6th field Hydrologic unitSampling Frame is a physical representation of the target populationIt consists of sample units that are potential members of the sampleExtent (size) of the frame is obtained by summationSample Frames almost always are not exact representations of the target populationSample Frame may not include some Target Population elements: undercoverageSample Frame may contain non-target elements, e.g., mis-identified sample units: OvercoverageA subset of the Sample Frame sample units are selected for sampling: the sampleProbability survey designs used to select the subsetOne design: Generalized Random Tessellation Stratified Designs - GRTSMay include stratification, unequal probability selection, panels for surveys over timeSample Frame overcoverage and sample site field access problems addressed by including an OversampleSampling Units are the Sites selected for samplingSampled Population is a conceptual population that is a subset of intersection the Target Population and the Sample FrameIt excludes portion of the Target Population within the Sample Frame that could not be sampled (conceptually) due to access problems, lost samples, or other reasons a sample could not be collectedIt doesn't include part of the Sample Frame that is determined to not be elements of the Target PopulationPopulation Estimates are based on All Sites Evaluated for potential field samplingSite Evaluation and Field Sampling Categorizing each Sample Site is critical informationTarget Sampled -- Site Information CollectedLandowner Denial -- Some landowners deny field crew accessPhysical Barrier -- Site can not physically be reached within protocols or for safety reasonsTarget Not-Sampled -- Sample lost, field season ended before site could be sampled, and many other reasonsNon-Target -- Site not element of target populationPopulation Extent estimates made for each Site CategoryProvides estimate of the Target Population extent if it is not knownProvides estimate of the Sample Frame overcoverage extent, i.e., how much too large is FrameProvides estimate of percent of Target Population that is expected to have landowners deny accessPopulation Status estimates based on Target Sampled Sites (e.g., IBI score, non- Impairment)Potential Corrections and AssumptionsNon-Target Site Information can be used to determine if Sample Frame should be improved (mis-identified units, extent)Estimates based on Target Sampled sites apply to the Sampled Population -- with no additional assumptionsEstimates based on Target Sampled sites can apply to the portion of Target Population within the Sample Frame ONLY IF assume that the Access Denied, Target Not-Sampled, etc., sites occurred randomly and independently of site characteristicsEstimates for Target Population NOT ONLY require assumptions above BUT ALSO that portions of Target Population that are not included in the Sample Frame have same characteristics as the Sampled PopulationIdeally, cyan, yellow, gray squares would overlap completely
12 Basic Spatial Survey Designs Simple Random SampleSystematic SampleRegular gridRegular spacing on linear resourceSpatially Balanced SampleCombination of simple random and systematic characteristicsGuarantees all possible samples are distributed across the resource (target population)Generalized Random Tessellation Stratified (GRTS) design
13 Generalized Random Tessellation Stratified (GRTS) Survey Designs Probability sample producing design-based estimators and variance estimatorsGive another option to simple random sample and systematic sample designsSimple random samples tend to “clump”Systematic samples difficult to implement for aquatic resources and do not have design-based variance estimatorEmphasize spatial-balanceEvery replication of the sample exhibits a spatial density pattern that closely mimics the spatial density pattern of the resourceDeveloped to meet needs of monitoring programs. This is an example of how long-term associations between statisticians and monitoring professionals can lead to new developments and improvements in cost-effectiveness and scientific-defensibility of monitoring programs.
15 Why aren’t Basic Designs Sufficient? Monitoring objectives may include requirements that basic designs can’t address efficientlyEstimates for particular subpopulations requires greater sampling effortAdministrative restrictions and operational costsNatural resource in study region makes basic designs inefficientResource may be known to be restricted to particular subregionsComplex designs may be more cost-effective
16 Example of a spatially-balanced design with unequal probability of selection based on lake area
17 Example of a spatially-balanced survey design with (1) unequal probability selection based on overlapping subpopulations that were of interest, and (2) nested subsampling of indicators related to increased cost to acquire some indicators.
18 National Wadeable Stream Assessment 2004 Spatially balanced survey design for streams with (1) stratification by states, (2) unequal probability selection based on Omernik ecoregions and stream Strahler order, and (3) intensive study regions for specific subpopulations.
19 Survey Design & Response Design Survey design is process of selecting sites at which a response will be determinedWhich sites will be visited (spatial component)Which monitoring season will sites be visited (temporal component, panel design)Response design is process of obtaining a response at a site:When site is to be visited within a monitoring seasonA single index period visit during a monitoring seasonMultiple visits during monitoring season: e.g. monthly, quarterlyField plot designProcess of going from basic field measurements to indicatorsMonitoring design can be thought of in terms of a survey design and a response design. The split is somewhat artificial but is useful in the design process.
20 Statistical perspective Collectfield andlab dataDesignmonitoringprogramCompileand managedataComponentsField methods (response design)Laboratory methodsMeasurement quality objectivesQuality assurance & quality controlLogistical plan and gaining site accessStatistical perspectiveExperimental designs to determine cost-effective and scientifically-defensible response designsStatistical quality controlMethods for minimizing non-responseMany statistical aspects involved in obtaining results that require laboratory analyses. Chemical laboratory operations have a long history in the use of statistics, including inter laboratory comparisons. Biological sample counting, and other physical sample operations, laboratories also incorporate statistics into their operations. The success of these operations is directly related to data quality and data comparability for the monitoring program for these samples.Like to give a few examples concerning response designs and then one example on non-response.
21 Response Design - Fish Species Richness (% of Maximum) Stream Length 1020304050607080100Stream Length(Channel Width Units)Species Richness(% of Maximum)1-pass samplingSpread effort throughout reachGet “common” species in approx. relative abundanceEMAP conducted species-area studies to determine the length of stream reach required to be sampled to capture fish community information.Not cost-effective (or even possible) to get 100% detection of all species in a large-scale stream monitoring program. Must address the question of what can be done that is scientifically-defensible AND still provides the information required by decision-makers.
22 Response Design: Benthos and Periphyton CKJIHGFEDFLOWDistance between transects=4 times mean wetted width at X-siteX-siteTotal reach length=40 times mean wetted width at X-site (minimum=150 m)RLSAMPLING POINTSL=Left C=Center R=RightFirst point (transect B) determined at randomSubsequent points assigned in order L, C, REMAP conducted research studies on the field plot design for collecting biological and physical habitat information to determine if the signal to noise ratio (variation across plots divided by repeat measurement variation) was strong enough.One basic principle: many small samples that are composited are better than a single large sample. Provides better coverage of variation in habitats.
23 US Forest Service Forest Inventory and Analysis (FIA) Plot Design FIA and FHM conducted studies on cost-efficiency of the field plot design. Although might be desireable to count/measure all vegetation within large annular plot, it is not cost-effective to do so.
24 Minimizing Non-Response: Prairie Potholes Landowner contact procedureObtain owner list from USDA ASCS local officeCover letter explained study, random selection, measurements, walking access only, timing/duration visit, offer to honor special owner conditionsConsent formMap of identifying wetland to be visitedTelephone contact 2-4 weeks after letter – list of FAQs and answers provided to personnelSecond letter 5-6 weeks after initial letterAccess rates: private land 42%25% of access approvals required multiple contactsFrom Lesser et al (2001)Sampling prairie pothole wetlands is a contentious issue since it typically involves gaining access to them through agricultural fields.In aquatic surveys of streams, EMAP has found that how landowners are approached also makes a considerable difference.Survey researchers have considerable experience in how to contact survey participants and increase the probability of their responding. Natural resource monitoring is beginning to take advantage of this knowledge.
25 Components: compile and manage data Collectfield andlab dataAssessand interpretdataCompileand managedataComponents: compile and manage dataData entryDatabase developmentMetadataData preservationData discovery and retrievalStatistical perspectiveStatistical QA checking of dataAccess to auxiliary data used in statistical analysesInfluence retrieval and database designImportance of preserving design informationChecking data can be a time-consuming and complex process.One type of checking is completed for each data item individually – typically involved checking that data value is legitimate response through comparison with acceptable responses or acceptable range.Second type of checking is across sites for each data item – are outliers present is the question relative to rest of the sites.Third type of checking is multivariate across sites and selected data items - % landcover can’t add up to more than 100%, etc.Auxiliary data:1. summary of sample frame characteristics may be necessary in weight adjustment process.2. May have known values that are used to constrain estimates – Total land area by county, etc.3. Remote sensing information that will be used in regression estimators
26 Examples STORET modified to include survey design information Which sites are part of the survey designStratification, weights, cluster variablesUSGS NWIS and NWISWebNWIS focus on input/site specific (typically time focus)NWISWeb focus on retrieval (typically spatial focus)National Resource Inventory’s analysis databaseStatistical imputation for missing dataStatistical creation of pseudo pointsIncorporate known informationLink across years for consistencyDetermination of single weight for each point in databaseResults in a single, consistent database for 1982, 1987, 1992, … that is easy to use for statistical analyses
27 Derived indicator construction Statistical Design-Based estimation ConveyResultsandfindingsCompileand managedataAssessand interpretdataDerived indicator constructionStatistical Design-Based estimationStatistical Model-assisted and model-based estimationInference to unsampled locationsSpatial pattern inference (or where is the map!)Semi-empirical modelingIncorporating physical processesEmpirical statistical modeling using auxiliary dataDerived indicators: Tree volume (FIA), Soil erosion (NRCS) , Index of Biotic Integrity (EMAP), nutrient loads (NAWQA)Considerable statistical modeling typically key part in the development of derived indicators.
28 Design-Based Population Estimation Scientific inference from sample to populationMinimizes assumptions used in the inference processRelies on principles of statistical survey design and analysisNatural resource programs who useForest Inventory & AnalysisNational Resource InventoryNational Wetland Status and Trends ProgramNational Agricultural Statistics Service programsEnvironmental Monitoring and Assessment Program (EMAP)
29 Estimating Site Occupancy Rates MacKenzie, D. I., J. D. Nichols, G. B. Lachman, S. Droege, J. A. Royle, and C. A. Langtimm Estimating site occupancy rates when detection probabilities are less than one. Ecology 83:Likelihood based model for estimationAssumes simple random sample of sitesSimilar to closed-population, mark-recapture modelEstimate probability of occupancy and probability of detectionEstimation with complex survey designsMaximum likelihood as beforeLikelihood must incorporate survey designStratificationUnequal probability of selectionCluster sample
31 Statistical Model-assisted and Model-based Estimation Improve estimation based on complete coverage informationAdjustment for non-response at the site levelSmall area estimationSpatially-explicit model of probability of impairmentIdentification of “hot spots” likely to be impairedWill see increased use of these techniques
32 Semi-parametric Small Area Model: Northeast Lakes ANC prediction for HUCs J. Breidt, J. Opsomer, G. Ranalli, G. Claeskens, G. KauermannColorado State University STARMAP research program sponsored by USEPA STAR grants program
33 Semi-empirical Modeling: USGS NAWQA Estimated nitrogen export (kg/km2/yr) for watersheds of the conterminous United States.SPARROW relates in-stream water-quality measurements to spatially referenced characteristics of watersheds, including contaminant sources and factors influencing terrestrial and stream transport.The model empirically estimates the origin and fate of contaminants in streams, and quantifies uncertainties in these estimates based on model coefficient error and unexplained variability in the observed data.
34 Questions to ask when planning reporting ConveyResultsandfindingsDevelopmonitoringobjectivesAssessand interpretdataQuestions to ask when planning reportingWhat is objective for communicating the results?Who is the target audience?What is message want to convey?What formats will be used to convey the message?Statistical perspectiveClarity on scope of inference: target population/sampled populationReporting of precision for resultsConstruction of statistical tablesConstruction of presentation quality statistical graphicsTwo of the identified weaknesses for monitoring programs are (1) not being tied to decision-making and (2) not convey results in a timely manner to key audiences.Monitoring program staff are dominated by scientists who are experienced in writing journal articles. They have less experience in communicating to other audiences.Also time to remember that the Monitoring Framework is an information system – not just a data generation system.To produce timely reports requires pre-planning. NASS is an example of an organization that has a history for timely production of reports based on survey results. It does take a “production mentality” to make that happen.Statistical perspective can contribute to effective and timely reporting.The first two have to do with the scientific-defensibility of the report – must be clear what natural resource the results apply to and how well the results are known.When a table or a graph is constructed they should be constructed to communicate a specific message. We no longer need to use tables as a data storage device – that can be done in other ways. Why should a table of results by state have the states listed in alphabetical order? Readers will find it difficult to see a message! A number of statisticians have contributed to our knowledge of how to construct tables and graphics for presentation purposes. Several are Tufte, Wainer, Cleveland, and Carr. For example, Wainer notes: "tables are for communication, not archiving” and “tables can be improved by making them more graphical”Like to show a few statistical graphics that have been used by monitoring programs. Not to say that these are always the best in all circumstances but have been useful.
35 IBI Results Geographic Distribution (InsufficientData)North-Central AppalachiansWestern AppalachiansUse of the “Stop light” color model: Red: Poor, Yellow: Fair, Green: GoodNote the clear identification of portion of streams where have insufficient data to make an assessment.Ridge and Blue RidgeValleys
36 Estuarine Stressor Comparison Benthic invertebrate conditionLouisianian ProvinceVirginian ProvinceDegraded18 ± 8%Degraded30 ± 6%Undegraded82 ± 8%Undegraded70 ± 6%ConditionUnknown10%Unknown39%Low DissolvedOxygen 49%An attempt at displaying associations. It only gives half of the picture. May be that have same Stressor percents for Undegraded portion of the resource.The graphic does make that point that although the percent degraded is not all that different between the two provinces, the stressors are very different.Habitat 14%Metals 42%Low D.O.Contaminants 10%Contaminants 28%Both2%Toxicity 4%Stressors Associated with Degraded Condition
37 MAIA: Relative Risk Assessment “The risk of Poor BMI is 1.6 timesgreater in streams with Poor SEDthan in streams with OK SED.”This graphic focuses on impact of stressors on biotic indicators in streams.Uses relative risk as one way of communicating impact of stressor to the biology. Uses the same language that is used to communicate stressor risks to humans. Change the stressors to those related to heart disease.Gives information on how big a problem a stressor is (extent) and increase in risk to the biota when it is present.
38 West Virginia has defined 25 Hydrologic units covering the state and reports of the condition of streams by these units. This graphic is a RowPlot of population estimates for the mean stream condition index and the std dev of stream condition index. With 95% confidence intervals.Note that it is sorted from good to poor scores for mean. What is missing are micromaps as another column that show the spatial pattern.
39 Same survey. Now the focus is on presenting summaries of the distribution using boxplots. Provides information on how variable the scores are within a reporting unit.
40 This is an illustration of reporting not only the overall index of stream condition (WVSCI in first column) but the seven components that go into the overall index.Again results sorted by mean. This plot is more technical – would be of interest to those familiar with the construction of the overall index – scientists.
42 Lake Ontario Diporeia Spatial Pattern Example where statistical spatial analyses were used to estimate a surface over an area.It falls short of being a good presentation graphic.
43 SummaryStatistical perspective is pervasive throughout the monitoring frameworkSubstantial advances in incorporating statistical perspective in monitoring have been made during the last half of the 20th centuryMany statistical methodology advances are on the horizon that will improve monitoring cost-effectivenessIncorporating a statistical perspective throughout the development and implementation of a monitoring program is no longer optional – it is essential
44 When will natural resource monitoring programs be able to support an Environmental Statistics Briefing Room?