Environmental Protection Agency Statistical Sampling Plans: Collaborative Efforts between Statisticians and Subject Matter Experts by Marla Smith ICES-III June 2007

2 Overview Our program and its objective Surveys in our program Multi-disciplinary teams Survey development Lessons learned and examples

President Nixon creates EPA 1972 Clean Water Act amendments shifts: -- Reliance on violations of water and quality standards as the primary enforcement tool, to -- Establishment of specific technology-based effluent guidelines that are enforceable as permit conditions Milestones -- Environmental Protection 22 June 1969 Cuyahoga River Fire (Cleveland, OH) –Time Magazine: Some river! Chocolate-brown, oily, bubbling with subsurface gases, it oozes rather than flows.

4 What are Effluent Guidelines (EG)? Restrictions on the types and amounts of pollutants that can be discharged into navigable waters of the U.S. by various industries Based upon best available process and/or wastewater technologies and performance Separately implemented through permits

5 After collecting data from industry, EPA analyzes the data to identify or refine: –Technology Options and the related pollutant reductions and incremental compliance costs –Economic Impacts –Industry subsectors –Environmental Benefits Industry Identified Sampling, Industry Survey, etc. Formulation of Options and Decision-making Analysis of Data (including public comments) Why Do We Conduct Surveys?

6 EG Surveys One-time surveys Duty to respond is mandatory under the Clean Water Act

7 EG Teams Each team has its own personality –Statistician has to adapt accordingly Typical team includes: –Team Leader –One or Two Engineers –One Economist –One Environment Assessor –One Statistician Everyone supports multiple projects –Some more part-time than others Each team member has contract resources –Additional level of coordination Management and lawyers provide oversight

8 Survey Development: EG Team Each project team expends considerable effort to: –Learn about each industry –Determine the data needs for each type of analysis –Refine study objectives –Find a sample design that will satisfy all requirements –Develop the survey instrument

9 Survey Development: Statistician Leads survey design work Translates team objectives into specifications for statistics contractor Reviews statistics contractor products –Often reviews multiple versions before sharing one with the team Discusses sample design concerns with management Documents and defends sample design as part of government approval process Reviews final draft of questionnaire

10 Examples from 3 Surveys Airport Deicing EG –Airlines and airports perform deicing and anti-icing of aircraft and airfield pavement Can result in environmental impacts –Survey Design includes two industries: Airports: Single phase: –Detailed questionnaire sent to stratified sample Airlines: Two-phase: –Screener sent to census of relatively active airlines selected airports –Detailed questionnaire sent to subset of deicers

11 Overview: Examples (cont) Drinking Water EG –Residuals are created when drinking water is treated. Sometimes the residuals are discharged back into the river, stream, etc. –Survey Design: Stratified sample Biosolids – not an EG –Biosolids result from the treatment of domestic sewage in a treatment facility (different than drinking water treatment) –Survey Design: Stratified sample –Team aspects similar to EG

12 Lessons Learned: General Listen –Learn as much as possible –Need to consider what isnt said –Participate in non-survey meetings Ask questions –Ask the same question at different meetings –Keep asking until theres an answer Revisit and confirm previous decisions –May get different answers at different times Leave the room last at a meeting –Listen to what is (still) being discussed –Allow time for questions about statistics and data Keep it simple –Make sure everyone understands –Bottom line is more important than how we got there

13 Lessons learned: Target Pop. Concept can be difficult for non-statisticians –Target population versus available data –Not everyone needs or should get a questionnaire Sub-populations need to be defined –Narrows down the target population –Provides insight to study objectives –Reduce eligible population Each pick means a loss of a pick elsewhere –Verify that legal authority encompasses all members of the target population –Decide if oversampling is appropriate Write down the definition –Team needs to review and agree

14 Target Pop. – Examples Airport Deicing –Not everyone should get a questionnaire: Military airports were excluded from survey –Questionnaire not the right vehicle for collecting data –Affordability questions are not relevant Still included in EG development Biosolids –Target population definition: Two alternatives: facilities existing today or in 1988 –Team knew a lot about the 1988 facilities –Existing data indicated differences in population over time Management wanted estimates of current conditions –Subpopulation review: 0.2 percent of the facilities generate 32 percent of the total nationwide flows Team decided to oversample

15 Target Pop. – Examples Drinking Water –Target population definition: Difficult to describe (series of criteria) Important during sample draw review –Subpopulation considerations: Excluded very small systems –Approximately 98 percent of all 160,000 drinking water systems are very small –Collectively, they have little environmental impact –Exclusion greatly simplified sample frame issues Oversampled certain systems –Team concluded they were most likely to have biggest impact in terms of engineering, economics, and environment assessment –Ratios based upon engineering judgment

16 Lessons Learned: Design Recognize that using a sample is a leap of faith for non-statisticians –Sometimes need to involve management –Estimate resources for different sample sizes Keep stratification under control –Too many variables will complicate the analyses and unnecessarily increase sample sizes Accommodate favorite picks early on –Team will reject sample draws without them –Provides additional insight into study objectives –Often can conclude these facilities are different enough that they can only represent themselves Prepare to develop multiple designs as the team goes through its thought process Keep management involved

17 Design – Examples Airport Deicing –Listen to whats not said: Proposed design: Stratified by four hub sizes –Hub sizes readily available in sample frame –But, team wasnt planning to stratify by size in any analyses Final design: Stratified by two size categories –This approach will simplify our data analyses Drinking Water –Keep management involved Proposed two-phase design –Management questioned sample size Final design –One stage with reduced stratification –Less detailed, shorter, questionnaire –Keep stratification under control Removed one variable to reduce sample size

18 Lessons Learned: Draw Include time in the schedule for a second draw and design tweak –Team generally concludes that the first draw will never work Reasons are often valid Discussion may reveal additional study objectives not previously recognized by the team –Design tweaks are generally very close to original design –Statistical adjustments to responses are possible Team needs to compare analysis complications versus redrawing sample Also recognize that no draw will be perfect

19 Draw – Examples Drinking Water: No draw will be perfect –Team wanted to reject certain systems based upon non-frame data –Team retained the draw after further review Concerns about bias Judgment that statistically valid adjustments can be made later Strong management support for statistics arguments Biosolids: Second draw was necessary –Contact information not available for small facilities Tend to have part-time staffing Difficult to find person with the key –Team redefined target population to exclude smalls –Team able to move forward with the second draw

20 Conclusion Listening and asking questions is a critical part of developing survey designs Flexibility is important in redesigning and reevaluating to find a workable design