Using Surveys to Design and Evaluate Watershed Education and Outreach Day 5 Methodologies for Implementing Mailed Surveys Alternatives to Mailed Surveys Interpreting Survey Results

Survey Design and Implementation

Using surveys Why? –Good for large, dispersed populations –Structured measures –Strong tools for analysis Why not? –Expensive & can be intrusive –Might miss the issue – options are pre- defined –Might oversimplify the issue

Steps in Survey Use What do you need to know? Who is your target population? What’s the best way to get the info? Create, pilot, refine questionnaire Census or sample? Send out your questionnaire. –Standard procedures Analyze, interpret and communicate the results

Sampling Probability –Simple Random, Stratified, Cluster Sampling Non-probability –Convenience, snowball, quota, purposive (though not typically done for survey research) Sample size—depends on: –Cost and time constraints –Degree of precision needed Larger samples provide more precise estimates of population parameters More heterogeneous populations require larger samples

Threats to Sample Representativeness Non-response Error –Bias that results when data are not collected from all members of a sample –Non-respondents may be systematically different from respondents

Response Rate & Non-Response Bias Higher response rate –yields greater statistical power –Reduces chance of bias Non-response bias can be assessed: –Contacting non-respondents with reduced set of questions –Comparing respondents to census data

Increasing Response Rates Use trusted networks to promote participation Use trusted source to endorse survey & personalize Use Dillman’s Tailored Design method

Increasing Response Rates Repeated contacts 1.Pre-notice postcard or letter 2.Survey with letter and stamped return envelope 3.Reminder post-card or letter 4.Second survey 5.Third survey (optional) or reminder letter

Increasing Response Rates Ease of response - make it easy for them to respond! –Easy to understand –Easy to complete –Easy to return Tips: –Appearance – attractive, interesting, engaging –Make sure the wording is clear and appropriate for the audience –Provide very clear directions –Reduce their costs for returning Pilot Test!

More ways… Incentives –Pre-payment of small cash incentive –Post-payment (e.g., offer to pay if R refuses) –Prize drawings, coupons, etc (not as effective) Sponsorship –University or government sponsorship often helps over commercial organizations Multiple-mode survey –Combining web and mail survey –Follow-up mail survey with telephone contact Reduce respondent burden

Costs of High Response Rates Staff time Printing Postage and mail supplies –Advance letters, thank you letters –multi-wave mail process –Sending final reports?

In Summary: Keys to Effective Surveys Good mailing list –Accurate representation of target audience –Up-do-date addresses Well-constructed questionnaire and cover letter Pilot test Follow Dillman’s Tailored Design method