1. Survey Design

2. So Many Surveys Political ‘push polls’ – Trying to influence, not collect data Marketing polls – Some are just advertisements News Polls

3. How to write a good survey Think in terms of ‘questionnaire’, not questions – How are questions related to each other? Have a clear topic – A research question – Know what concepts you are measuring Think in terms of hypotheses, IVs and DVs – Every question should have a clear reason for inclusion

4. The Instrument – the whole package Make it look/sound appealing – Clear directions that make it sound interesting/important – shorter is always better (i.e. 15 minutes) Consistency is good – Use similar response formats for questions

5. Introductions and Instructions What is the study about? Informed and Voluntary Consent – Institutional Review Boards Contact information for researcher Let them know they can have the results

6. Example of Instructions

7. Question Order Early questions make identities salient – Example: Political ID and Political Issues So, Put space between the questions Early questions should not be threatening – How often do you watch pornography? – Have you ever been convicted of a crime? Early questions should not be boring – How old are you? Of course, this all depends – Different survey settings call for different strategies – It’s a judgment call

8. Bad question order example (and probing)

9. Do previous questions influence answers?

10. Social Desirability Sensitive Topics – sex, drugs, crime, embarrassment, church attendance, etc…. People will sometimes tell you who they wish they were, not who they really are Advice: write questions so that undesirable answers are made to sound ok.

11. Social Desirability Example 1. How often do you attend mass at this parish? 1 Daily 2 Weekly 3 Few times a month 4 Few times a year 5 I’m a visitor today 6 I don’t know

12. Open and Closed Ended Closed ended: give responses Open ended: let respondent give comments If closed: – Mutually exclusive and exhaustive Allow don’t know / not sure ? If open: – Will have to code after data collection – Might effect length of interview Could be quicker or slower

13. Open vs. Closed Example How many people do you consider close friends? ________ How many people do you consider close friends? 1.0 2.1 to 5 3.6 to 10 4.More than 10

14. Avoid Double Barreled Questions A single question that really asks two separate things. – Watch out for this if you use the word “and” – Try not to preface any question with something people might disagree with Do you think the United States government should raise taxes and spend more money on education?

15. Double Barreled

16. Skip Patterns Contingency Questions – Answer to a question sends you down a different path If yes, go to question number X Use it to shorten survey for respondent Too many can get complicated – Use with caution on a self administered survey

17. Skip Pattern Example

18. Negative or Biased Terms Negative terms may be confusing The death penalty should be illegal – SDA DA N A SA – What does it mean if you disagree? Avoid Biased Terms – Biased questions may encourage a certain answer – Depends on connotation of words

19. “Forbid” versus “Not Allow” Do you think the U.S. should forbid public speeches against democracy? Do you think the U.S. should allow public speeches against democracy? “Americans are much more willing to not allow speeches than they are to forbid them.”

20. Some general tips If asking about the past, use a time period people can remember – “In the last week…” Only ask about personal experiences – Don’t ask about other people’s impressions Beware of hidden contingencies – In the past month, have you crossed the street to avoid someone you found frightening?

21. Response Rate % of those sampled who participated Larger RR usually means more representative data General Guidelines, no clear rule – Shoot for 70% – Happy with 50% – Don’t be surprised with 30% or lower

22. Methods of Administration – Mail Survey Self Administered – Relies on Participant Enthusiasm Low Response Rates Appearance is Most important in this case Give an Incentive? Slow Send a prior letter, maybe call Send the survey Send a reminder letter to those who haven’t replied Contact again

23. Methods of Administration –Phone Survey Random Digit Dial (RDD) – Very Fast – Very Expensive Participation gets worse every year Cooperation Rates – 20% to 30%, or lower – Statistically, in 30% range is not that different from in 60% range as long as sample is representative

24. Methods of Administration -In person interview Best response rate & Most Expensive – Contact ahead of time to arrange meeting – Sit with respondent while collecting data Well trained interviewers – Dress, Race, Gender, etc. – Threatening Situations Mixed Methods – Computer Assisted – Self administered portions of an in person interview

25. Methods of Administration -Web Survey Still a new method, very few clear guidelines Visually, I’d say follow same guidelines as a a mail survey How do you get your sample? – Do you trust online identities? Can be very affordable – Survey Monkey Survey Monkey

26. General comments on survey error Sample surveys are way to generate estimates There are multiple sources of error – Sampling error – Low response rates – Poorly written questions