1. Survey Methods Overview
Source: Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley

2. ©2007 Mark Mitchell & Janina Jolley
Overview
Questions to ask before doing survey research
The advantages and disadvantages of different survey instruments
Planning your survey
Administering your survey
Analyzing your data
PowerPoint presentation to accompany Research Design Explained , 6th edition;
©2007 Mark Mitchell & Janina Jolley

3. Questions to Ask Before Doing Survey Research
Do you have a clear hypothesis?
Do your questions focus on that hypothesis?
Will participants answers provide accurate answers to your questions?
To whom will your results apply?
Hypothesis - A proposition made as a basis for reasoning, without any assumption of its truth.
PowerPoint presentation to accompany Research Design Explained , 6th edition;
©2007 Mark Mitchell & Janina Jolley

4. Answers May Be Inaccurate Because:
People don’t know the causes of their own behavior
People’s memories are inaccurate
People are not good at predicting their future behavior
People are not telling you what they know because of:
Social desirability bias
Demand characteristics
Response sets
PowerPoint presentation to accompany Research Design Explained , 6th edition;
©2007 Mark Mitchell & Janina Jolley

5. Generalizing Your Results
Do you know what your population is?
How can you get a representative sample of that population?
Will nonresponse bias prevent you from generalizing your results to that population?
PowerPoint presentation to accompany Research Design Explained , 6th edition;
©2007 Mark Mitchell & Janina Jolley

6. Advantages & Disadvantages
Easy way to get a lot of information
However, that information:
Will not have internal validity
May not have construct validity because of self-report problems
May not have external validity because of poor sampling or because of non-response bias
May not answer research question because survey questions weren’t focused on hypotheses
PowerPoint presentation to accompany Research Design Explained , 6th edition;
©2007 Mark Mitchell & Janina Jolley

7. The Advantages and Disadvantages of Different Survey Instruments
Written Instruments
Interviews
PowerPoint presentation to accompany Research Design Explained , 6th edition;
©2007 Mark Mitchell & Janina Jolley

8. ©2007 Mark Mitchell & Janina Jolley
Written Instruments
Self-administered questionnaires
Cheap, easy to distribute to large sample--but nonresponse bias is big problem
Anonymous which may promote honest responses
Investigator-administered questionnaires
Higher response rates
May hurt sense of anonymity and thus decrease honesty of responses
Note: A highly refined version of the investigator-administered questionnaire is the psychological test
PowerPoint presentation to accompany Research Design Explained , 6th edition;
©2007 Mark Mitchell & Janina Jolley

9. ©2007 Mark Mitchell & Janina Jolley
Psychological tests
May be part or all of a questionnaire
Even if you don’t use questions from tests, you can try to use strategies that test-makers use
PowerPoint presentation to accompany Research Design Explained , 6th edition;
©2007 Mark Mitchell & Janina Jolley

10. Things Questionnaire Writers Can Learn from Test Makers
Pretest questions and measure
Standardize how measure is administered
Be objective
Be aware of response sets
Seek evidence for measure’s reliability and validity
PowerPoint presentation to accompany Research Design Explained , 6th edition;
©2007 Mark Mitchell & Janina Jolley

11. ©2007 Mark Mitchell & Janina Jolley
Interviews
May be worth added expense if
It increases response rate
Need ability to clarify questions, follow up on responses
May not be worth added expense if construct validity is harmed by
Interviewer bias
Participant trying to impress interviewer
Telephone interviews may be ideal solution
PowerPoint presentation to accompany Research Design Explained , 6th edition;
©2007 Mark Mitchell & Janina Jolley

12. Advantages of Telephone Interviews
Cheaper than face-to-face interviews
Can have high relatively high response rate
Interviewer bias controlled by
Lack of body cues
Monitoring/taping interviewer’s calls
PowerPoint presentation to accompany Research Design Explained , 6th edition;
©2007 Mark Mitchell & Janina Jolley

13. ©2007 Mark Mitchell & Janina Jolley
Planning a Survey
Deciding on a research question
Choosing the format of your questions
Choosing the format of your interview--if you use an interview
Editing your questions
Sequencing your questions
Refining your survey instrument
Choosing a sampling strategy
PowerPoint presentation to accompany Research Design Explained , 6th edition;
©2007 Mark Mitchell & Janina Jolley

14. Choosing the Format of Your Questions
Fixed alternative
Yes/No
Reliable
Not powerful
Likert
Open-ended
May not be properly answered
May be difficult to score
Likert (1932) developed the principle of measuring attitudes by asking people to respond to a series of statements about a topic, in terms of the extent to which they agree with them, and so tapping into the cognitive and affective components of attitudes.
A Likert-type scale assumes that the strength/intensity of experience is linear, i.e. on a continuum from strongly agree to strongly disagree, and makes the assumption that attitudes can be measured. Respondents may be offered a choice of five to seven or even nine pre-coded responses with the neutral point being neither agree nor disagree.
PowerPoint presentation to accompany Research Design Explained , 6th edition;
©2007 Mark Mitchell & Janina Jolley

15. Choosing the Format of Your Interview
Unstructured
Interviewer bias is a serious problem
Data may not be hard to analyse
Semi-structured
Follow-up questions allowed
Probably best for pilot studies
Structured
Standardized, reducing interviewer bias
PowerPoint presentation to accompany Research Design Explained , 6th edition;
©2007 Mark Mitchell & Janina Jolley

16. Editing Questions: Nine Mistakes to Avoid
1. Avoid leading questions 2. Avoid questions that invite the social desirability bias 3. Avoid double-barreled questions 4. Avoid long questions
5. Avoid negations 6. Avoid irrelevant questions 7. Avoid poorly worded response options 8. Avoid big words 9. Avoid ambiguous words & phrases
PowerPoint presentation to accompany Research Design Explained , 6th edition;
©2007 Mark Mitchell & Janina Jolley

17. ©2007 Mark Mitchell & Janina Jolley
Sequencing Questions
To boost response rate, put innocuous questions first, personal questions last
To avoid wasting time, qualify early
To increase accuracy, keep similar questions together
To boost response rate, put demographic questions last
PowerPoint presentation to accompany Research Design Explained , 6th edition;
©2007 Mark Mitchell & Janina Jolley

18. Putting the Final Touches on Your Survey Instrument
Professional appearance
Proof reading
Pilot testing
Practice coding responses--may lead to refining questionnaire so that it is easier to code responses
PowerPoint presentation to accompany Research Design Explained , 6th edition;
©2007 Mark Mitchell & Janina Jolley

19. Choosing a Sampling Strategy
Random sampling
Proportionate stratified random sampling
Convenience sampling
Quota sampling
Conclusions
Only random sampling will be representative
Nonresponse bias may ruin your sample
PowerPoint presentation to accompany Research Design Explained , 6th edition;
©2007 Mark Mitchell & Janina Jolley

20. Administering the Survey
Informed consent
Clear instructions
Debriefing
Confidentiality
PowerPoint presentation to accompany Research Design Explained , 6th edition;
©2007 Mark Mitchell & Janina Jolley

21. ©2007 Mark Mitchell & Janina Jolley
Analyzing Survey Data
Summarizing data
Inferential statistics
PowerPoint presentation to accompany Research Design Explained , 6th edition;
©2007 Mark Mitchell & Janina Jolley

22. ©2007 Mark Mitchell & Janina Jolley
Summarising Data
Interval or ratio data
Mean
Correlation coefficients
Tables of means
Nominal data
Frequencies, percentages
Phi coefficient
Tables of frequencies
PowerPoint presentation to accompany Research Design Explained , 6th edition;
©2007 Mark Mitchell & Janina Jolley

23. Using Inferential Statistics
Interval or ratio data
Looking at relationships between pairs of variables
If have two groups, could use t-test between means
If not, should use test to see whether the correlation between two variables was significant
Be aware that if you do numerous statistical tests, you may be setting yourself up for a Type 1 error
To look at more than two variables at once, you can do
ANOVA
Multivariate analysis of variance, multiple regression, factor analysis, and other sophisticated tests
PowerPoint presentation to accompany Research Design Explained , 6th edition;
©2007 Mark Mitchell & Janina Jolley

24. Using Inferential Statistics
Nominal data
Chi-Square test
Be aware that if you do numerous statistical tests, you may be setting yourself up for Type 1 errors
PowerPoint presentation to accompany Research Design Explained , 6th edition;
©2007 Mark Mitchell & Janina Jolley

25. ©2007 Mark Mitchell & Janina Jolley
Concluding Remarks
Survey research is the most used research method
Survey research is the most misused research method
You know how to use rather than abuse survey research
PowerPoint presentation to accompany Research Design Explained , 6th edition;
©2007 Mark Mitchell & Janina Jolley

26. ©2007 Mark Mitchell & Janina Jolley
Any Questions?
PowerPoint presentation to accompany Research Design Explained , 6th edition;
©2007 Mark Mitchell & Janina Jolley