Survey Methods Overview Source: Research Design Explained 6th edition; ©2007 Mark Mitchell & Janina Jolley

©2007 Mark Mitchell & Janina Jolley Overview Questions to ask before doing survey research The advantages and disadvantages of different survey instruments Planning your survey Administering your survey Analyzing your data PowerPoint presentation to accompany Research Design Explained , 6th edition; ©2007 Mark Mitchell & Janina Jolley

Questions to Ask Before Doing Survey Research Do you have a clear hypothesis? Do your questions focus on that hypothesis? Will participants answers provide accurate answers to your questions? To whom will your results apply? Hypothesis - A proposition made as a basis for reasoning, without any assumption of its truth. PowerPoint presentation to accompany Research Design Explained , 6th edition; ©2007 Mark Mitchell & Janina Jolley

Answers May Be Inaccurate Because: People don’t know the causes of their own behavior People’s memories are inaccurate People are not good at predicting their future behavior People are not telling you what they know because of: Social desirability bias Demand characteristics Response sets PowerPoint presentation to accompany Research Design Explained , 6th edition; ©2007 Mark Mitchell & Janina Jolley

Generalizing Your Results Do you know what your population is? How can you get a representative sample of that population? Will nonresponse bias prevent you from generalizing your results to that population? PowerPoint presentation to accompany Research Design Explained , 6th edition; ©2007 Mark Mitchell & Janina Jolley

Advantages & Disadvantages Easy way to get a lot of information However, that information: Will not have internal validity May not have construct validity because of self-report problems May not have external validity because of poor sampling or because of non-response bias May not answer research question because survey questions weren’t focused on hypotheses PowerPoint presentation to accompany Research Design Explained , 6th edition; ©2007 Mark Mitchell & Janina Jolley

The Advantages and Disadvantages of Different Survey Instruments Written Instruments Interviews PowerPoint presentation to accompany Research Design Explained , 6th edition; ©2007 Mark Mitchell & Janina Jolley

©2007 Mark Mitchell & Janina Jolley Written Instruments Self-administered questionnaires Cheap, easy to distribute to large sample--but nonresponse bias is big problem Anonymous which may promote honest responses Investigator-administered questionnaires Higher response rates May hurt sense of anonymity and thus decrease honesty of responses Note: A highly refined version of the investigator-administered questionnaire is the psychological test PowerPoint presentation to accompany Research Design Explained , 6th edition; ©2007 Mark Mitchell & Janina Jolley

©2007 Mark Mitchell & Janina Jolley Psychological tests May be part or all of a questionnaire Even if you don’t use questions from tests, you can try to use strategies that test-makers use PowerPoint presentation to accompany Research Design Explained , 6th edition; ©2007 Mark Mitchell & Janina Jolley

Things Questionnaire Writers Can Learn from Test Makers Pretest questions and measure Standardize how measure is administered Be objective Be aware of response sets Seek evidence for measure’s reliability and validity PowerPoint presentation to accompany Research Design Explained , 6th edition; ©2007 Mark Mitchell & Janina Jolley

©2007 Mark Mitchell & Janina Jolley Interviews May be worth added expense if It increases response rate Need ability to clarify questions, follow up on responses May not be worth added expense if construct validity is harmed by Interviewer bias Participant trying to impress interviewer Telephone interviews may be ideal solution PowerPoint presentation to accompany Research Design Explained , 6th edition; ©2007 Mark Mitchell & Janina Jolley

Advantages of Telephone Interviews Cheaper than face-to-face interviews Can have high relatively high response rate Interviewer bias controlled by Lack of body cues Monitoring/taping interviewer’s calls PowerPoint presentation to accompany Research Design Explained , 6th edition; ©2007 Mark Mitchell & Janina Jolley

©2007 Mark Mitchell & Janina Jolley Planning a Survey Deciding on a research question Choosing the format of your questions Choosing the format of your interview--if you use an interview Editing your questions Sequencing your questions Refining your survey instrument Choosing a sampling strategy PowerPoint presentation to accompany Research Design Explained , 6th edition; ©2007 Mark Mitchell & Janina Jolley

Choosing the Format of Your Questions Fixed alternative Yes/No Reliable Not powerful Likert Open-ended May not be properly answered May be difficult to score Likert (1932) developed the principle of measuring attitudes by asking people to respond to a series of statements about a topic, in terms of the extent to which they agree with them, and so tapping into the cognitive and affective components of attitudes. A Likert-type scale assumes that the strength/intensity of experience is linear, i.e. on a continuum from strongly agree to strongly disagree, and makes the assumption that attitudes can be measured. Respondents may be offered a choice of five to seven or even nine pre-coded responses with the neutral point being neither agree nor disagree. PowerPoint presentation to accompany Research Design Explained , 6th edition; ©2007 Mark Mitchell & Janina Jolley

Choosing the Format of Your Interview Unstructured Interviewer bias is a serious problem Data may not be hard to analyse Semi-structured Follow-up questions allowed Probably best for pilot studies Structured Standardized, reducing interviewer bias PowerPoint presentation to accompany Research Design Explained , 6th edition; ©2007 Mark Mitchell & Janina Jolley

Editing Questions: Nine Mistakes to Avoid 1. Avoid leading questions 2. Avoid questions that invite the social desirability bias 3. Avoid double-barreled questions 4. Avoid long questions 5. Avoid negations 6. Avoid irrelevant questions 7. Avoid poorly worded response options 8. Avoid big words 9. Avoid ambiguous words & phrases PowerPoint presentation to accompany Research Design Explained , 6th edition; ©2007 Mark Mitchell & Janina Jolley

©2007 Mark Mitchell & Janina Jolley Sequencing Questions To boost response rate, put innocuous questions first, personal questions last To avoid wasting time, qualify early To increase accuracy, keep similar questions together To boost response rate, put demographic questions last PowerPoint presentation to accompany Research Design Explained , 6th edition; ©2007 Mark Mitchell & Janina Jolley

Putting the Final Touches on Your Survey Instrument Professional appearance Proof reading Pilot testing Practice coding responses--may lead to refining questionnaire so that it is easier to code responses PowerPoint presentation to accompany Research Design Explained , 6th edition; ©2007 Mark Mitchell & Janina Jolley

Choosing a Sampling Strategy Random sampling Proportionate stratified random sampling Convenience sampling Quota sampling Conclusions Only random sampling will be representative Nonresponse bias may ruin your sample PowerPoint presentation to accompany Research Design Explained , 6th edition; ©2007 Mark Mitchell & Janina Jolley

Administering the Survey Informed consent Clear instructions Debriefing Confidentiality PowerPoint presentation to accompany Research Design Explained , 6th edition; ©2007 Mark Mitchell & Janina Jolley

©2007 Mark Mitchell & Janina Jolley Analyzing Survey Data Summarizing data Inferential statistics PowerPoint presentation to accompany Research Design Explained , 6th edition; ©2007 Mark Mitchell & Janina Jolley

©2007 Mark Mitchell & Janina Jolley Summarising Data Interval or ratio data Mean Correlation coefficients Tables of means Nominal data Frequencies, percentages Phi coefficient Tables of frequencies PowerPoint presentation to accompany Research Design Explained , 6th edition; ©2007 Mark Mitchell & Janina Jolley

Using Inferential Statistics Interval or ratio data Looking at relationships between pairs of variables If have two groups, could use t-test between means If not, should use test to see whether the correlation between two variables was significant Be aware that if you do numerous statistical tests, you may be setting yourself up for a Type 1 error To look at more than two variables at once, you can do ANOVA Multivariate analysis of variance, multiple regression, factor analysis, and other sophisticated tests PowerPoint presentation to accompany Research Design Explained , 6th edition; ©2007 Mark Mitchell & Janina Jolley

Using Inferential Statistics Nominal data Chi-Square test Be aware that if you do numerous statistical tests, you may be setting yourself up for Type 1 errors PowerPoint presentation to accompany Research Design Explained , 6th edition; ©2007 Mark Mitchell & Janina Jolley

©2007 Mark Mitchell & Janina Jolley Concluding Remarks Survey research is the most used research method Survey research is the most misused research method You know how to use rather than abuse survey research PowerPoint presentation to accompany Research Design Explained , 6th edition; ©2007 Mark Mitchell & Janina Jolley

©2007 Mark Mitchell & Janina Jolley Any Questions? PowerPoint presentation to accompany Research Design Explained , 6th edition; ©2007 Mark Mitchell & Janina Jolley