1. Chapter 8
Survey Research

2. Overview
Questions to ask before doing survey research
The advantages and disadvantages of different survey instruments
Planning your survey
Administering your survey
Analyzing your data

3. Questions to Ask Before Doing Survey Research
Do you have a clear hypothesis?
Do your questions focus on that hypothesis?

4. Advantages and Disadvantages of Survey Research:Conclusions
Easy way to get a lot of information
However, that information:
Will not have internal validity
May not have construct validity because of self-report problems
May not have external validity because of poor sampling or because of nonresponse bias
May not answer research question because survey questions weren’t focused on hypotheses

5. The Advantages and Disadvantages of Different Survey Instruments
Written Instruments
Interviews

6. Self-administered questionnaires
Written Instruments
Self-administered questionnaires
Cheap, easy to distribute to large sample--but nonresponse bias is big problem
Anonymous which may promote honest responses
Investigator-administered questionnaires
Higher response rates
May hurt sense of anonymity and thus decrease honesty of responses
Note: A highly refined version of the investigator-administered questionnaire is the psychological test

7. May be worth added expense if
Interviews
May be worth added expense if
It increases response rate
Need ability to clarify questions, follow up on responses
May not be worth added expense if construct validity is harmed by
Interviewer bias
Participant trying to impress interviewer
Telephone interviews may be ideal solution*

8. Planning a Survey
Deciding on a research question
Choosing the format of your questions
Choosing the format of your interview--if you use an interview
Editing your questions
Sequencing your questions
Refining your survey instrument
Choosing a sampling strategy

9. Choosing the Format of Your Questions
Fixed alternative
Yes/No
Reliable
Not powerful
Likert
Open-ended
May not be properly answered
May be difficult to score

10. Choosing the Format of Your Interview
Unstructured
Interviewer bias is a serious problem
Data may not be hard to analyze
Semi-structured
Follow-up questions allowed
Probably best for pilot studies
Structured
Standardized, reducing interviewer bias

11. Editing Questions: Nine Mistakes to Avoid
1. Avoid leading questions 2. Avoid questions that invite the social desirability bias 3. Avoid double-barreled questions 4. Avoid long questions
5. Avoid negations 6. Avoid irrelevant questions 7. Avoid poorly worded response options 8. Avoid big words 9. Avoid ambiguous words & phrases

12. Sequencing Questions
To boost response rate, put innocuous questions first, personal questions last
To avoid wasting time, qualify early
To increase accuracy, keep similar questions together
To boost response rate, put demographic questions last

13. Putting the Final Touches on Your Survey Instrument
Professional appearance
Proof reading
Pilot testing
Practice coding responses--may lead to refining questionnaire so that it is easier to code responses

14. Choosing a Sampling Strategy
Random sampling
Proportionate stratified random sampling
Convenience sampling
Quota sampling
Conclusions
Only random sampling will be representative
Nonresponse bias may ruin your sample

15. Administering the Survey
Informed consent
Clear instructions
Debriefing
Confidentiality

16. Analyzing Survey Data
Summarizing data
Inferential statistics

17. Summarizing Data Interval or ratio data Nominal data Mean
Correlation coefficients
Tables of means
Nominal data
Frequencies, percentages
Phi coefficient
Tables of frequencies

18. Using Inferential Statistics
Interval or ratio data
Looking at relationships between pairs of variables
If have two groups, could use t-test between means
If not, should use test to see whether the correlation between two variables was significant
Be aware that if you do numerous statistical tests, you may be setting yourself up for a Type 1 error
To look at more than two variables at once, you can do
ANOVA
Multivariate analysis of variance, multiple regression, factor analysis, and other sophisticated tests

19. Using Inferential Statistics
Nominal data
Chi-Square test
Be aware that if you do numerous statistical tests, you may be setting yourself up for Type 1 errors

20. Concluding Remarks
Survey research is the most used research method
Survey research is the most misused research method
You know how to use rather than abuse survey research

21. Chapter 9
Internal Validity

22. Overview
Problems with two-group designs
Problems with pretest-posttest designs
Internal versus external validity

23. Two-Group Designs
The selection problem
Unsuccessful approaches to the selection problem

24. Sources of Selection Bias
Self-assignment to group produces selection bias
Researcher assignment to group produces selection bias
Arbitrary assignment to group produces selection bias
Choosing groups based on their differences results in having groups that are different

25. Matching: A Valiant, but Unsuccessful Strategy for Getting Identical Groups
The impossibility of perfectly matching individual participants: Identical participants do not exist
The difficulty of matching groups on every variable: There are too many variables
Two difficulties with matching groups on every relevant variable
Too many relevant variables
We don’t know what the relevant variables are
Problems with matching on pretest scores*

26. Problems with Matching on Pretest Scores
Selection by maturation interactions: participants growing In different ways
Regression

27. Problems with the Pretest-Posttest Design
Even without the treatment, results may change from pretest to posttest

28. Conclusions about Establishing Internal Validity
Trying to keep everything except the treatment constant is impossible.
Only option is to rule out extraneous variables.
Without random assignment, have to identify extraneous variables and then try to rule them out.
Campbell and Stanley’s 8 threats to validity provides a convenient way to identify extraneous variables.

29. The Relationship between Internal and External Validity
Attempts to rule out threats to internal validity may hurt external validity
Reducing selection threat at the expense of not studying a heterogeneous group of participants
Reducing history threat at expense of not studying participants in a naturalistic setting
Internal and external validity are not completely incompatible

30. Concluding Remarks
Be cautious about accepting cause-effect statements--if the study is not an experiment, the study’s internal validity is probably threatened by at least one of Campbell and Stanley’s 8 threats to validity.
Experimental designs, as you will soon see, automatically rule out the 8 threats to internal validity.

31. Chapter 10
The Simple Experiment

32. Overview Basic Logic and Terminology
Errors in determining whether results are statistically significant
Statistics and the design of the simple experiment
Nonstatistical considerations in the design of the simple experiment
Analyzing data from the simple experiment

33. Basic Logic and Terminology
Hypotheses
Manipulating the independent variable (IV) and measuring the dependent variable (DV)
Experimental and control groups
The importance of independence
The importance of assignment
Statistically significant vs. null results

34. Hypotheses
Experimental hypothesis: The treatment has an effect (IV-->DV)
Null hypothesis: The treatment does not have an effect

35. Manipulating the Independent Variable
Experimental and control groups: Similar, but treated differently

36. The Value of Assignment (Manipulating the Treatment)
Random assignment makes the treatment the only systematic difference between groups
Without random assignment you do not have an experiment

37. The Statistical Significance Decision:
The decision is whether to declare that a difference is not due to chance.
Statistically significant results
Nonsignificant (null) results

38. Statistically Significant Results: Declaring that the Treatment Has an Effect
Statistically significant effects are not necessarily large
Statistically significant results may not be in the direction you expect

39. Null Results: Why We Can’t Draw Conclusions from Nonsignificant Results
Nonsignificant results are not significant
Null results do not prove the null hypothesis: “I didn’t find it” doesn’t mean it doesn’t exist

40. Errors in Determining Whether Results Are Statistically Significant
Type 1 Errors: “Crying wolf*
Type 2 Errors: Failing to announce the wolf

41. Type 1 Errors: “Crying Wolf”
Reducing the risk of a Type 1 error
Accepting the risk of a Type 1 error

42. Type 2 Errors: Failing to Announce the Wolf
The need to prevent type 2 errors: Why you want the power to find significant differences

43. Statistics and the Design of the Simple Experiment
Power
Other statistical issues

44. Power and the Design of the Simple Experiment
Reduce the effect of random error
Standardize procedures and use reliable measures
Use a homogeneous group of participants
Code data carefully
Let random error balance out
Create larger effects: Bigger effects are easier to see

45. Inability to prove the null hypothesis limits hypotheses
How Statistical Issues Other Than Power Affect the Design of the Simple Experiment
Inability to prove the null hypothesis limits hypotheses
Requirement of independent random assignment limits what you can study and how you assign participants
Independence assumption may cause you to test participants individually

46. Nonstatistical Considerations in the Design of the Simple Experiment
External validity versus power
Construct validity versus power
Ethics versus power

47. Analyzing Data from the Simple Experiment: Basic Logic
Estimating what you want to know: your means are sample means
Calculating sample means: Getting your estimates
Comparing sample means: How to compare two imperfect estimates
Why we must do more than subtract the means from each other

48. How Random Error Affects Data from the Simple Experiment
Random error makes scores within a group differ
Random error can make group means differ

49. When Is a Difference Too Big to Be Due to Random Error?
Bigger differences are less likely to be due to chance alone
“Too big to be due to chance” depends on
1. How big chance is
Differences within groups tell you how big chance is
2. The extent to which chance balances out
With larger samples, random error tends to balance out

50. Analyzing the Results of the Simple Experiment: The t Test Using the t table
Decide on significance level before doing experiment
Look in table under appropriate significance level and degrees of freedom (N - 2)
Absolute value of t must exceed tabled value

51. Assumptions of the t Test
Two Critical Assumptions
Observations are independent
Data are interval or ratio
Two Less Critical Assumptions
Underlying distributions are normally distributed
Both groups’ distributions have the same variance

52. Questions Raised by Results
Questions Raised by Nonsignificant Results
Questions about power, such as “Did the experiment study enough participants?”
Questions Raised by Significant Results
External validity
Do results generalize to other levels of the IV?
Do other variables moderate the IV’s effect?
Construct validity (Was control group good enough?)

53. Concluding Remarks
Simple experiment is a simple way to establish internal validity
Independent random assignment is its cornerstone
Logic behind the simple experiment can be used to create experiments that use more than two groups. Such experiments may
Have more construct validity than simple experiments
Have more external validity than simple experiments
Answer more questions than simple experiments