1. Non-Experimental designs: Surveys & Quasi-Experiments
Psych 231: Research Methods in Psychology

2. Announcements
Lab attendance is critical this week because group projects are being administered
Attendance will be taken.
Turn in the group project rating sheet 1

3. Non-Experimental designs
Sometimes you just can’t perform a fully controlled experiment
Because of the issue of interest
Limited resources (not enough subjects, observations are too costly, etc).
Surveys
Quasi-Experiments
Developmental designs
Small-N designs

4. Surveys What are they (review chpt 7)? Why conduct them?
Questionnaires and interviews that ask people to provide information about themselves
Why conduct them?
To compliment experimental work
Good/common first step, can collect a lot of data about a lot of variables
Best way to collect some kinds of information:
Descriptive, behavioral, and preferential
(e.g. demographic information, recreational behavior, and attitudes)

5. Surveys
Advantages
Can generalize about an entire population based on relatively small samples of individuals
Large amounts of data can be collected quickly with relatively little cost (effort, time, etc.)
But they’re often not as “cheap” as you may think
One can investigate internal events (for example, attitudes)

6. Surveys Disadvantages Correlational: causal claims shouldn’t be made
Non-response bias
Why doesn’t everybody respond?
Does response rate interact with variables of interest?
Large data sets are sometimes difficult to analyze
Self-reports may not be truthful
Response set - tendency to respond from a particular perspective (e.g., how a “moral” person would answer)

7. Stages of survey research
Stage 1) Identify the focus of the study and select your research method
What are the objectives of the research?
Is a survey method the best approach?
What kind of survey should be used?

8. Surveys methods Many different methods are used to administer surveys
Group administration (e.g. MASS testing session)
Mail surveys
Internet surveys
Telephone surveys
Face-to-face interviews
Focus group interviews

9. Stages of survey research
Stage 2) Determining the research schedule and budget
Stage 3) Establishing an information base
Find out what’s been done, what’s known
E.g., Find other related surveys
Stage 4) Identify the sampling frame
The actual population that the sample is drawn from (as opposed to the ideal population)
Think of it as operationalizing the conceptual level population

10. Stages of survey research
Stage 5) Determining the sample size and sampling method
Review Probability and Non-Probability methods

11. Voluntary response methods
A kind of convience sampling methods commonly used
TV uses a lot of these
call XXX-YYYY if you support Y
call XXX-ZZZZ if you support Z
Problem: You typically get only individuals with strong opinions to respond, so the results are often extremely biased

12. Importance of sample size
Sampling error - how is the sample different from the population?
Confidence intervals
An estimate of where the mean or percentage in the overall population is, based on the sample data
“John Doe has 55% of the vote, with a margin of error ± 3%”
Margin of error (that “± 3%” part)
Which would you be more likely to believe
We asked 10 people …
We asked 1000 people …
The larger your sample size, the smaller your margin of error will be.

13. Survey Questions Stage 6) Designing the survey instrument
Question construction: How the questions are written is very important
Clearly identify the research objectives
Do your questions really target those research objectives?
Take care wording of the questions
Keep it simple, don’t ask two things at once, avoid loaded or biased questions, etc.
How should questions be answered?

14. Survey Questions Question types
Open-ended (fill in the blank, short answer)
Can get a lot of information, but
Coding is time intensive and potentially ambiguous
Close-ended (pick best answer, pick all that apply)
Easier to code
Response alternatives are the same for everyone
Rating scales
Used for “how much” judgments
e.g., Likert scale – measures attitudes, agree/disagree
Take care with your labels
Range of scores, anchors

15. Stages of survey research cont.
Stage 7) Pre-testing the survey instrument
Fix what doesn’t seem to be working
Stage 8) Selecting and training interviewers
For telephone and in-person surveys
Need to avoid interviewer bias
Stage 9) Implementing the survey
Stage 10) Coding and entering the data
Stage 11) Analyzing the data and preparing a final report

16. Error in survey research
Sampling error
Response rate
What proportion of the sample actually responded to the survey
Hidden costs here - what can you do to increase response rates
Non-response error (bias)
Is there something special about the data that you’re missing? From the people who didn’t respond
Measurement error
Are your questions really measuring what you want them to?

17. Quasi-experiments What are they? General types
Almost “true” experiments, but with an inherent confounding variable
General types
An event occurs that the experimenter doesn’t manipulate
Something not under the experimenter’s control
(e.g., flashbulb memories for traumatic events)
Interested in subject variables
high vs. low IQ, males vs. females
Time is used as a variable

18. Quasi-experiments Advantages
Allows applied research when experiments not possible
Threats to internal validity can be assessed (sometimes)

19. Quasi-experiments Disadvantages Threats to internal validity may exist
Designs are more complex than traditional experiments
Statistical analysis can be difficult
Most statistical analyses assume randomness

20. Quasi-experiments Program evaluation
Research on programs that is implemented to achieve some positive effect on a group of individuals.
e.g., does abstinence from sex program work in schools
Steps in program evaluation
Needs assessment - is there a problem?
Program theory assessment - does program address the needs?
Process evaluation - does it reach the target population? Is it being run correctly?
Outcome evaluation - are the intended outcomes being realized?
Efficiency assessment- was it “worth” it? The the benefits worth the costs?

21. Quasi-experiments Nonequivalent control group designs
with pretest and posttest (most common)
(think back to the second control lecture)
participants
Experimental
group
Control
Measure
Non-Random
Assignment
Independent Variable
Dependent Variable
But remember that the results may be compromised because of the nonequivalent control group (review threats to internal validity)

22. Quasi-experiments Interrupted time series designs
Observe a single group multiple times prior to and after a treatment
Obs Obs Obs Obs Treatment Obs Obs Obs Obs
Look for an instantaneous, permanent change
Variations of basic time series design
Addition of a nonequivalent no-treatment control group time series
O O O T O O O & O O O _ O O O
Interrupted time series with removed treatment
If treatment effect is reversible

23. Next time Go to labs this week, attendance will be taken
Non experimental designs cont.
Read chapters 9 & 13
Reminder, journal summary 2 is coming up