Presentation on theme: "Non-Experimental designs: Surveys & Quasi-Experiments"— Presentation transcript:
1 Non-Experimental designs: Surveys & Quasi-Experiments Psych 231: Research Methods in Psychology
2 AnnouncementsLab attendance is critical this week because group projects are being administeredAttendance will be taken.Turn in the group project rating sheet 1
3 Non-Experimental designs Sometimes you just can’t perform a fully controlled experimentBecause of the issue of interestLimited resources (not enough subjects, observations are too costly, etc).SurveysQuasi-ExperimentsDevelopmental designsSmall-N designs
4 Surveys What are they (review chpt 7)? Why conduct them? Questionnaires and interviews that ask people to provide information about themselvesWhy conduct them?To compliment experimental workGood/common first step, can collect a lot of data about a lot of variablesBest way to collect some kinds of information:Descriptive, behavioral, and preferential(e.g. demographic information, recreational behavior, and attitudes)
5 SurveysAdvantagesCan generalize about an entire population based on relatively small samples of individualsLarge amounts of data can be collected quickly with relatively little cost (effort, time, etc.)But they’re often not as “cheap” as you may thinkOne can investigate internal events (for example, attitudes)
6 Surveys Disadvantages Correlational: causal claims shouldn’t be made Non-response biasWhy doesn’t everybody respond?Does response rate interact with variables of interest?Large data sets are sometimes difficult to analyzeSelf-reports may not be truthfulResponse set - tendency to respond from a particular perspective (e.g., how a “moral” person would answer)
7 Stages of survey research Stage 1) Identify the focus of the study and select your research methodWhat are the objectives of the research?Is a survey method the best approach?What kind of survey should be used?
8 Surveys methods Many different methods are used to administer surveys Group administration (e.g. MASS testing session)Mail surveysInternet surveysTelephone surveysFace-to-face interviewsFocus group interviews
9 Stages of survey research Stage 2) Determining the research schedule and budgetStage 3) Establishing an information baseFind out what’s been done, what’s knownE.g., Find other related surveysStage 4) Identify the sampling frameThe actual population that the sample is drawn from (as opposed to the ideal population)Think of it as operationalizing the conceptual level population
10 Stages of survey research Stage 5) Determining the sample size and sampling methodReview Probability and Non-Probability methods
11 Voluntary response methods A kind of convience sampling methods commonly usedTV uses a lot of thesecall XXX-YYYY if you support Ycall XXX-ZZZZ if you support ZProblem: You typically get only individuals with strong opinions to respond, so the results are often extremely biased
12 Importance of sample size Sampling error - how is the sample different from the population?Confidence intervalsAn estimate of where the mean or percentage in the overall population is, based on the sample data“John Doe has 55% of the vote, with a margin of error ± 3%”Margin of error (that “± 3%” part)Which would you be more likely to believeWe asked 10 people …We asked 1000 people …The larger your sample size, the smaller your margin of error will be.
13 Survey Questions Stage 6) Designing the survey instrument Question construction: How the questions are written is very importantClearly identify the research objectivesDo your questions really target those research objectives?Take care wording of the questionsKeep it simple, don’t ask two things at once, avoid loaded or biased questions, etc.How should questions be answered?
14 Survey Questions Question types Open-ended (fill in the blank, short answer)Can get a lot of information, butCoding is time intensive and potentially ambiguousClose-ended (pick best answer, pick all that apply)Easier to codeResponse alternatives are the same for everyoneRating scalesUsed for “how much” judgmentse.g., Likert scale – measures attitudes, agree/disagreeTake care with your labelsRange of scores, anchors
15 Stages of survey research cont. Stage 7) Pre-testing the survey instrumentFix what doesn’t seem to be workingStage 8) Selecting and training interviewersFor telephone and in-person surveysNeed to avoid interviewer biasStage 9) Implementing the surveyStage 10) Coding and entering the dataStage 11) Analyzing the data and preparing a final report
16 Error in survey research Sampling errorResponse rateWhat proportion of the sample actually responded to the surveyHidden costs here - what can you do to increase response ratesNon-response error (bias)Is there something special about the data that you’re missing? From the people who didn’t respondMeasurement errorAre your questions really measuring what you want them to?
17 Quasi-experiments What are they? General types Almost “true” experiments, but with an inherent confounding variableGeneral typesAn event occurs that the experimenter doesn’t manipulateSomething not under the experimenter’s control(e.g., flashbulb memories for traumatic events)Interested in subject variableshigh vs. low IQ, males vs. femalesTime is used as a variable
18 Quasi-experiments Advantages Allows applied research when experiments not possibleThreats to internal validity can be assessed (sometimes)
19 Quasi-experiments Disadvantages Threats to internal validity may exist Designs are more complex than traditional experimentsStatistical analysis can be difficultMost statistical analyses assume randomness
20 Quasi-experiments Program evaluation Research on programs that is implemented to achieve some positive effect on a group of individuals.e.g., does abstinence from sex program work in schoolsSteps in program evaluationNeeds assessment - is there a problem?Program theory assessment - does program address the needs?Process evaluation - does it reach the target population? Is it being run correctly?Outcome evaluation - are the intended outcomes being realized?Efficiency assessment- was it “worth” it? The the benefits worth the costs?
21 Quasi-experiments Nonequivalent control group designs with pretest and posttest (most common)(think back to the second control lecture)participantsExperimentalgroupControlMeasureNon-RandomAssignmentIndependent VariableDependent VariableBut remember that the results may be compromised because of the nonequivalent control group (review threats to internal validity)
22 Quasi-experiments Interrupted time series designs Observe a single group multiple times prior to and after a treatmentObs Obs Obs Obs Treatment Obs Obs Obs ObsLook for an instantaneous, permanent changeVariations of basic time series designAddition of a nonequivalent no-treatment control group time seriesO O O T O O O & O O O _ O O OInterrupted time series with removed treatmentIf treatment effect is reversible
23 Next time Go to labs this week, attendance will be taken Non experimental designs cont.Read chapters 9 & 13Reminder, journal summary 2 is coming up