2 Outlines Experimental Design Types of Experimental Designs DefinitionProcessA Key PointTypes of Experimental DesignsQuasi-Experimental DesignDesignsStrengthsWeaknessesExamples in GeographyOngoing Debate
3 Definition of Experimental Design A structured, organized methodTo determine whether some program or treatment causes some outcome or outcomes to occur.If X, then YBecause there may be lots of reasons, other than the program, for why you observed the outcome,If not X, then not Y needs to be addressed, too
4 Process of Experimental Design To show that there is a casual relationship,Two “equivalent” groupsThe program or treatment group gets the programThe comparison or control group does notThe groups are treated the same in all other respectsDifferences in outcomes between two groups must be due to “the program”
5 A Key Point of Experimental Design How do we create two groups that are “equivalent”?Assign people randomly from a common pool of people into the two groupsThe experiment relies on the idea of “random assignment” to obtain two similar groups.A key to the success of the experimentAssume that two groups are “probabilistically equivalent”
6 Types of Designs Is random assignment used? NoYesRandomized orTrue experimentIs there a control group ormultiple measures?NoYesNon-experiment: A research design in which a researcher observes a phenomenon without manipulating the independent variables(s)Non-experimentQuasi-experiment
7 Quasi-Experimental Design Similar to the experimental design, but lacks the key ingredient, “random assignment”Easily and more frequently implementedExtensively used in the social sciencesA useful method for measuring social variablesTwo classic quasi-experimental designsThe Nonequivalent Groups DesignThe Regression-Discontinuity Design
8 The Nonequivalent Groups Design The most frequently used in social researchTry to select groups that are as similar as possible to compare the treated one with the comparison onee.g. two comparable classrooms or schoolsCannot be sure whether the groups are comparableThe groups may be different prior to the studyAny prior differences between the groups may affect the outcome of the studyRequire a pretest and posttest
10 The Regression-Discontinuity Design A useful method for determining whether a program of treatment is effectiveParticipants are assigned to program or comparison groups based on a cutoff score on a preteste.g. Evaluating new learning method to children who obtained low scores at the previous test.Cutoff score = 50The treatment group: children who obtained 0 to 50The comparison group: children who obtained 51 to 100The program (treatment) can be given to those most in need
11 The Regression-Discontinuity Design With no treatment effectAll points to the left of the cutoff (treatment group) have been raised by 10 points on the posttest.The dashed line shows what we expect the treated group’s regression line to look like if the program had no effect.With Ten point treatment effect
12 The Regression-Discontinuity Design The program effect is suggested when we observe a discontinuity in the regression lines at the cutoff point.Also, we can now see how this design got its name.
13 Strengths of Quasi-Experimental Design Useful in generating results for general trends in social sciencesDifficult pre-selection and randomization of groupsEasily integrated with individual case studiesGenerated results can reinforce the findings in a case studyAllow statistical analysis to take placeEnable to reduce the time and resources required for experimentationNot required extensive pre-screening and randomization
14 Weaknesses of Quasi-Experimental Design Without proper randomization, statistical tests can be meaninglessDo not explain any pre-existing factors and influences outside of the experimentThe researcher needs to control additional factors that may have affected the resultsSome form of pre-testing or random selection may be necessary to explain statistical results thoroughly
15 Quasi-experiments vs. Non-experiments to address similar questions Both designs are applicable when the subjects are not able to be randomizedSome variables cannot ethically be randomizede.g. Studying the effect of maternal alcohol use when the mother is pregnantQuasi-experimentsNon-experimentsStrengthsEnable to compare with other groupsEnable to focus on one variableWeaknessesUnexpected factors might affect the resultsInterpretations might be improper
16 Example of Quasi-Experimental Design in Geography Baker and White (2003)The Effects of GIS on Students’ Attitudes, Self-efficacy, and Achievement in Middle School Science ClassroomsConducted the Nonequivalent quasi-experimental designTwo eighth grade teachers, across ten classroomsTotal 192 eighth grade students participatedTreatment group: used a Web-based GIS applicationControl group: used paper mapsTreatment GroupControl GroupInstructor 15136Instructor 24263
17 Example of Quasi-Experimental Design in Geography Impossible to randomly assign each student to a GIS or paper mapping conditionsRandomly assigned whole classes to two conditionsDifferent instructors affected the results differentlyInstructor effect played a substantial role in student attitudes and self-efficacy
18 Ongoing DebateWhether true experiments or quasi-experiments represents the superior designSupporters of true experimentsDifficult to isolate the program effects using quasi-experimentsQuasi-experimental results are biased and sensitive to minor changesNot sure about whether quasi-experimental designs can adequately control selection biasHard to determine better designTrue experiments are impossible and impractical in some casesThere has been a lively debate for over a decade on whether…A strict experimental design would include that mothers were randomly assigned to drink alcohol.This would be highly illegal due to the possible harm to the embryos.