2 Today’s GoalsIdentify issues of internal and external validity with various experimental designsDesign an experiment for a given topicCritique advantages and disadvantages of different designs
3 Ethical and logistical considerations. To ReviewWhy is most educational research comprised of non-experimental research designs?Ethical and logistical considerations.
4 To Review What is the purpose of non-experimental research? It describes current existing characteristics of the topic under study.
5 To ReviewHow does the independent variable function in non-experimental research?It is not manipulated.
6 To ReviewCan non-experimental research claim causality?NO!
7 An exampleRead the example given in class and in pairs respond to the questions
8 Experimental Research PurposeTo make causal inferences about the relationship between the independent and dependent variablesCharacteristicsDirect manipulation of the independent variableControl of extraneous variablesEliminate the variable from the studyStatistically adjust for the effect of the variable
9 Experimental Designs Single Group Post-test Single Group Pre-test Post-testNon-Equivalent Groups Post-testQuasi-Experimental DesignRandomized Post-test onlyRandomized Pre-test Post-testFactorial
10 Experimental Validity Internal validityThe extent to which the independent variable, and not other extraneous variables , produced the observed effect on the dependent variableExternal validityThe extent to which the results are generalizable
11 Internal ValidityThreats that reduce the level of confidence in any causal conclusionsKey Question: Is this a plausible threat to the internal validity of the study?
12 Threats to Internal Validity HistoryExtraneous events have an effect on the subjects’ performance on the dependent variableThe crash of the stock market, 9-11, the invasion of Iraq, etc.SelectionGroups that are initially not equal due to differences in the subjects in those groupsPositive and negative attitudes, high and low achievers, etc.
13 Threats to Internal Validity MaturationChanges experienced within the subject over timePretestingThe effect of having taken a pretestInstrumentationPoor technical quality (i.e. validity, reliability) or changes in instrumentation
14 Threats to Internal Validity Subject attritionDifferential loss of subjects from groupsStatistical regressionThe natural movement of extreme scores toward the meanDiffusion of treatmentThe treatment is given to the control groupExperimenter effectsDifferent characteristics or expectations of those implementing the treatments across groups
15 Threats to Internal Validity Subject effectsThe effects of being aware that one is involved in a studyFour typesHawthorne effectJohn Henry effectNovelty effect
16 Internal ValidityKey Point: Ultimately, validity is a matter of judgment. Ask if it is reasonable that possible threats are likely to affect the results.
17 External ValidityThe extent to which results can be generalized from a sample to a particular population.Question – Why would really good internal validity often result in poor external validity?
18 External Validity Factors affecting external validity Subjects Representativeness of the sample in comparison to the populationPersonal characteristics of the subjectsSituations - characteristics of the settingSpecific environmentSpecial situationParticular school
19 External ValidityImportance of explanation of sampling procedures
20 Experimental Designs Examples Single Group Post-test Single Group Pre-test Post-test –Non-Equivalent Groups Post-test –Quasi-Experimental Design –Randomized Post-test only –Randomized Pre-test Post-test –Examples
21 Your TaskBased on the topic of your proposal, design an experimental study using the design you were assigned.Write a research question and hypothesis.Sketch out the methods.Identify strengths and weaknesses of the design.
22 Experimental Designs Notation R indicates random selection or random assignmentO indicates an observationTestObservation scoreScale scoreX indicates a treatmentA, B, C, ... indicates a group
23 Pre-Experimental Designs No pre-experimental design controls internal validity threats wellSingle group pretest onlyA X OInternal validity threatsHistory, maturation, attrition, experimenter effects, subject effects, and instrumentation are viable threatsUseful only when the research is sure of the status of the knowledge, skill, or attitude being changed and there are no extraneous variables affecting the results
24 Pre-Experimental Designs Single group pretest post-testA O X OInternal validity threatsMaturation and pretesting are threatsHistory and instrumentation are potential threatsUseful when subject effects will not influence the results, history effects can be minimized, and multiple pretests and post-tests are used
25 Pre-Experimental Designs Non-equivalent groups post-test onlyA X O B OInternal validity threatsDefinite Threat: SelectionPotential Threats: History, maturation, and instrumentationUseful when groups are comparable and subjects can be assumed to be about the same at the beginning of the study
26 Quasi-Experimental Designs TypesNon-equivalent pretest/post-test, experimental control groupsA O X O B O ONon-equivalent pretest/post-test, multiple treatment groupsA O X1 O B O X2 OUseful when subjects are in pre-existing groups (e.g. classes, schools, teams, etc.)
27 Quasi-Experimental Designs Threats to internal validitySelection is the major concernControls for statistical regressionLikely to control for most other threats, provided the groups are not significantly different from one anotherSee Table 9.2 for specific threats related to each design
28 True Experimental Designs Important terminologyRandom assignmentSubjects placed into groups by randomEnsures equivalency of the groupsRandom selection of subjectsSubjects chosen from population by randomEnsures generalizability to the population from which the subjects were selected (i.e. external validity)
29 True Experimental Designs TypesRandomized post-test only experimental control groupsR A X O R B ORandomized post-test only multiple treatment groupsR A X1 O R B X2 O
30 True Experimental Designs Types (continued)Randomized pretest/post-test multiple treatment groupsR A O X1 O R B O X2 ORandomized pretest/post-test experimental control groupsR A O X O R B O O
31 True Experimental Designs Threats to internal validityControls for selection, maturation, and statistical regressionLikely to control for most other threatsSee Table 9.2 for specific threats related to each design
32 Factorial DesignsResearch designs containing two or more independent variablesExample: A study of the effects of two instructional strategies on male and female students’ math achievementExamples of factorial designs
33 Types of Effects Main effects For each independent variable i.e., one main effect for instructional strategy and one main effect for math achievement
34 Types of Effects Interaction effects Consider the vitamins you take. Iron decreases fatigue.Vitamin C decreases stress.Vitamin C boosts the absorption of iron.If you are fatigued and stressed, you may want to take both iron and Vitamin C.The interaction of Vitamin C and iron means you may want to skip an iron supplement when taking Vitamin C.
35 Types of Effects Interaction effects A different effect for the level of the first independent variable across the levels of the second independent variablei.e., the first instructional strategy could be effective for males but not females, whereas the second instructional strategy could be effective for females but not malesOne cannot state the effectiveness of the treatment (i.e., instructional strategy) without qualifying it relative to the dependent variable (gender).
36 Evaluating Experimental Designs Criteria for evaluating experimental researchThe primary purpose is to test causal hypothesesThere should be direct manipulation of the independent variableThere should be clear identification of the specific research design
37 Evaluating Experimental Designs Criteria for evaluating experimental researchThe design should provide maximum control of extraneous variablesTreatments are substantively different from one anotherThe number of subjects is dependent on or equal to the number of treatment replications