1. Experimental Research Designs
Chapter 9
This multimedia product and its contents are protected under copyright law. The following are prohibited by law:
• Any public performance or display, including transmission of any image over a network;
• Preparation of any derivative work, including the extraction, in whole or in part, of any images;
• Any rental, lease, or lending of the program.
Copyright © Allyn & Bacon 2008

2. Experimental Research
Purpose
To make causal inferences about the relationship between the independent and dependent variables
Characteristics
Direct manipulation of the independent variable
Control of extraneous variables
Eliminate the variable from the study
Statistically adjust for the effect of the variable
Copyright © Allyn & Bacon 2008

3. Experimental Research
Difficulties
Difficult to carry out in educational settings
Difficult to control all factors that might affect the outcome
Consumers should know the limitations associated with a study to judge the usefulness of the findings
Copyright © Allyn & Bacon 2008

4. Experimental Research
Interesting websites
A discussion of experimental designs
A non-technical discussion of experimental designs
Some examples of experimental and quasi-experimental designs
Copyright © Allyn & Bacon 2008

5. Experimental Validity
Two issues
Internal validity
The extent to which the independent variable, and not other extraneous variables, produce the observed effect on the dependent variable
External validity
The extent to which the results are generalizable
Copyright © Allyn & Bacon 2008

6. Copyright © Allyn & Bacon 2008
Internal Validity
Internal validity is discussed in terms of factors, called threats, that reduce the level of confidence in any causal conclusions
Eleven (11) specific threats
History
Extraneous events have an effect on the subjects’ performance on the dependent variable
The crash of the stock market, 9-11, the invasion of Iraq, etc.
Copyright © Allyn & Bacon 2008

7. Copyright © Allyn & Bacon 2008
Internal Validity
Eleven (11) specific threats (continued)
Selection
Groups that are initially not equal due to differences in the participants in those groups
Positive and negative attitudes, high and low achievers, etc.
Maturation
Participants’ maturation over the course of the study
Pretesting
The effect of having taken a pretest
Copyright © Allyn & Bacon 2008

8. Copyright © Allyn & Bacon 2008
Internal Validity
Eleven (11) specific threats (continued)
Instrumentation
Poor technical quality (i.e., validity, reliability) or changes in instrumentation
Treatment replications
Insufficient replications of the treatment
Subject attrition
Differential loss of participants from groups
Statistical regression
The natural movement of extreme scores toward the mean
Copyright © Allyn & Bacon 2008

9. Copyright © Allyn & Bacon 2008
Internal Validity
Eleven (11) specific threats (continued)
Diffusion of treatment
The treatment is given, usually inadvertently, to the control group
Experimenter effects
Different characteristics or expectations of those implementing the treatments across groups
Copyright © Allyn & Bacon 2008

10. Copyright © Allyn & Bacon 2008
Internal Validity
Eleven (11) specific threats (continued)
Subject effects
The effects of being aware one is involved in a study
Four types
Hawthorne effect
John Henry effect
Resentful demoralization
Novelty effect
Copyright © Allyn & Bacon 2008

11. Copyright © Allyn & Bacon 2008
Internal Validity
Control of effects through the choice of specific research designs
Interpretation of the results is tempered by the existence of internal validity concerns
Internal validity
Conceptual definition of internal validity
Copyright © Allyn & Bacon 2008

12. Copyright © Allyn & Bacon 2008
External Validity
Five factors affecting external validity
Participants
Representativeness of the sample in comparison to the population
Consistency of the results across subgroups within the sample
Personal characteristics of the participants
Participant’s awareness of being involved in a study
Copyright © Allyn & Bacon 2008

13. Copyright © Allyn & Bacon 2008
External Validity
Five factors affecting external validity
Situations - characteristics of the setting
Specific environment
Special situation
Particular school
Time - explanations can change over time
Treatments - specific way in which an experimental treatment is conceptualized, operationalized, and administered
Copyright © Allyn & Bacon 2008

14. Copyright © Allyn & Bacon 2008
External Validity
Five factors affecting external validity
Measures
Different instruments measure content or constructs differently
Measures chance across studies
Control of both types of threats through sampling procedures
Generalization of results is tempered by external validity concerns
Copyright © Allyn & Bacon 2008

15. Copyright © Allyn & Bacon 2008
Experimental Designs
Four types of designs
Pre-experimental
Quasi-experimental
True experimental
Factorial
Copyright © Allyn & Bacon 2008

16. Copyright © Allyn & Bacon 2008
Experimental Designs
Notation
R indicates random selection or random assignment
O indicates an observation
Test
Observation score
Scale score
X indicates a treatment
A, B, C, ... indicates a group
Copyright © Allyn & Bacon 2008

17. Pre-Experimental Designs
No pre-experimental design controls internal validity threats well
Single group pretest only
A X O
Internal validity threats
History, maturation, attrition, experimenter effects, subject effects and instrumentation are viable threats
Useful only when the research is sure of the status of the knowledge, skill, or attitude being changed and there are no extraneous variables affecting the results
Copyright © Allyn & Bacon 2008

18. Pre-Experimental Designs
Single group pretest posttest
A O X O
Internal validity threats
Maturation and pretesting are threats
History and instrumentation are potential threats
Useful when subject effects will not influence the results, history effects can be minimized, and multiple pretests and posttests are used
Copyright © Allyn & Bacon 2008

19. Pre-Experimental Designs
Non-equivalent groups posttest only
A X O B O
Internal validity threats
Selection is a threat
History, maturation, and instrumentation are potential threats
Useful when groups are comparable and participants can be assumed to be about the same at the beginning of the study
Copyright © Allyn & Bacon 2008

20. Quasi-Experimental Designs
Types
Non-equivalent pretest-posttest, experimental control groups
A O X O B O O
Non-equivalent pretest-posttest, multiple treatment groups
A O X1 O B O X2 O
Useful when participants are in pre-existing groups (e.g., classes, schools, teams, etc.)
Copyright © Allyn & Bacon 2008

21. Quasi-Experimental Designs
Threats to internal validity
Selection is the major concern
Controls for statistical regression
Likely to control for most other threats provided the groups are not significantly different from one another
See Table 9.3 for specific threats related to each design
Copyright © Allyn & Bacon 2008

22. True Experimental Designs
Important terminology
Random assignment
Participants are placed into groups using a random procedure
This ensures equivalency of the groups
Random selection of subjects
Participants are chosen from a population using random procedures
This ensures generalizability to the population from which the participants were selected (i.e., external validity)
Copyright © Allyn & Bacon 2008

23. True Experimental Designs
Types
Randomized posttest only experimental control groups
R A X O R B O
Randomized posttest only multiple treatment groups
R A X1 O R B X2 O
Randomized pretest-posttest experimental control groups
R A O X O R B O O
Copyright © Allyn & Bacon 2008

24. True Experimental Designs
Types (continued)
Randomized pretest-posttest multiple treatment groups
R A O X1 O R B O X2 O
Threats to internal validity
Controls for selection, maturation, and statistical regression
Likely to control for most other threats
See Table 9.3 for specific threats related to each design
Copyright © Allyn & Bacon 2008

25. Copyright © Allyn & Bacon 2008
Factorial Designs
Research designs containing two or more independent variables
A study of the effects of two instructional strategies on male and female students’ math achievement
A study of two counseling approaches on middle and secondary students’ self-esteem
Examples of factorial designs
Copyright © Allyn & Bacon 2008

26. Copyright © Allyn & Bacon 2008
Factorial Designs
Types of effects
Main effects
There is a main effect for each independent variable
For the first example above, there is one main effect for instructional strategy and one main effect for math achievement
For the second example above, there is one main effect for counseling approach and one main effect for school level
Copyright © Allyn & Bacon 2008

27. Copyright © Allyn & Bacon 2008
Factorial Designs
Types of effects (continued)
Interaction effect
A different effect for the level of the first independent variable across the levels of the second independent variable
For the first example above, the first instructional strategy could be effective for males but not females, whereas the second instructional strategy could be effective for females but not males
Copyright © Allyn & Bacon 2008

28. Copyright © Allyn & Bacon 2008
Factorial Designs
Interaction effect (continued)
For the second example above, the first counseling approach could be effective for secondary students but not so for middle school students, while the second strategy could be effective for middle school students but not so for secondary students
In both examples one cannot state the effectiveness of the instructional strategy or counseling method without qualifying it relative to the gender or school level respectively
Copyright © Allyn & Bacon 2008

29. Evaluating Experimental Designs
Criteria for evaluating experimental research
The primary purpose is to test causal hypotheses
There should be direct intervention of the independent variable
There should be clear identification of the specific research design
Copyright © Allyn & Bacon 2008

30. Evaluating Experimental Designs
Criteria for evaluating experimental research
The design should provide maximum control of extraneous/confounded variables
The intervention should be described and implemented as planned
The number of participants is dependent on equal to the number of intervention replications
Copyright © Allyn & Bacon 2008

31. Single Subject Designs
Designs in which the effect of an experimental treatment is studied for one participant
Repeated measurement of the dependent variable before, during, and after implementing treatment
Not restricted to one (1) participant, but rarely involves more than three (3) participants
Used extensively in studies involving exceptional children or counseling
Copyright © Allyn & Bacon 2008

32. Single Subject Designs
Characteristics
Reliable measurement
Repeated measurement
Clear description of the conditions
Baseline and treatment conditions
One variable investigated
Notation
A indicates a baseline condition without treatment
B indicates a treatment condition
Copyright © Allyn & Bacon 2008

33. Single Subject Designs
Two specific designs
A B A
Multiple observations are made during initial baseline time frame; multiple observations during treatment implementation time frame; treatment withdrawn and multiple observations during the second baseline time frame
Variations on this design include A B A B (i.e., including a second treatment phase)
Limitations
Complicated statistical analysis of the data
Interpretation of specific outcomes (e.g., a lasting effect of treatment that does not diminish in the second baseline observations)
Copyright © Allyn & Bacon 2008

34. Single Subject Designs
Two specific designs (continued)
Multiple baseline designs
Extension of the A B A design to include more than one subject, behavior, or setting
These designs enhance the generalizability of the results
Copyright © Allyn & Bacon 2008

35. Single Subject Designs
Criteria for evaluating single subject research
Reliable measurement of the target behavior
Target behavior is clearly defined in operational terms
Sufficient measurements are made during each time frame to establish stability
Copyright © Allyn & Bacon 2008

36. Single Subject Designs
Criteria for evaluating single subject research
Full descriptions of the procedures, participants, and settings are provided
Use of one (1) standard treatment
Control of experimenter and/or observer effects
Results should have practical significance
Copyright © Allyn & Bacon 2008