1. Chapter 9 Experimental Research Gay, Mills, and Airasian
Educational Research
Chapter 9
Experimental Research
Gay, Mills, and Airasian

2. Topics Discussed in this Chapter
Defining characteristics of experimental research
Conducting experimental research
Manipulation and control
Threats to validity
Internal validity
External validity
Group designs

3. Defining Characteristics
Research designed to investigate cause and effect relationships through the direct manipulation of an independent variable and control of extraneous variables
Independent variable – the variable being manipulated
Dependent variable – the variable in which the effect of the manipulation of the independent variable is observed
Researcher manipulation and control – choice of treatments, choice of a research design, use of specific procedures, etc.
Objectives 1.1 & 1.2

4. Experimental Process Six steps to conducting experimental research
Selection and definition of the problem
Statement of a hypothesis indicating a causal relationship between variables
Selection of participants and instruments
Random selection of a sample of subjects from a larger population
Random assignment of members of the sample to each group
Selection of valid and reliable instruments
Objectives 2.1 & 2.2

5. Experimental Process
Six steps to conducting experimental research (cont.)
Selection of a research plan
Three types of comparisons
Comparison of two different approaches
Comparison of new and existing approaches
Comparison of different amounts of a single approach
Execution of the research plan
Two concerns
Sufficient exposure to the treatment
Substantively different treatments
Objectives 2.1, 2.3, & 2.6

6. Experimental Process
Six steps to conducting experimental research (cont.)
Analysis of data
Formulation of conclusions
Objective 2.1

7. Manipulation and Control
The researcher’s decisions related to what constitutes the independent variable
Active and assigned variables
Active variables are those the researcher actively manipulates
Choice of an instructional strategy
A particular counseling approach
Assigned variables are those that cannot be manipulated by the researcher but are of interest
Gender
Race
Objectives 3.1 & 3.2

8. Manipulation and Control
The researcher’s efforts to remove the influence of any extraneous variables that might have an effect on the dependent variable
The goal is to be assured the only differences between groups is that related to the independent variable
Participant variables – characteristics of the subjects
Pre-existing achievement levels
Differences in attitudes
Environmental variables – characteristics of the context
Learning materials
Differences in the time available for treatment between groups
Objectives 3.3 & 3.4

9. Experimental Validity
Internal validity – the degree to which the results are attributable to the independent variable and not some other rival explanation
External validity – the extent to which the results of a study can be generalized
Population validity – generalizations related to other groups of people
Ecological validity – generalizations related to other settings, times, contexts, etc.
Objectives 4.1, 4.2, 6.1, & 6.2

10. Experimental Validity
Relative importance of internal and external validity
Internal and external validity are related reciprocally
Controlling internal validity decreases external validity
Controlling external validity decreases internal validity
First demonstrate an effect in a highly controlled environment (i.e., prioritize internal validity)
Second replicate the study in a more realistic, natural setting (i.e., prioritize external validity)
Objective 4.3 & 4.4

11. Threats to Internal Validity
History
Maturation
Testing
Instrumentation
Statistical regression
Differential selection of participants
Mortality
Selection-maturation interaction, etc.
Objectives 5.1 – 5.8

12. Threats to External Validity
Pre-test treatment interaction
Multiple treatment interference
Selection treatment interaction
Specificity of variables
Participants
Operational definition of the treatment
Operational definition of the dependent variable
Specific times
Specific circumstances
Treatment diffusion
Objectives 6.3 – 6.7

13. Threats to External Validity
Experimenter effects
Reactive arrangements
Artificial environment
Hawthorne effect
John Henry effect
Placebo effect
Novelty effect
Objectives 6.8 & 6.9

14. Controlling for Extraneous Variables
Extraneous variables must be controlled to be able to attribute the effect to the treatment
Group equivalency must be assured
Four major means to achieve control
Randomization
Selection – controls for representation
Assignment – controls for group equivalency
Objectives 2.5, 7.1, & 8.1

15. Controlling for Extraneous Variables
Matching
Identifying pairs of subjects “matched” on specific characteristics of interest
Randomly assigning subjects from each pair to different groups
Difficulty with subjects for whom no match exists
Comparing homogeneous groups
Restricting subjects to those with similar characteristics
Restricting subjects results in problems related to generalization
Objectives 8.4 & 8.5

16. Controlling for Extraneous Variables
Using subjects as their own controls
Multiple treatments across time
Problem with carry-over effects
Analysis of covariance (ANCOVA)
Statistically adjusting the posttest scores for the subjects in each group for pretest differences that existed at the beginning of the study
Creates statistically equivalent groups
Objectives 8.6 & 8.7

17. Controlling for Extraneous Variables
Other ways to control extraneous variables
Holding variables constant
Using only males rather than males and females
Selecting teachers with only similar levels of experience
Selecting only one grade level
Stipulating the specific length of a treatment
Objective 8.2

18. Group Designs Two major classes of group designs
Single-variable designs – one independent variable
Factorial designs – two or more independent variables
Three types of experimental designs
Pre-experimental designs
Experimental designs
Quasi-experimental designs
Objectives 9.1, 9.2, 10.1, & 10.2

19. Pre-Experimental Designs
Three types
One-shot case study
X O
One-group pretest-posttest design
O X O
Static group comparison
X1 O
X2 O
Threats to internal validity – see Figure 9.1
Objective 9.2

20. True Experimental Designs
Three types
Pretest-posttest control group design
R O X O
R O O
Posttest only control group design
R X O
R O
Objective 9.4

21. True Experimental Designs
Three types (cont.)
Solomon four-group comparison
R O X O
R O O
R X O
R O
Threats to internal validity – see Figure 9.2
Objective 9.4

22. Quasi-Experimental Designs
Three types
Non-equivalent control group design
O X O
O O
Time series design
O O O O X O O O O
Counterbalanced design
O X1 O X2 O X3 O
O X3 O X1 O X2 O
O X2 O X3 O X1 O
Threats to internal validity – see Figure 9.2
Objective 9.5

23. Factorial Designs Two independent variables and one dependent variable
The effect of teaching strategy and gender on students’ achievement
The effect of a particular counseling technique and the clients’ ethnicity on the success of the treatment
The effect of a specific coaching approach and children in three age groups on the ability to perform certain physical tasks
Objective10.1

24. Factorial Designs
This design increases explained variance and reduces unexplained variance
Explained variance is that which can be accounted for by the independent variable(s)
By adding an additional variable into the design the explained variance is likely going to increase
Unexplained variance is that which cannot be accounted for by the independent variable(s)
Because it is unexplained, the researcher assumes this is due to error
This is also known as error variance
Objectives 10.1 & 10.2

25. Factorial Designs Interaction
The degree to which changes in the dependent variable are different depending on the levels of each of the independent variables
A particular instructional strategy is more effective for males than females
A particular counseling technique is more effective when the ethnicity of the counselor and client are similar
Objective 11.1

26. Factorial Designs Interaction
Visually explained by a graph of performance of all levels of both independent variables
Parallel lines indicate no interaction
Non-parallel lines indicate an interaction
Statistically examined with an F-test of the statistical significance of the interaction
Objective 11.1

27. Factorial Designs Interaction
Visual presentation of a significant interaction
Statistical analysis of this significant interaction: F1,16 = 4.88 (p = .0420)
Objective 11.1

28. Factorial Designs Interaction
Visual presentation of a non-significant interaction
Statistical analysis of this non-significant interaction: F1,16 = 0.97 (p = .340)
Objective 11.1