1. © LOUIS COHEN, LAWRENCE MANION AND KEITH MORRISON
EXPERIMENTS
© LOUIS COHEN, LAWRENCE MANION AND KEITH MORRISON

2. STRUCTURE OF THE CHAPTER
Randomized controlled trials
Designs in educational experiments
True experimental designs
Quasi-experimental designs
Single-case ABAB design
Procedures in conducting experimental research
Threats to internal and external validity in experiments
The timing of the pre-test and the post-test
The design experiment
Internet-based experiments
Ex post facto research

3. CAUSALITY
Experiments are held up to be able to identify causality through control and manipulation of variables.
Examine the effect of an independent variable on a dependent variable.
Identifying the effects of causes by implementing interventions in a controlled environment.
Held up to be able to offer explanations for outcomes.

4. INDEPENDENT AND
DEPENDENT VARIABLES

5. RANDOMIZATION
Random sampling of a population and random allocation to either a control or an experimental group.
Randomization allows for the many additional uncontrolled and, hence, unmeasured variables that may be part of the make-up of the groups in question.
Randomization operates the ceteris paribus condition (all other things being equal), assuming that the distribution of extraneous variables is more or less even and perhaps of little significance.
Randomization strives to address Holland’s (1986) ‘fundamental problem of causal inference’, which is that a person may not be in both a control group and an experimental group simultaneously.

6. CONCERNS IN EXPERIMENTS
It may not be possible or desirable to isolate, control and manipulate variables and people under laboratory conditions.
The ‘real’, social world is not the antiseptic, sealed, artificial world of the laboratory. The social world is far more complex.
Cannot assume that a single cause produces a single effect.
Measurements are of averages, overlooking distributions, outliers, intervention-response differences, within-group differences, between-group differences and sub-sample differences.
The setting affects the outcomes.
Limited generalizability in practice.

7. HOW TO JUDGE ‘WHAT WORKS’
Null hypothesis significance testing (which is problematical)
Effect size
Statistical power
The subtraction approach
Considering rival explanations
The contingency approach
The ethical dimension

8. BLIND AND DOUBLE-BLIND EXPERIMENTS
Blind experiment: participants do not know to which group they are assigned.
Double blind experiment: neither the researcher nor the participants know to which group the participants are assigned.

9. KINDS OF EXPERIMENT
Pretest-post-test control and experimental group
Two control groups and one experimental group pretest-post-test
Post-test control and experimental group
Post-test two experimental groups
Pretest-post-test two treatment
Matched pairs;
Factorial design;
Parametric design;
Repeated measures design;
Laboratory experiments (controlled, artificial conditions):
one-group pretest-post-test;
non-equivalent control group design;
time series
Field experiments (controlled conditions in the ‘real world’):
Natural experiments (no control over real world conditions)

10. FEATURES OF A TRUE EXPERIMENT
Random selection of sample.
Random allocation of sample to control or experimental groups.
Identification and isolation of key variables.
Control of the key variables.
Exclusion of any other variables.
Special treatment (the intervention) given to the experimental group (i.e. manipulating the independent variable) whilst holding every other variable constant for the two groups.
Ensuring that the two groups are entirely separate throughout the experiment (non-contamination).
Final measurement of outcomes to compare control and experimental groups and for differences from the pre-test results (the post-test).
Comparison of one group with another.

11. Stages in an experiment Randomly assign subjects
to two matched groups:
control and experimental group
Conduct pre-test
Isolate and control variables,
exclude other variables
Administer intervention to
experimental group
Conduct post-test and compare
control and experimental groups

12. ‘TRUE’ EXPERIMENTAL DESIGN
Intervention
EXPERIMENT
PLUS
Matched on
pre-test
Post-test
Isolate,
control and
manipulate
variables
Random group
assignment
CONTROL
CONTROL

13. MEASURING EFFECTS Average causal effect (A): where:
= (E1E2)  (C1C2)
where:
E1 = post-test for experimental group;
E2 = pre-test for experimental group;
C1 = post-test for control group;
C2 = pre-test for control group.

14. CAMPBELL’S AND STANLEY’S NOTATION
X represents the exposure of a group to an experimental variable or event, the effects of which are to be measured.
O refers to the process of observation or measurement.
Xs and Os in a given row are applied to the same persons.
Left to right order indicates temporal sequence.
Xs and Os vertical to one another are simultaneous.
R indicates random assignment to separate treatment groups.
Parallel rows unseparated by dashes represent comparison groups equated by randomization, while those separated by a dashed line represent groups not equated by random assignment.

15. CAMPBELL’S AND STANLEY’S SYMBOLIC REPRESENTATION OF ‘TRUE’ EXPERIMENTS
RO1 X O2
RO3 O4
(Campbell, D. T. and Stanley, J. (1963) Experimental and Quasi-experimental Designs for Research on Teaching. Boston: Houghton Mifflin Co.)

16. Control1 RO3 RO4 Control2 X RO5
TWO CONTROL GROUPS AND ONE EXPERIMENTAL GROUP PRE-TEST–POST-TEST DESIGN
Experimental RO1 X RO2
Control1 RO3 RO4
Control2 X RO5

17. THE POST-TEST CONTROL AND EXPERIMENTAL GROUP DESIGN
Experimental R1 X O1 Control R 2 O2

18. THE POST-TEST TWO EXPERIMENTAL GROUPS DESIGN
Experimental1 R1 X1 O1 Experimental2 R2 X2 O2

19. THE PRE-TEST–POST-TEST TWO TREATMENT DESIGN
Experimental1 RO1 X1 O1 Experimental2 RO3 X2 O4

20. THE TRUE EXPERIMENT ONE CONTROL AND TWO EXPERIMENTAL GROUPS
Experimental1 RO1 X1 O1 Experimental2 RO3 X2 O4 Control RO5 O6

21. THE PRE-TEST TWO TREATMENT DESIGN
Experimental1 RO1 X1 O1 Experimental2 RO3 X2 O4

22. MATCHED PAIRS DESIGN Stage 1 Stage 2. Stage 3. Stage 4 Stage 5
Step One: Measure the dependent variable.
Stage 2.
Step Two: Assign participants to matched pairs, based on the scores and measures established from Step One.
Stage 3.
Step Three: Randomly assign one person from each pair to the control group and the other to the experimental group
Stage 4
Step Four: Administer the intervention to the experimental group and, if appropriate, a placebo to the control group. Ensure that the control group is not subject to the intervention.
Stage 5
Step Five: Carry out a measure of the dependent variable with both groups and compare/measure them in order to determine the effect and its size on the dependent variable.

23. 9 combinations: 1+4; 1+5; 1+6; 2+4; 2+5; 2+6; 3+4; 3+5; 3+6
FACTORIAL DESIGN
Performance in an examination may depend on availability of resources and motivation for the subject studied
INDEPENDENT VARIABLE
LEVEL
ONE
TWO
THREE
Availability of resources
limited availability (1)
moderate availability (2)
high
availability (3)
Motivation for the subject studied
little
motivation (4)
moderate motivation (5)
motivation (6)
9 combinations: 1+4; 1+5; 1+6; 2+4; 2+5; 2+6; 3+4; 3+5; 3+6

24. Factorial designs must address the interaction of the independent variables.
Difference for motivation in mathematics is not constant between males and females, but varies according to age of participants: an interaction effect (age and sex).

25. PARAMETRIC DESIGN
Participants are randomly assigned to groups whose parameters are fixed in terms of the levels of the independent variable that each receives.
Parametric designs are useful if an independent variable has different levels or a range of values which may have a bearing on the outcome (confirmatory research) or if the researcher wishes to discover whether different levels of an independent variable have an effect on the outcome (exploratory research).

26. REPEATED MEASURES
Participants in the experimental groups are tested under two or more experimental conditions.
The order in which the interventions are sequenced may have an effect on the outcome (e.g. the first intervention may have an influence – a carry-over effect – on the second, and the second intervention may have an influence on the third).
Early interventions may have a greater effect than later interventions.
Repeated measures designs are useful if it is considered that order effects are either unimportant or unlikely.

27. REPEATED MEASURES (two groups receiving both conditions)
With no intervention
Matched on pre-test
Random allocation to groups
Group 2
With intervention
Post-test

28.       Independent groups Noise condition No noise condition
  
Sara Rob Peter
Jane Jack Jim
  
Joan Susan John
Lyn Sally Alan

29.       Repeated measures Noise condition No noise condition
  
Sara Rob Peter
Jane Jack Jim
  
Joan Susan John
Lyn Sally Alan
Sara Rob Peter

30. QUASI-EXPERIMENTS: NON-EQUIVALENT CONTROL GROUP DESIGN
Pre-experimental design: the one-group pre-test–post-test
Experimental O1 X O2
Pre-experimental design: the one-group post-test only design
Experimental O1
The post-tests only non-equivalent groups design
Experimental O1
Control O2

31. QUASI-EXPERIMENTS: NON-EQUIVALENT CONTROL GROUP DESIGN
The pre-test–post-test non-equivalent group design
Experimental O1 X O2
Control O3 O4

32. PROCEDURES IN CONDUCTING EXPERIMENTS1
Identify research problems 2
Formulate hypotheses 3
Select appropriate levels at which to test the independent variables 4
Decide which kind of experiment to adopt 5
Decide population and sampling 6
Select instruments for measurement 7
Decide how the data will be analyzed 8
Pilot experimental procedures 9
Carry out the refined procedures 10
Analyze results 11
Report the results

33. A TEN-STEP MODEL FOR CONDUCTING EXPERIMENTS
Identify the purpose of the experiment.
Step 2
Select the relevant variables.
Step 3
Specify the level(s) of the intervention (e.g. low, medium high intervention).
Step 4
Control the experimental conditions and environment.
Step 5
Select appropriate experimental design.
Step 6
Administer the pretest.
Step 7
Assign the participants to the group(s).
Step 8
Conduct the intervention.
Step 9
Conduct the post-test.
Step 10
Analyze the results.

34. PROCEDURES IN CONDUCTING EXPERIMENTS: HYPOTHESES
Null hypothesis (H1).
Alternative hypothesis (H0).
Direction of hypothesis: states the kind of difference or relationship between two conditions or two groups of participants.
One-tailed (directional): e.g. ‘people who study in silent surroundings achieve better than those who study in noisy surroundings’.
Two-tailed (no direction): e.g. ‘there is a difference between people who study in silent surroundings and those who study in noisy surroundings’.

35. OPERATIONALIZING HYPOTHESES
Hypothesis: ‘people who study in quiet surroundings achieve better than those who study in noisy surroundings’.
What do ‘work better’, ‘quiet’ and ‘noisy’ mean? Define the operations:
‘work better’ = obtain a higher score on the Wechsler Adult Intelligence Scale
‘quiet’ = silence
‘noisy’ = CD music playing
Operationalized hypothesis: ‘people who study in silence achieve a higher score on the Wechsler Adult Intelligence Scale than those who study with CD music playing’.

36. DIRECTIONAL AND NON-DIRECTIONAL HYPOTHESES
People who do homework without the TV on produce better results than those who do homework with the TV on.
Directional (one-tailed)
There is a difference between work produced in noisy or silent conditions.
Non-directional (two-tailed)

37. TESTING VALIDITY AND RELIABILITY
MATURATION
HISTORY
TESTING
DIRECTION
OF CAUSALITY
INSTRUMENT-
ATION
THREATS TO
VALIDITY AND
RELIABILITY
TYPE I AND
TYPE II
ERRORS
EXPERIMENTAL
MORTALITY
OPERATIONAL-
IZATION
CONTAMIN-
ATION
REACTIVITY

38. TIMING OF PRE-TEST AND POST-TEST
As close to the start of the experiment as possible (to avoid contamination of other variables).
Post-test
As close to the end of the intervention as possible.
Too soon a post-test
Misses long-term/delayed effect and only measures short-term gain (which may be lost later).
Too long a time lapse before a post-test
Becomes impossible to determine whether it was a particular independent variable that caused a particular effect, or whether other factors have intervened since the intervention, to produce the effect.

39. ADVANTAGES OF INTERNET-BASED EXPERIMENTS
Attraction over laboratory and conventional experiments
Greater generalizability because of their wider sampling
Demonstrate greater ecological validity as they are often conducted in settings which are familiar to the participants and at times suitable to the participants
Have a high degree of voluntariness

40. FOUR TYPES OF INTERNET-BASED EXPERIMENTS
Those that present static printed materials (e.g. printed text or graphics)
Those that make use of non-printed materials (e.g. video or sound)
Reaction-time experiments
Experiments that involve some form of interpersonal interaction

41. INTERNET-BASED EXPERIMENTS
Check download speeds and time, anticipate problems of different browsers and platforms.
Can experience greater problems of dropout than conventional experiments.

42. EX POST FACTO RESEARCH Co-relational and criterion groups designs
Characteristics of ex post facto research
Occasions when appropriate
Advantages and disadvantages of ex post facto research
Designing an ex post facto investigation
Procedures in ex post facto research

43. TWO APPROACHES TO EX POST FACTO RESEARCH
Commence with subjects who differ on an independent variable, for example, their years of study in mathematics, and then study how they differ on the dependent variable, for example, a mathematics test.
Commence with subjects who differ on the dependent variable (e.g. their performance in a mathematics test) and discover how they differ on a range of independent variables, for example, their years of study, their liking for the subject, the amount of homework they do in mathematics.

44. EX POST FACTO RESEARCH AND INDEPENDENT VARIABLES
Differing on the independent variable:
Presence of independent variable
Absence of independent variable
Degrees of independent variable
Investigate
Effect on the dependent variable
Same on the independent variable(s)
Investigate
Effect on the dependent variable

45. EX POST FACTO RESEARCH AND DEPENDENT VARIABLES
Differing on the dependent variable
Differing on independent variables:
Presence of independent variables
Absence of independent variables
Degrees of independent variables
Investigate
Same on the dependent variable
Differing on independent variables:
Presence of independent variables
Absence of independent variables
Degrees of independent variables
Investigate

46. CO-RELATIONAL AND CRITERION GROUP STUDY
Co-relational study (causal research)
to identify the antecedents of a present condition.
collect two sets of data, one of which will be retrospective, with a view to determining the relationship between them.
Criterion group study (causal-comparative research)
to discover possible causes for a phenomenon being studied.
Compare the subjects in which the variable is present with similar subjects in whom it is absent.

47. TWO CAUSES AND TWO EFFECTS IN CRITERION GROUP STUDY
Effective teaching
Ineffective teaching
EFFECT
POSSIBLE CAUSE
Presence of collegial curriculum planning
Absence of collegial curriculum planning
Two criterion groups: (a) Presence of collegial planning (b) Absence of collegial planning
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

48. CHARACTERISTICS OF EX POST FACTO RESEARCH
In ex post facto research the researcher takes the effect (or dependent variable) and examines the data retrospectively to establish causes, relationships or associations, and their meanings.

49. EX POST FACTO RESEARCH IS USEFUL WHEN . . .
. . . it is impossible, impractical, costly or unethical to conduct an experiment;
. . . it is not possible to select, control and manipulate the factors necessary to study cause-and-effect relationships directly;
. . . the control of all variables except a single independent variable may be unrealistic and artificial;
. . . the independent variable lies outside the researcher’s control.

50. ADVANTAGES OF EX POST FACTO RESEARCH
Useful where the more rigorous experimental approach is not possible.
Useful to study what goes with what and under what conditions.
Useful where the setting up of the latter would introduce a note of artificiality into research proceedings.
Useful where simple cause-and-effect relationships are being explored.
It can give a sense of direction and provide a source of hypotheses that can subsequently be tested by the more rigorous experimental method.

51. DIFFICULTIES IN EX POST FACTO RESEARCH
Direction of causality difficult to establish: what caused what.
Lack of control of the independent variable or variables.
Impossible to isolate and control every possible variable, or to know with absolute certainty which are the most crucial variables.
Randomization impossible.
Can provide support for any number of different, even contradictory, hypotheses.
Correlation does not equal cause.
Lack of control: the researcher cannot manipulate the independent variable or randomize her subjects.

52. DISADVANTAGES OF EX POST FACTO RESEARCH
One cannot know for certain whether the causative factor has been included or even identified.
It may be that no single factor is the cause.
A particular outcome may result from different causes on different occasions.
It is not possible to disconfirm a hypothesis.
Classifying into dichotomous groups can be problematic.
As the researcher attempts to match groups on key variables, this leads to shrinkage of sample.
Conclusions may be based on too limited a sample or number of occurrences.
It may fail to single out the really significant factor(s).

53. DESIGN AND PROCEDURES IN AN EX POST FACTO INVESTIGATION
Identify the problem area to be investigated.
Formulate a hypothesis to be tested or questions to be answered.
Make explicit the assumptions on which the hypothesis and subsequent procedures will be based.
Review of the research literature will follow to ascertain the kinds of issues, problems, obstacles and findings disclosed by previous studies in the area.
Plan the actual investigation:
identify the population and samples;
select and construct techniques for collecting data;
establish categories for classifying the data.
Describe, analyse and interpret the findings.

54. DESIGN AND PROCEDURES IN AN EX POST FACTO INVESTIGATION
Stage 1
Define the problem and survey the literature
Stage 2
State the hypotheses and the assumptions or premises on which the hypotheses and research procedures are based
Stage 3
Select the subjects (sampling) and identify the methods for collecting the data
Stage 4
Identify the criteria and categories for classifying the data to fit the purposes of the study

55. DESIGN AND PROCEDURES IN AN EX POST FACTO INVESTIGATION
Stage 5
Gather data on those factors which are always present in which the given outcome occurs, and discard the data in which those factors are not always present
Stage 6
Gather data on those factors which are always present in which the given outcome does not occur
Stage 7
Compare the two sets of data (i.e. subtract the former (Stage Five) from the latter (Stage Six), in order to infer the causes that are responsible for the occurrence or non-occurrence of the outcome
Stage 8
Analyze, interpret and report findings

56. CONTROLS IN EX POST FACTO RESEARCH
Match the subjects in the experimental and control groups where the design is causal-comparative.
Build the extraneous independent variables into the design and then use an analysis of variance technique.
Select samples that are as homogeneous as possible on a given variable.
State and test alternative hypotheses that might be plausible explanations for the empirical outcomes of the study.