1. Experimental, Quasi-experimental, and Single Subject Research
774/801 Sept 1, 2004
John Hattie & Tony Hunt

2. It is simple: There is no perfect experiment in education
There is nearly always a trade off between the
Power to generalise
- from sample to population
- from items to the behaviour domain
- from conditions in the study to all intended conditions
and the
Power to convince
- there are many audiences PG PC

3. PG
Power to Generalise PC
How confident can be generalise from the study to all “similar” situations
Is the design replicable/reproducible/exchangeable?
Is the evidence/conclusions unique to this study?
Have the generalisations taken into account all possible competing views – plausible alternative rival explanations (PARE)

4. Power to convince Who are we trying to convincePC PG
Power to convince
Who are we trying to convince
If it is a colleague(s) then more situation specificity may be convincing (kids/classrooms/schools like mine)
If it is the educational community, then situation needs to be less critical

5. Resolution: Linking Power
Experimental design consists of a series of links:
It is as strong as the weakest link
Each link influences the next link
Desirable to have equal strength
Does each link have explanatory power
Are conclusions credible to the intended audience

6. History Stanley and Campbell (1963) Cook and Campbell (1979)
Shadish, Cook & Campbell (2002)
Evidence based
All based on designing studies that can lead to explanation and claims of causality

7. Explanation and Cause Cause and effect must be related
(e.g., self-concept & achievement)
There needs to be temporal order (cause before effect)
Need to rule out other explanations/ other Plausible Alternative Rival Explanations (PARE)

8. Campbell & Stanley (1963) Pretest-Posttest Control Group Design
Pre Treatment Post
R O X O
R O O
Randomisation – aiming for representativeness

9. But can we randomise
No Child Left Behind
Tennessee Class Size Study

10. Quasi-experimentation:
When you do not have so much control over allocation of treatment, conditions, sample
When you have non-equivalent groups
In quasi-experimentation, the researcher has to enumerate alternative explanations one by one, decide which are plausible, and then use logic, design, and measurement to assess whether each one is operating in a way that might explain any observed effect (Shadish, Cook & Campbell, 2002, p. 14)
Relates to the Popper notion of falsification: What evidence would you accept that you are wrong?

11. Examples of Quasi-experimental designs
a. Time Series
O1 O2 O3 O4 O5 X O6 O7 O8 O9
Divorce Laws
Ozdowski, S.A. & Hattie, J.A. (1981). The impact of divorce laws on divorce rate in Australia: A time series analysis. Australian Journal of Social Issues, 16, 3-17.

12. ABA design. A. B. A. O1. O2. O3. X4. X5. X6. O7. O8. O9 Le Fevre, et
ABA design A B A O1 O2 O3 X4 X5 X6 O7 O8 O9 Le Fevre, et.al. (2002). Adequate Decoders

13. Multilevel Design: Hierarchical Linear Modelling
Students within classes within schools
E.g., Tracking/Streaming
School 1 School 2…
Teacher 1 Teacher 2
Class 1 Class 2 Class 1 Class 2

14. Structural Equation Modelling

15. Minimal requirements for Studies
Sampling
Items to behaviour domains
People to all possible people
Conditions to all possible conditions
Representative sampling via
Random sampling
Stratified random sampling

16. Variables
At the end of your study, can I say “Aha, so that is what you mean, now I am clear”
Open constructs NOT Definitions
No such thing as immaculate perception
Dependent - Manipulable
Independent - Nonmanipulable

17. Dependability
How reliable/consistent/replicable are your measures/ observations

18. Validity = Interpretations
Validity - "an integrated evaluative judgement of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores or other modes of assessment".
Not validity of a test, but validity of interpretations

19. Validity of your study …
Is related to having ruled out
Plausible Alternative Rival Explanations (PARE)
CONTROL CONTROL CONTROL
… some examples

20. 1. PARE: Power
Is your study POWERFUL enough to detect the effect you are investigating

21. Do chickens have lips? 1. PARE: Power
Is your study POWERFUL enough to detect the effect you are investigating
Do chickens have lips?

23. 2. PARE: Chance Did the effect/conclusion occur by chance
E.g., That two means are the same – the hypothesis of no difference
Setting a rejection level, say a =.05

24. 3. PARE: Type II errors
Type I errors – Rejecting a claim when it is true (a =.05)
Type II errors – Accepting a claim when it is false (e.g., chickens do not have lips, if it is indeed true)

25. 4. PARE Reliability of your measures
If the reliability is low, then the scores “wobble” and no guarantee you will get same results using these instruments (tests, observations, interviews, etc.)
Was the treatment “consistent” in the various classes/implementations?

26. 5. PARE: Was the treatment implemented?
Degree of implementation
The Hong Kong Practical Science Study (Cheung, Hattie, & Bucat, 1997)

27. 6. PARE: Maturation
Showing change may not be enough as kids improve anyway (e.g., by maturation)
Method to measure change = Effect-sizes
Post-Pre/spread = Effect-size
X2 – X1
sddiff
e.g., Before = 12, After = 15, spread = 6
= .5 6

28. Distribution of effects
Average effect
Zero achievement

29. Distribution of effects
Maturation

30. The disasters … 71 programmed instruction 801 .14 72 finances 1634 73
problem based learning 41
.12 74
diet
255 75
gender (female-male)
9020
.09 76
inductive teaching
570
.06 77
team teaching 78
ability grouping
3355
.05 79
class size
2559 80
open vs. traditional
3426
-.01 81
summer vacation
269
-.06 82
retention
3626
-.17 83
transfer of school
354
-.26 84
disruptive students
1511
-.78

31. The also rans … 56 metacognitive intervention 921 .29 57 math programs
3326
.27 58
audio-visual
2699
.26 59
gifted programs 47
.25 60
coaching
1076
.24 61
behavior objectives
157 62
calculators
238 63
mainstreaming
1641
.21 64
questioning
493
.20 65
learning hierarchies
168
.19 66
attitude to math
1122 67
desegregation
1590
.18 68
play
129
.16 69
television
4337
.15

32. Almost there … 42 tutoring 136 .35 43 activity-based programs 674 44
remedial programs
1438 45
classroom climate
2726 46
social skills training
5472 47
time
1680
.34 48
CAI
18231
.32 49
inquiry based teaching
2740 50
preschool
242 51
whole language
198
.31 52
within class grouping
2359 53
testing
1463 54
problem solving
1141
.30 55
background
692

33. In the middle … parent involvement 2597 .46 bilingual programs 150129
parent involvement
2597
.46 30
bilingual programs
1501 31
adjunct aids
659
.45 32
concept mapping 18 33
advance organizers
2106
.44 34
hypermedia instruction
317 35
socio economic status
1657 36
perceptual-motor skills
7592
.42 37
individualised instruction
5948 38
homework
568
.41 39
competitive learning
144 40
simulations
972
.37 41
expectations
912
.36

34. Worth having … self-assessment 152 .54 mastery learning 1933 .5314
self-assessment
152
.54 15
mastery learning
1933
.53 16
creativity programs
2340
.52 17
interactive video
1152 18
psycho-linguistics
4404
.51 19
goals
959 20
peer influence
366
.50 21
early intervention
30971
.49 22
outdoor education
294 23
science
4124 24
inservice ed
18644
.48 25
acceleration
371
.47 26
motivation
2196

35. The MAJOR Influences … Direct instruction 1925 .93 Reciprocal teaching
# effects
Mean 1
Direct instruction
1925
.93 2
Reciprocal teaching 52
.86 3
Feedback
13209
.81 4
Cognitive strategy training
7649
.80 5
Classroom behaviour
361
.71 6
Prior achievement
2094 7
Phonological awareness
2630
.70 8
Home encouragement
25706
.69 9
Piagetian programs
786
.63 10
Cooperative learning
1153
.59 11
Reading programs
14945
.58 12
Quality of teaching
808
.55 13
Study skills
3224
.54

36. Identifying that what matters

37. 7. PARE: Testing
People become test wise and/or may respond different when under test conditions
White space and testing in asTTle
Testwiseness

38. Test of Objective Evidence Each of the questions in the following set has a logical or "best" answer from its corresponding multiple choice answer set. Please record your eight answers.
The purpose of the cluss in 2 Trassig is true when
furmpaling is to remove
A clump trasses the von
A cluss-prags B the viskal flans, if the viskal is
B tr s donwil or zortil
C cloughs C the belgo fruls
D pluomots D dissels lisk easily
3 The sigia frequently overfesks the 4. The fribbled breg will minter best
trelsum because with an
A all sigias are mellious A derst
B all sigias are always votial B morst
C the trelsum is usually tarious C sortar
D no trelsa are feskable D ignu

39. Test of Objective Evidence, Part II
The reasons for tristal doss are 6 Which of the following is/are
always present when trossels are
being gruven?
A the sabs foped and the doths tinzed A rint and vost
B the dredges roted with the crets B vost
C few rakobs were accepted in sluth C shum and vost
D most of the polats were thonced D vost and plone
7 The mintering function of the ignu is most 8
effectively carried out in connection with
A a razma toi A
B the groshing stantol B
C the fribbled breg C
D a frailly sush D

40. 8. PARE Statistical Regression
When taking extreme groups the means tend to move to the middle.
Why do the tallest fathers have shorter sons, and the shortest fathers have taller sons?

41. …. Regression to the Mean
Special Education (e.g., Sesame Street)
Effective schools
Gifted education

42. 9. PARE Response rates The returns of questionnaires/tests/interviews
should be high
What is typical?

43. Meta-analyses of Response Rates
Typical return is 50%
Three major factors:
Salience (77% vs 42%)
Number of follow ups (halve each time)
Lack of clutter/ orderliness
Not length (ave 7 pages, 72 questions), colour,

44. 10. Change scores The difference between post-pre scores Problems
Unreliable
Are you measuring same thing both times
Regression to the mean

45. 11 PARE: Experimenter effects
Hawthorne effect: Because we know we are in an experiment this alters our responses
Hans the Horse
Pygmalion in the classroom
Christine Rubie’s thesis
Stanley Milgrim’s experiment

46. 12. PARE: Restriction of range
When you choose/focus on a narrow range of abilities (etc.) this can be misleading
Picture …

49. 13. PARE: Specification of target and accessible sample/population
Most experiments are highly local but have general aspirations
Often, there are two groups you are generalising to: e.g., all secondary students in NZ, and to all secondary students you have access -- from which to sample

51. 14: Interactions
The model of individual differences indicates that we should modify our teaching methods to allow for individual differences in the class

52. Maximising the chance that the conclusions are defensible and
The art of research design is to devise experiments to identify the explanation and cause of effects – by
Maximising the chance that the conclusions are defensible and
Minimising the PAREs
Such that you have
Power to Generalise and
Power to Convince

53. Unobtrusive measures Which painting do most people watch?
Friendship in cities
Racism in suburbs/cities

54. Statistical Methods to assist …
Correlation
Analysis of variance (anova)
Cross-tabulation

57. Comparing means: Magnitude and Chance
Magnitude Effect-sizes
Chance Analysis of Variance

58. Well-being
What are the differences in levels of WELL-BEING among males and females, and between Australia and New Zealand
Country * GENDER Cross tabulation
Count
GENDER
MALE FEMALE Total
Country New Zealand
Australia
Total

59. Australia Mn sd
Effect-size
Male
45.7
10.6
Female
46.2
0.04
Total
46.0
New Zealand
53.6
7.5
54.3
7.4
0.08
54.0
NZ - Australia
0.89

60. anova Source df MS F p Country 1 35211.9 416.71 <.001 Gender 151.8
1.80
.180
Country * Gender
6.1
0.07
.787
Error
2271
84.5