Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 Repeated-measures data in educational research trials – how should it be analysed? Ben Styles Senior Statistician National Foundation for Educational.

Similar presentations


Presentation on theme: "1 Repeated-measures data in educational research trials – how should it be analysed? Ben Styles Senior Statistician National Foundation for Educational."— Presentation transcript:

1 1 Repeated-measures data in educational research trials – how should it be analysed? Ben Styles Senior Statistician National Foundation for Educational Research

2 2

3 3 Two sweeps example Cluster randomised trial of reading materials Baseline reading test, 10 week intervention, follow-up reading test Two parallel versions of the Suffolk Reading Scale

4 4 Using baseline data as a covariate in a multi- level (pupil, school) regression model Different analysis, different results OutcomeBackgroundCoefficientSEp Post-test scoreConstant10.950.68760.000*** Intervention0.6720.57420.242NS Pre-test score0.77080.01670.000***

5 Different analysis, different results Using time as a level in a repeated measures multi-level (time, pupil, school) regression model 5 OutcomeBackgroundCoefficientSEp Total scoreConstant32.351.2060.000*** Time3.3870.35930.000*** Intervention-1.6821.6840.318NS Time*intervention1.2380.49160.012*

6 Interaction 6

7 Six sweeps example Mentoring scheme for struggling readers Pupil-level randomisation Questionnaire administered once at baseline and then every four months for the next two years 7

8 Six sweeps example Using time as a level in a repeated-measures multi-level (time, pupil, school) regression model 8 OutcomeBackgroundCoefficientSEp Aspirations for the futureConstant 22.650.1310.000*** Time -0.073580.026250.005** Intervention -0.27960.16940.099NS Time*intervention 0.081310.038010.032*

9 Reading Two-waves studies cannot describe individual trajectories of change and they confound true change with measurement error (Singer and Willett, 2002) ANCOVA is valid even with pre-test measurement error (Senn, 2004) Unconditional change models described in text books have three or more time-points The ANCOVA will almost always provide a more powerful test of the hypothesis of interest than will the repeated measures ANOVA approach (Dugard and Todman, 1995) 9

10 Change model assumption violation (2 sweeps) 10

11 Change model assumption OK (six sweeps) 11 Correlations r1r2r3r4r5r6 r1Pearson Correlation 1.000-0.002 0.023-0.002-0.003 Sig. (2-tailed) 0.0000.957 0.5490.9690.957 N 843680675674656347 r2Pearson Correlation -0.0021.000-0.0020.016 0.121 -0.039 Sig. (2-tailed) 0.9570.0000.9680.700 0.003 0.490 N 680 610606 591 315 r3Pearson Correlation -0.002 1.0000.047-0.002-0.003 Sig. (2-tailed) 0.9570.9680.0000.2370.9680.955 N 675610675622603316 r4Pearson Correlation 0.0230.0160.0471.000-0.0190.065 Sig. (2-tailed) 0.5490.7000.2370.0000.6360.237 N 674606622674612331 r5Pearson Correlation -0.002 0.121 -0.002-0.0191.000-0.003 Sig. (2-tailed) 0.969 0.003 0.9680.6360.0000.956 N 656 591 603612656321 r6Pearson Correlation -0.003-0.039-0.0030.065-0.0031.000 Sig. (2-tailed) 0.9570.4900.9550.2370.9560.000 N 347315316331321347

12 Measurement error problematic 12

13 Measurement error problematic 13

14 Measurement error not a problem 14

15 A better repeated measures model 15

16 A (slightly) better conditional model 16

17 Conclusion No consensus but it is probably safer to use a conditional model for a pre-test post-test design Designs with three or more sweeps will benefit from a repeated measures multi- level model Care with level 1 residual autocorrelation Try a few models and check assumptions Don’t get hung up on significance 17

18 Questions and advice 18

19 Acknowlegements Pearson Business in the Community and Queen’s University, Belfast Tom Benton Dougal Hutchison NFER 19


Download ppt "1 Repeated-measures data in educational research trials – how should it be analysed? Ben Styles Senior Statistician National Foundation for Educational."

Similar presentations


Ads by Google