How to analyze data What do we do with the collected data? By Yun Jin Rho

Contents 1. Overview of data analysis MasteringAstronomy 2. Case studies - Group comparison (MyITLab) - Causal relationship (MyMathTest) - Item level analysis (MasteringEngineering) 3. Closing comments How to analyze data l 09/24/11

Overview of data analysis 1 How to analyze data l 09/24/11

Raw data (MasteringAstronomy) ID ADT-Pre Final Grade ADT-Post 1 15 0.863 18 2 3 0.832 8 11 0.685 4 6 0.855 5 0.853 16 10 0.894 7 0.823 0.773 9 0.787 0.238 0.842 13 12 0.841 0.944 14 0.952 0.749 … How to analyze data l 09/24/11

Descriptive statistics N Mean SD Median Min Max Range SE Pre 314 7.812 3.245 7 1 18 17 0.183 Post 291 11.825 3.909 12 2 21 19 0.229 Final 378 0.796 0.145 0.831 0.967 0.007 Improvement from the Pre scores (M = 7.812) to the Post scores (M = 11.825) was observed.  Is this amount of improvement statistically significant? How to analyze data l 09/24/11

Distributions of test scores How to analyze data l 09/24/11

Hypothesis testing Let’s assume the distributions of test scores are all normal. How can we test if this difference of 4.263 is statistically significant?  Dependent (paired) t-test Hypotheses - H0: the improvement is not different from 0 (Pre = Post) - Ha: the improvement is significantly different from 0 (Pre ≠ Post) Results 95% Confidence Interval (C.I.) of the mean difference: [3.889, 4.637] t(250) = -22.44, p = 0.000 < 0.05 (significance level, ) Conclusion The improvement of 4.263 was statistically significant. How to analyze data l 09/24/11

Effect size Then, how big is this improvement or effect? Effect size : measure of distance between the two different distributions Cohen’s d = 1.36 Conclusion There was a large learning effect. Cohen’s d < 0.3: small effect Cohen’s d  0.5: medium effect Cohen’s d > 0.8: large effect How to analyze data l 09/24/11

Historical / Control without Mastering Experimental with Mastering What does this large learning effect mean? Was the students’ improvement in their test scores because of using Mastering? Is this improvement solely from using Mastering? Pretest Posttest Reading textbooks, Homework, TA’s help, Quality of teaching, Self-studying… Self-studying, Mastering… Historical / Control without Mastering Experimental with Mastering How to analyze data l 09/24/11

Case studies 2 How to analyze data l 09/24/11

Group comparison MyITLab ....... 2011 1st semester using the labs 2009 2nd semester without the labs UID Report 25 Word 10 Excel 15 Exam 50 Total 100 20111002 22.8 13.8 46.15384615 93 20111003 23 9.33 12.6 43.07692308 88 20111004 23.2 8.67 4.8 48.46153846 85 20111005 20.2 10.2 39.23076923 79 20111006 14 11.4 44.61538462 20111007 16.5 40.76923077 20111008 13.2 40 78 20111009 20111010 17.4 7.2 76 … UID Word  10 Report  30 Excel  15 Exam 45 Total 100 20092002 8 27 13.5 76 20092003 7 24 31.84615 73 20092004 9 12.75 25.61538 71 20092005 23 13.4 24.92308 70 20092006 17.5 13 30.46154 20092007 26 12 20092008 27.69231 68 20092009 21.5 11.75 26.30769 20092010 17 28.38462 67 … ....... How to analyze data l 09/24/11

Group comparison With the labs vs. Without the labs Mean S.D. S.E. Without Labs Word 123 75.58 16.89 1.52 Excel 122 70.23 17.88 1.62 Exam 124 55.77 10.27 0.92 Total 61.37 9.73 0.87 With Labs 228 79.88 20.13 1.33 231 77.82 21.3 1.4 233 64.86 12.41 0.81 69.39 12.09 0.79 How to analyze data l 09/24/11

Path analysis (causal relationship) MyMathTest IRB_ID PALG MALG PCALC ACTMATH 1109GRADE 1113GRADE 10766 32 A 10909 B 11291 97 16 F 10736 89 64 10747 28 21 W 10749 84 80 10944 13 20 10618 9 26 10902 34 24 10908 87 62 18 11254 54 27 D 11264 10582 93 92 91 29 10595 85 99 81 … .. How to analyze data l 09/24/11

Path analysis Causal model How to analyze data l 09/24/11

Item level analysis MasteringEngineering Pre-test Student 1 2 3 4 5 … 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 Post-test Student 1 2 3 4 5 … 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 How to analyze data l 09/24/11

Item level analysis: Pre-test How to analyze data l 09/24/11

Item level analysis: Post-test How to analyze data l 09/24/11

Item level analysis: Posttest - Pretest How to analyze data l 09/24/11

Summary 3 How to analyze data l 09/24/11

Closing comments To analyze the data for efficacy studies, the most important thing is the study design to collect data. Educational interpretation will be more important than the data analysis result. yunjin.rho@pearson.com How to analyze data l 09/24/11

Thank you How to analyze data l 09/24/11