2. How to analyze data What do we do with the collected data?
By Yun Jin Rho

3. Contents 1. Overview of data analysis MasteringAstronomy
2. Case studies
- Group comparison (MyITLab)
- Causal relationship (MyMathTest)
- Item level analysis (MasteringEngineering)
3. Closing comments
How to analyze data l 09/24/11

4. Overview of data analysis1
How to analyze data l 09/24/11

5. Raw data (MasteringAstronomy)ID
ADT-Pre
Final Grade
ADT-Post 1 15
0.863 18 2 3
0.832 8 11
0.685 4 6
0.855 5
0.853 16 10
0.894 7
0.823
0.773 9
0.787
0.238
0.842 13 12
0.841
0.944 14
0.952
0.749 …
How to analyze data l 09/24/11

6. Descriptive statisticsN
Mean SD
Median
Min
Max
Range SE
Pre
314
7.812
3.245 7 1 18 17
0.183
Post
291
11.825
3.909 12 2 21 19
0.229
Final
378
0.796
0.145
0.831
0.967
0.007
Improvement from the Pre scores (M = 7.812) to the Post scores (M = ) was observed.
 Is this amount of improvement statistically significant?
How to analyze data l 09/24/11

7. Distributions of test scores
How to analyze data l 09/24/11

8. Hypothesis testing
Let’s assume the distributions of test scores are all normal.
How can we test if this difference of is statistically significant?
 Dependent (paired) t-test
Hypotheses
- H0: the improvement is not different from 0 (Pre = Post)
- Ha: the improvement is significantly different from 0 (Pre ≠ Post)
Results
95% Confidence Interval (C.I.) of the mean difference: [3.889, 4.637] t(250) = , p = < 0.05 (significance level, )
Conclusion
The improvement of was statistically significant.
How to analyze data l 09/24/11

9. Effect size Then, how big is this improvement or effect? Effect size :
measure of distance between the two different distributions
Cohen’s d = 1.36
Conclusion
There was a large learning effect.
Cohen’s d < 0.3: small effect
Cohen’s d  0.5: medium effect
Cohen’s d > 0.8: large effect
How to analyze data l 09/24/11

10. Historical / Control without Mastering Experimental with Mastering
What does this large learning effect mean?
Was the students’ improvement in their test scores because of using Mastering? Is this improvement solely from using Mastering?
Pretest
Posttest
Reading textbooks, Homework,
TA’s help, Quality of teaching,
Self-studying…
Self-studying, Mastering…
Historical / Control without Mastering
Experimental with Mastering
How to analyze data l 09/24/11

11. Case studies 2
How to analyze data l 09/24/11

12. Group comparison MyITLab
2011 1st semester using the labs
2009 2nd semester without the labs
UID
Report 25
Word 10
Excel 15
Exam 50
Total
100
22.8
13.8 93 23
9.33
12.6 88
23.2
8.67
4.8 85
20.2
10.2 79 14
11.4
16.5
13.2 40 78
17.4
7.2 76 …
UID
Word  10
Report  30
Excel  15
Exam 45
Total
100 8 27
13.5 76 7 24 73 9
12.75 71 23
13.4 70
17.5 13 26 12 68
21.5
11.75 17 67 …
How to analyze data l 09/24/11

13. Group comparison With the labs vs. Without the labs
Mean
S.D.
S.E.
Without Labs
Word
123
75.58
16.89
1.52
Excel
122
70.23
17.88
1.62
Exam
124
55.77
10.27
0.92
Total
61.37
9.73
0.87
With Labs
228
79.88
20.13
1.33
231
77.82
21.3
1.4
233
64.86
12.41
0.81
69.39
12.09
0.79
How to analyze data l 09/24/11

14. Path analysis (causal relationship) MyMathTest
IRB_ID
PALG
MALG
PCALC
ACTMATH
1109GRADE
1113GRADE
10766 32 A
10909 B
11291 97 16 F
10736 89 64
10747 28 21 W
10749 84 80
10944 13 20
10618 9 26
10902 34 24
10908 87 62 18
11254 54 27 D
11264
10582 93 92 91 29
10595 85 99 81 … ..
How to analyze data l 09/24/11

15. Path analysis Causal model
How to analyze data l 09/24/11

16. Item level analysis MasteringEngineering
Pre-test
Student 1 2 3 4 5 … 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Post-test
Student 1 2 3 4 5 … 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
How to analyze data l 09/24/11

17. Item level analysis: Pre-test
How to analyze data l 09/24/11

18. Item level analysis: Post-test
How to analyze data l 09/24/11

19. Item level analysis: Posttest - Pretest
How to analyze data l 09/24/11

20. Summary 3
How to analyze data l 09/24/11

21. Closing comments
To analyze the data for efficacy studies, the most important thing is the study design to collect data.
Educational interpretation will be more important than the data analysis result.
How to analyze data l 09/24/11

22. Thank you
How to analyze data l 09/24/11