1. Pilar Muñoz, José-Antonio González, Erik Cobo, Lluis Jover JORNADES D’INNOVACIÓ DOCENT A LA UPC: Presentació de resultats dels projectes MQD ICE 28/06/07 Avaluació experimental de la millora de l’aprenentatge en estadística

2. PRESENTACIÓ DE RESULTATS MQD 2 Outline 1. Aims and motivation 2. Model & subjects 3. Results 4. Conclusions and future work

3. PRESENTACIÓ DE RESULTATS MQD 3 Outline 1. Aims and Motivation 2. Model & subjects 3. Results 4. Conclusions and future work

4. PRESENTACIÓ DE RESULTATS MQD 4 To improve student learning To provide to students an automatic IT which generates and solves individual exercises To apply statistical theory to formally measure its effects To employ real examples in teaching Motivation

5. PRESENTACIÓ DE RESULTATS MQD 5 Learning Model

6. PRESENTACIÓ DE RESULTATS MQD 6 Factors to be efficient in the teaching process Training Practice of methods and techniques with realistic cases Instant feedback for students Immediate marking of work Evaluation of knowledge gained Evaluation of effort Feedback for teachers To monitor progress (globally or individually)

7. PRESENTACIÓ DE RESULTATS MQD 7 The tool: e-status (1) Learning by practicing: –The exercise can be repeated –Initial data are always different Immediate feedback providing: –Right answer –Error reason (if predicted) Students’ assessment: –Any criteria (best answer, average, …) Students’ follow up: –Both individual data and group summaries Broad range of problems: based on R Web-based tool

8. PRESENTACIÓ DE RESULTATS MQD 8 The tool: e-status (2) More information on e-status: Gonzalez & Muñoz (2006) (CAEE, 14(2): 151-159)

9. PRESENTACIÓ DE RESULTATS MQD 9 How e-status looks Wording Question Answer Grading

10. PRESENTACIÓ DE RESULTATS MQD 10 e-status previous experience Experience since 2003: –Four schools: –About 10 subjects –More than 2000 students –About 25000 executions High positive correlation between e-status use and exam performance… but …experience is not experiment Computer Science Maths and Stat Medicine Dentistry

11. PRESENTACIÓ DE RESULTATS MQD 11 Outline 1. Aims and Motivation 2. Model & subjects 3. Results 4. Conclusions and future work

12. PRESENTACIÓ DE RESULTATS MQD 12 Biostatistics course in the Dentistry School of the University of Barcelona Prior experience in 2004/2005 academic course Duration of the stat subject: 35 hours Teachers: –theoretical classes: 1 –lab groups: 3 Experiment: 2005/2006 year (fall) Outcome: written final examination Setting

13. PRESENTACIÓ DE RESULTATS MQD 13 Operation

14. PRESENTACIÓ DE RESULTATS MQD 14 Two “topics” A Descriptive statistics and graphical representation Agreement Inference about one proportion Comparison of two means Comparison of two proportions B Probabilities with Normal distribution Interval estimation of proportion and mean Assessment of sample size Inference about one mean Goodness of fit chi-square Every student had access to e-status Students were randomly allocated to group 1 or 2 Each group only had access to e-status exercises on only one topic Final exam contained questions about both topics A and B Stat course content was divided in two ‘balanced’ topics

15. PRESENTACIÓ DE RESULTATS MQD 15 Outcome Exam Topic Problems in the practical exam e11e12e21e22e31e32 A B Score Y t A i Score Y t B i

16. PRESENTACIÓ DE RESULTATS MQD 16 Participants & allocation All students (N =121) enrolled in the course Random assignment to 1 or 2 balanced with respect to Lab group and new/old profile Teacher was not involved in randomization neither data analysis Final exam evaluator was masked to allocation group

17. PRESENTACIÓ DE RESULTATS MQD 17 Hypothesis Y t j i stands for the exam performance: by student i assigned to intervention, t=1, 2 in topic j=A, B Hypothesis: If e-status is effective, students in group 1 (2) trained with e-status exercises in topic A (B) should get better exam results in topic A (B)

18. PRESENTACIÓ DE RESULTATS MQD 18 Y tji = µ +  t + π j + Φ i +  ij  t : fixed effect of intervention t=1, 2 π j : fixed effect of exam topic j=A, B Φ i : random effect of student i  ij : measure error assessing performance in student i for question j Assumptions: Access to e-status topic A(B) has no effect on exam topic B(A). Error independence between students Statistical model

19. PRESENTACIÓ DE RESULTATS MQD 19 Statistical analysis Let D 1 i (D 2 i ) be the difference of scores in part A and B for the student i receiving intervention 1(2): D 1i = Y 1Ai – Y 1Bi = (µ + τ 1 + π A + Φ i +  iA )-(µ + π B + Φ i +  iB ) = τ 1 + π A - π B +  iA -  iB D 2i = Y 2Ai – Y 2Bi = (µ + π A + Φ i +  iA )-(µ + τ 2 + π B + Φ i +  iB ) = - τ 2 +π A -π B +  iA -  iB E(D 1i )= τ 1 + π A - π B As E(D 2i )= -τ 2 + π A - π B V(D ji )= V(  iA -  iB ) = 2σ 2 ε Then 50 students per group 80% power to highlight an effect equal to 0.7x σ  (α=0.05) 50 students per group provide 80% power to highlight an effect equal to 0.7x σ  (α=0.05)

20. PRESENTACIÓ DE RESULTATS MQD 20 Outline 1. Aims and Motivation 2. Model & subjects 3. Results 4. Conclusions and future work

21. PRESENTACIÓ DE RESULTATS MQD 21 Main results

22. PRESENTACIÓ DE RESULTATS MQD 22 The random model, fitted with R, replicated the results S Φ = 2.81 (CI 95% : 2.43 to 3.24) S ε = 1.48 (CI 95% : 1.31 to 1.68) Linear Mixed-Effects Model

23. PRESENTACIÓ DE RESULTATS MQD 23 Outline 1. Aims and Motivation 2. Model & subjects 3. Results 4. Conclusions and future work

24. PRESENTACIÓ DE RESULTATS MQD 24 +1. It is feasible to evaluate interventions in teaching with formal experiments Random allocation Masked evaluation Without interfering in course development +2. e-status improves student exam performance In an exam over 10 points, e-status improves performance by 0.96 points (CI 95% : 0.20, 1.72) Conclusions

25. PRESENTACIÓ DE RESULTATS MQD 25 What did the students do when they had no access to e-status? That is, what is the reference for the intervention? a) Did they not study statistics at all? b) Did they spend their time on another kinds of exercises? If a, the estimated e-status effect is mediated by an increase in the time spent by the student to study. If b, e-status increases learning efficiency, since it improves the amount of learning with respect to the alternative method. Interpretation (1)

26. PRESENTACIÓ DE RESULTATS MQD 26 If e-status influences advanced capabilities, such as motivation or statistical reasoning, there may be some contamination between interventions A and B: that is, intervention 1 (2) employing e ‑ status on set A (B) would have some learning effect τ‘ 1 (τ‘ 2 ) on set B (A), and then E( ) = (τ 1 + τ 2 ) – (τ‘ 1 + τ‘ 2 ) = (τ 1 - τ‘ 1 ) + (τ 2 - τ‘ 2 ) If so, this design estimates e-status direct effect minus delayed, cross-over, effects If τ‘ positive, this design underestimates overall effects on learning Interpretation (3)

27. PRESENTACIÓ DE RESULTATS MQD 27 1. Repeat the randomized experiment many times in many courses You are invited to do it! 2. Also you are invited: - to use it in your teaching work http://key.upc.es/estatus 3. To share your ideas with us pilar.munyoz@upc.edu, jose.a.gonzalez@upc.edu, erik.cobo@upc.edu Future work

28. PRESENTACIÓ DE RESULTATS MQD 28 Thanks for your attention Q & A