1. Researching for Better Online Instructional Methods using AB Testing --- Zhongzhou Chen, Christopher Chudzicki, Peter Bohacek, Neset Demirci and David Pritchard

2. Teaching is an old practice…

3. We enter the age of the Internet

4. Now we can teach like this

5. Can we find better ways to teach in an online environment?

6. In a normal MOOC Entire MOOC Population Course Material

7. A/B Testing Traditional Teaching MethodExperimental Teaching Method

8. A/B Testing Simple Design of an A/B Expriment TreatmentControl Common Assessment

9. Multiple A/B Experiments in a Course Unit 1 Experiment 1

10. Multiple A/B Experiments in a Course Unit 1Unit 2 Independent Random Splits Experiment 1Experiment 2

11. Multiple A/B Experiments in a Course Unit 1 Experiment 1 (part 1) Unit 2 Experiment 2 Unit 3 Experiment 1 (part 2)

12. Summary of A/B Experiment Features No limit on the number of groups, nor on the number of experiments in a course. Each group can receive a different number of materials. Group assignment is random. No student can "opt-out" of any of the experiments.

13. Examples of A/B Experiments in our MOOC

14. Our MOOC: 8.MReV Content: College introductory level mechanics (calculus based) Length: 14 Weeks (12 required, 2 optional) May 29 th ------- Sep 14 th Initially Registered: 10709 Attempted Second Week’s Quiz: ~4100 (estimated)

15. Our MOOC: 8.MReV Total Number of AB experiments: 7

16. Experiment 1: Does length of video make a difference?

17. Does length of video make a difference? Office hour videos: ~10+ minutes Talk about a challenging homework problem

18. Does length of video make a difference? ~2 minute video segments Short Video

19. Does length of video make a difference? ~2 minute video segments Short Video

20. Does length of video make a difference? ~2 minute video segments Short Video ~10 minutes whole video Long Video

21. Does length of video make a difference? ~2 minute video segments Short Video ~10 minutes whole video Long Video

22. Experiment Procedure Challenging Homework Problem (Graded)

23. Experiment Procedure Long VideoShort VideoChallenging Homework Problem (Graded)

24. Experiment Procedure Long VideoShort VideoChallenging Homework Problem (Graded)

25. Does length of video make a difference? Research Questions: Difference in time spent watching video? Difference in number of questions answered? Difference in successfully solving the challenging homework problem?

26. Increasing Interactivity of Course Material

27. Interactivity of Course Material Traditional problem vs. Direct measurement video

28. Traditional Physics problem Provides all necessary information

29. Direct measurement video (by Peter Bohacek) Time measured by counting frames Player can play frame by frame Distance measured by on screen ruler.

30. Direct Measurement Video Common Homework Problem Traditional Homework Direct Measurement Video Survey question

31. Direct Measurement Video Research Questions: Does the extra interactivity result in more participation, or does the extra work lead to less participation? Can we identify any positive impact on students’ knowledge to justify the cost of creating these videos?

32. Interactivity of Course Material Multiple Choice vs. Drag and Drop Problem

33. Multiple Choice

36. Extraneous Cognitive Load Traditional Multiple Choice question

37. Drag and Drop Example

38. Drag and Drop Example 2

39. Drag and Drop and Deliberate Practice Deliberate Practice: (Erickson)

40. Drag and Drop and Deliberate Practice Deliberate Practice: Focus on practicing one skill at a time

41. Drag and Drop and Deliberate Practice Deliberate Practice: Focus on practicing one skill at a time Providing immediate specific feedback

42. Drag and Drop and Deliberate Practice Deliberate Practice: Focus on practicing one skill at a time Providing immediate specific feedback Repetitive and highly focused training

43. Drag and Drop and Deliberate Practice Deliberate Practice: Focus on practicing one skill at a time Providing immediate specific feedback Repetitive and highly focused training Domain Specific Problem Design Problem Design/Auto Grading/ Customized Feedback Drag and Drop format

44. Experimental Setup Entire MOOC population

45. Experimental Setup

47. Multiple Choice vs. Deliberate Practice Research Questions: Does increasing interactivity result in more engagement? (i.e. Are students more “deliberate” in their practice?) Compared to traditional problems, do Drag and Drop problems serve as better/worse assessments? Do “deliberate practice” result in any detectable learning gain?

48. Thank you Comments and Questions?

49. Discussion Questions Other innovative instructional methods worth putting online and testing via AB experiment?

50. Discussion Questions How should we design our experiments to suit the (unique) MOOC population? How can we transfer the experimental results obtained through a MOOC to a "flipped" classroom despite the difference in population?

51. Discussion Questions What criteria should we use to judge whether one pedagogy is better than another? Learning? Engagement? More discussion? Less attrition?

52. Discussion Questions How do we design effective assessments of learning when students have the freedom to access any external resources?