Researching for Better Online Instructional Methods using AB Testing --- Zhongzhou Chen, Christopher Chudzicki, Peter Bohacek, Neset Demirci and David Pritchard

Teaching is an old practice…

We enter the age of the Internet

Now we can teach like this

Can we find better ways to teach in an online environment?

In a normal MOOC Entire MOOC Population Course Material

A/B Testing Traditional Teaching MethodExperimental Teaching Method

A/B Testing Simple Design of an A/B Expriment TreatmentControl Common Assessment

Multiple A/B Experiments in a Course Unit 1 Experiment 1

Multiple A/B Experiments in a Course Unit 1Unit 2 Independent Random Splits Experiment 1Experiment 2

Multiple A/B Experiments in a Course Unit 1 Experiment 1 (part 1) Unit 2 Experiment 2 Unit 3 Experiment 1 (part 2)

Summary of A/B Experiment Features No limit on the number of groups, nor on the number of experiments in a course. Each group can receive a different number of materials. Group assignment is random. No student can "opt-out" of any of the experiments.

Examples of A/B Experiments in our MOOC

Our MOOC: 8.MReV Content: College introductory level mechanics (calculus based) Length: 14 Weeks (12 required, 2 optional) May 29 th Sep 14 th Initially Registered: Attempted Second Week’s Quiz: ~4100 (estimated)

Our MOOC: 8.MReV Total Number of AB experiments: 7

Experiment 1: Does length of video make a difference?

Does length of video make a difference? Office hour videos: ~10+ minutes Talk about a challenging homework problem

Does length of video make a difference? ~2 minute video segments Short Video

Does length of video make a difference? ~2 minute video segments Short Video

Does length of video make a difference? ~2 minute video segments Short Video ~10 minutes whole video Long Video

Does length of video make a difference? ~2 minute video segments Short Video ~10 minutes whole video Long Video

Experiment Procedure Challenging Homework Problem (Graded)

Experiment Procedure Long VideoShort VideoChallenging Homework Problem (Graded)

Experiment Procedure Long VideoShort VideoChallenging Homework Problem (Graded)

Does length of video make a difference? Research Questions: Difference in time spent watching video? Difference in number of questions answered? Difference in successfully solving the challenging homework problem?

Increasing Interactivity of Course Material

Interactivity of Course Material Traditional problem vs. Direct measurement video

Traditional Physics problem Provides all necessary information

Direct measurement video (by Peter Bohacek) Time measured by counting frames Player can play frame by frame Distance measured by on screen ruler.

Direct Measurement Video Common Homework Problem Traditional Homework Direct Measurement Video Survey question

Direct Measurement Video Research Questions: Does the extra interactivity result in more participation, or does the extra work lead to less participation? Can we identify any positive impact on students’ knowledge to justify the cost of creating these videos?

Interactivity of Course Material Multiple Choice vs. Drag and Drop Problem

Multiple Choice

Extraneous Cognitive Load Traditional Multiple Choice question

Drag and Drop Example

Drag and Drop Example 2

Drag and Drop and Deliberate Practice Deliberate Practice: (Erickson)

Drag and Drop and Deliberate Practice Deliberate Practice: Focus on practicing one skill at a time

Drag and Drop and Deliberate Practice Deliberate Practice: Focus on practicing one skill at a time Providing immediate specific feedback

Drag and Drop and Deliberate Practice Deliberate Practice: Focus on practicing one skill at a time Providing immediate specific feedback Repetitive and highly focused training

Drag and Drop and Deliberate Practice Deliberate Practice: Focus on practicing one skill at a time Providing immediate specific feedback Repetitive and highly focused training Domain Specific Problem Design Problem Design/Auto Grading/ Customized Feedback Drag and Drop format

Experimental Setup Entire MOOC population

Experimental Setup

Multiple Choice vs. Deliberate Practice Research Questions: Does increasing interactivity result in more engagement? (i.e. Are students more “deliberate” in their practice?) Compared to traditional problems, do Drag and Drop problems serve as better/worse assessments? Do “deliberate practice” result in any detectable learning gain?

Thank you Comments and Questions?

Discussion Questions Other innovative instructional methods worth putting online and testing via AB experiment?

Discussion Questions How should we design our experiments to suit the (unique) MOOC population? How can we transfer the experimental results obtained through a MOOC to a "flipped" classroom despite the difference in population?

Discussion Questions What criteria should we use to judge whether one pedagogy is better than another? Learning? Engagement? More discussion? Less attrition?

Discussion Questions How do we design effective assessments of learning when students have the freedom to access any external resources?