Modeling Student Benefits from Illustrations and Graphs Michael Lipschultz Diane Litman Intelligent Tutoring Systems Conference (2014)

Motivation Best representation varies – Gender – Knowledge – Skills Identify situations when illustrations or graphs improved learning gains – Future: adapt to students/situations 2

Data Prior study – Problem-solving + post-problem discussion – Saw either illustrations only or graphs only – Pretest & Post-test – to measure learning gains – 29 subjects: 2,042 data points Features: – Student information ] – Student skill – Domain information – Contextual information 3

Modeling with Stepwise Regression 1. Stepwise Linear Regression Postscore = terms + prescore terms: representation*(tutoring context) representation*partition*rule – Illustration*(PreScore=High)*(ResponseTime=Fast) – For high pretesters, when ResponseTime=Fast, show illustrations – Binary features Keeps only predictive terms 4

Modeling with Stepwise Regression Algorithm 1.Stepwise Linear Regression 2.Identify Problematic Rules – (Potentially) Mutually Exclusive – Non-Adaptive 3.Handle Problematic Rules – Remove Lesser Rule in Pair 4.Relearn Model – Regular Regression 5

Modeling the Best Representation: Experiment Model Types – Baseline: just show one kind (illustration) – 1 Factor: 1 Tutoring Context factor in term – 2 Factors: Partition data along 1 variable High pretesters vs. Low pretesters – 3 Factors: Partition along 2 variables 6

Modeling the Best Representation: Results 7 2 Factors3 Factors (PreScore and …)

Modeling the Best Representation: PreScore Model 8 High Pretesters (n=11) 1.If many correct answers during session, show illustrations 2.If few correct answers in problem, show illustrations 3.If later in tutoring, show illustrations Low Pretesters (n=18) 1.If few correct answers during walk throughs or reflection dialogues, show graphs 2.If many correct answers during session, show illustrations 3.If later in tutoring, show illustrations 4.If few correct answers in problem, show graphs

Interpreting the Model PreScore*Gender 9 (n=8) 1.If few correct answers in walk throughs or reflections, show graphs 2.If many correct answers in session or problem, show illustrations 3.If early in problem or session, show graphs (n=9) 1.If many correct answers in session, show graphs 2.If early in session, show illustrations 3.If many correct answers in problem, show illustrations 4.If early in problem, show illustrations 5.If few correct answers in reflections, show illustrations (n=3) 1.If few correct in reflections, show illustrations 2.If many correct in session, show illustrations 3.If few correct in walk throughs, show illustrations (n=9) 1.If few correct answers in walk throughs or reflections, show illustrations 2.If many correct answers in session, show illustrations 3.If early in session or problem, show graphs 4.If many correct answers in problem, show illustrations Females Males High Pretesters Low Pretesters

Conclusion Developed modeling technique – Unknown gold standard – Handles “problematic” rules 5 models outperform baseline – Possible to model benefit Partitioning Useful: PreScore & Gender 10

Future Work Empirical Evaluation of Model – currently working on – Is adapting visual representation helpful? Develop method of selecting partition features – Partial correlation with postscore (covars=existing partitions)? Does modeling algorithm transfer to other tasks? Compare performance to Reinforcement Learning 11

Thank you 12