Mingyu Feng Neil Heffernan Joseph Beck Using learning decomposition to analyze instructional effectiveness in the ASSISTment system Mingyu Feng Neil Heffernan Joseph Beck

Introduction Many intelligent tutoring systems have similar items called Group of Learning Opportunities (Feng et al., 2008) when presented randomly share the same prerequisite knowledge target the same set of skills. Include questions and tutoring instructions A basic question of instruction is how effective each item is in promoting student learning.

Introduction One popular method for answering this questions is to run a randomized controlled study. However, such a study can be expensive # of students required Considerable duration of the experiments Administration of pretest and posttest Introducing a new method using learning decomposition (Beck, 2006)

Goal The goals of this work are Estimating and comparing the relative impact of various tutoring components in the ASSISTment system. Presenting a case study of applying the learning decomposition technique to a domain, mathematics. The method has been applied in the subject area of reading.

Methodology – Learning Decomposition An easy recipe to tell which type of practice is more effective in helping students learning An extension of classical exponential learning curve by taking into account the heterogeneity of different learning opportunities for a single skill # of prior trials of practice type 1 # of prior trials of practice type 2 t: total # of prior trials, t = t1+t2

Approach – Basic facts of data 181 Assistments items organized in 39 GLOPs, with 2 to 11 items in each GLOP Data collected from Oct. 31st, 2006 to Oct. 11th, 2007 2502 8th grade students involved, each on average worked on 22 items 54600 data points

Approach - Outcome measures Continuous metrics, e.g. reading time Binary correctness data (0/1) logistic model Performance reflected by odds ratio of success to failure Indicating how effective type 1 practice is comparing to type 2

Approach – data Each item in a GLOP is considered as a different type of practice; then students’ practice opportunities are factored into practice at each individual item. Table 1. Raw response data of student A Student ID Item ID Timestamp Previous trials (t) Correct? A 1045 12:30:31AM 1 1649 12:31:15 AM 1263 12:43:40 AM 2 1022 12:46:09 AM 3 1660 12:48:20 AM 4 Table 2. Decomposed response data of student A Student ID Item ID Trials (t) Correct? Prior encounters Item 1022 Item 1045 Item 1263 Item 1649 Item 1660 A 1045 1 1649 1263 2 1022 3 1660 4

Approach Fitting a logistic regression model Estimates of the coefficient Bi represents the amount of learning caused by item I

Result Table 3. Coefficients of logistic regression model for items in GLOP 1 and GLOP 4 GLOP ID Item ID B (Coefficient) (higher is better) std. err 1 1022 0.464 0.257 1660 0.414 0.247 1045 0.127 0.254 1263 0.011 0.241 1649 -0.176 0.261 4 2264 0.707 0.225 2236 0.079 0.236 9086 -0.014 0.232 2239 -0.236 0.237 2274 -0.274 0.240

Result Effectiveness

Conclusion We explored the research question of measuring the instructional effectiveness of different problems, and associated tutoring Provides evidence of the generalarity of learning decomposition Demonstrate a low cost approach for evaluating instructional contents Some contents in ASSISTments have more impact on student learning while some are not so effective

Future work Validating the estimates Human raters Synthetic data of a computer simulation study Build and analyze items with same main questions but different tutoring strategies