1. Mingyu Feng Neil Heffernan Joseph Beck
Using learning decomposition to analyze instructional effectiveness in the ASSISTment system
Mingyu Feng
Neil Heffernan
Joseph Beck

2. Introduction Many intelligent tutoring systems have similar items
called Group of Learning Opportunities (Feng et al., 2008) when presented randomly
share the same prerequisite knowledge
target the same set of skills.
Include questions and tutoring instructions
A basic question of instruction is how effective each item is in promoting student learning.

3. Introduction
One popular method for answering this questions is to run a randomized controlled study. However, such a study can be expensive
# of students required
Considerable duration of the experiments
Administration of pretest and posttest
Introducing a new method using learning decomposition (Beck, 2006)

4. Goal The goals of this work are
Estimating and comparing the relative impact of various tutoring components in the ASSISTment system.
Presenting a case study of applying the learning decomposition technique to a domain, mathematics. The method has been applied in the subject area of reading.

5. Methodology – Learning Decomposition
An easy recipe to tell which type of practice is more effective in helping students learning
An extension of classical exponential learning curve by taking into account the heterogeneity of different learning opportunities for a single skill
# of prior trials
of practice
type 1
# of prior trials
of practice
type 2
t: total # of prior trials, t = t1+t2

6. Approach – Basic facts of data
181 Assistments items organized in 39 GLOPs, with 2 to 11 items in each GLOP
Data collected from Oct. 31st, 2006 to Oct. 11th, 2007
2502 8th grade students involved, each on average worked on 22 items
54600 data points

7. Approach - Outcome measures
Continuous metrics, e.g. reading time
Binary correctness data (0/1)
logistic model
Performance reflected by odds ratio of success to failure
Indicating how effective type 1 practice is comparing to type 2

8. Approach – data
Each item in a GLOP is considered as a different type of practice; then students’ practice opportunities are factored into practice at each individual item.
Table 1. Raw response data of student A
Student ID
Item ID
Timestamp
Previous trials (t)
Correct? A
1045
12:30:31AM 1
1649
12:31:15 AM
1263
12:43:40 AM 2
1022
12:46:09 AM 3
1660
12:48:20 AM 4
Table 2. Decomposed response data of student A
Student ID
Item ID
Trials
(t)
Correct?
Prior encounters
Item 1022
Item 1045
Item 1263
Item 1649
Item 1660 A
1045 1
1649
1263 2
1022 3
1660 4

9. Approach Fitting a logistic regression model
Estimates of the coefficient Bi represents the amount of learning caused by item I

10. Result
Table 3. Coefficients of logistic regression model for items in GLOP 1 and GLOP 4
GLOP ID
Item ID
B (Coefficient)
(higher is better)
std. err 1
1022
0.464
0.257
1660
0.414
0.247
1045
0.127
0.254
1263
0.011
0.241
1649
-0.176
0.261 4
2264
0.707
0.225
2236
0.079
0.236
9086
-0.014
0.232
2239
-0.236
0.237
2274
-0.274
0.240

11. Result
Effectiveness

12. Conclusion
We explored the research question of measuring the instructional effectiveness of different problems, and associated tutoring
Provides evidence of the generalarity of learning decomposition
Demonstrate a low cost approach for evaluating instructional contents
Some contents in ASSISTments have more impact on student learning while some are not so effective

13. Future work Validating the estimates
Human raters
Synthetic data of a computer simulation study
Build and analyze items with same main questions but different tutoring strategies