1. How to interact with the system?
What is ITS?
An intelligent tutoring system (ITS), broadly defined, is any computer system that provides direct customized instruction or feedback to students, i.e. without the intervention of human beings
How to interact with the system?
The interface module is the way the student interacts with the ITS, usually through a graphical user interface

2. The ASSISTment System
A web-based assessment system, designed to collect formative assessment data on student math skills.
Students are tutored on items that they get incorrect.
Currently, thousands of students use the system.

3. Levels of interaction
Scaffolding + hints represents the most interactive experience because students must answer scaffolding questions, i.e. learning by doing.
Hints on demand are less interactive because students do not have to respond to hints, but they can get the same information while solving problems as in the scaffolding questions by requesting hints.
Delayed feedback is the least interactive condition because students must wait until the end of the assignment to get any feedback.

4. What is Scaffolding?
Scaffolding essentially means doing some of the work for the student who isn't quite ready to accomplish a task independently
- In the ASSISTment system, scaffolding consists of breaking down a problem into steps based on knowledge components.

5. 1 3 4 2
1 original question with 3 scaffolding questions

6. Research: Scaling, load-balancing, parallelization

7. Predicting Student Answers in the ASSISTment System
Zach Pardos
Neil Heffernan
Brigham Anderson
Cristina Heffernan
1/15/2019

8. Goal
“To investigate the utility of fine grain math skill models within the Assistment tutoring system.”
Predict end of year Test Scores
Predict answers on the ASSISTment System
1/15/2019

9. Skill Tagging
Skill to skill mapping
Skill tagging (Q-matrix) for the 1, 5 and 39 skill models was derived from a mapping of WPI-106 skills to WPI-39 skills to WPI-5 skills. Skill Model to Skill Model mapping also constructed by subject matter experts Neil and Cristina Heffernan.
1/15/2019

10. Student Knowledge Models
Graphical Representation:
Example of 1 skill model
Example of 5 skill model
1/15/2019

11. Bayesian Networks
Bayesian Belief Network created from Skill Model Q-Matrix/DAG
Guess & Slip Parameters Defined “Ad Hock”
Gates used to simplify network, avoids exponential numbers of CPTs having to be defined.
Network assumes a 10% guess and 5% slip parameter for all questions and all models.
1/15/2019

12. Tutor (Online) Network
Bayesian Prediction
Model prediction evaluation process (1 user at a time)
Step 1
Enter user’s responses to questions as evidence in the Online network
↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑
Tutor (Online) Network
1/15/2019

13. Tutor (Online) Network
Bayesian Prediction
Model prediction evaluation process (1 user at a time)
Step 2
Infer user’s skill knowledge probabilities given the evidence
? ? ? ? ?
↓ ↓ ↓ ↓ ↓
Tutor (Online) Network
Inference Result: Algebra: 0.40, Geometry: 0.33
Data Analysis & probability: 0.73
Number Sense: 0.88 Measurement: 0.79
1/15/2019

14. Bayesian Prediction
Model prediction evaluation process (1 user at a time)
Step 3
Enter skill inference probabilities as soft evidence in Test network
↓ ↓ ↓ ↓ ↓
MCAS (Test) Network
1/15/2019

15. Bayesian Prediction
Model prediction evaluation process (1 user at a time)
Step 4
Predict probability of user answering test questions correctly given soft, probabilistic skill evidence
MCAS (Test) Network
↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑
Summation of predicted question probabilities = Predicted score on the test
Predicted score = 16. Actual score = 18. Absolute Difference between predicted and actual score = 2. Error = 2/29 (total possible points) = 6.9%
1/15/2019

16. Bayesian Results Model Test Prediction Performance/Error Results:
MAD is Mean Average Difference. The test is out of 29 points so an Error of 13.7% is equivalent to over or under predicting by 4 points.
Predicting Test Scores
Predicting ASSISTment Answers
1/15/2019

17. References
To read these and other papers on ASSISTment and this dataset, please
visit and click on the [paper] button.
[1] Razzaq, Feng, Heffernan, Koedinger, Nuzzo-Jones, Junker, Macasek, Rasmussen, Turner & Walonoski (2007). Blending Assessment and Instructional Assistance. In Nadia Nedjah, Luiza deMacedo Mourelle, Mario Neto Borges and Nival Nunesde Almeida (Eds). Intelligent Educational Machines within the Intelligent Systems Engineering Book Series . pp (see Springer Berlin / Heidelberg.
[2] Pardos, Z. A., Heffernan, N. T., Anderson, B., Heffernan, C. (2007) The Effect of Model Granularity on Student Performance Prediction Using Bayesian Networks. In Ardissono, Brna & Mitroivc (Eds) User Modeling 2007; 10th International Conference. Springer.
[3] Pardos, Z., Feng, M. & Heffernan, N. T. & Heffernan-Lindquist, C. (Submitted) Analyzing fine-grained skill models using bayesian and mixed effect methods. In Luckin & Koedinger (Eds) Proceedings of the 13th Conference on Artificial Intelligence in Education. IOS Press.
1/15/2019

18. Do Research at WPI CS Dept!
References
Thanks!
Do Research at WPI CS Dept!
1/15/2019

19. Outline Goal Models Dataset/ASSISTment Methodology Results Future work
1/15/2019

20. Skill Models Skill Models
WPI-1
WPI-5
WPI-39
WPI-106
Each skill model covers all of 8th grade Mathematics using a different number of skills and skill generality.
1/15/2019

21. Skill Models Model Skills
WPI-1: “8th Grade Math”
WPI-5: Data Analysis, Geometry, Measurement, Number Sense Operations, Algebra/Pattern Relations
WPI-39: a subset of the WPI-5 skills (Geometry.understanding-and-applying-pythagorean-theorem).
WPI-106: Created by Neil and Cristina Heffernan at Worcester Polytechnic Institute
Skill names for the WPI-5 and WPI-39 come from The Massachusetts Department of Education.
1/15/2019

22. Skill Tagging
Skills from the WPI-106 were tagged to around 300 math questions (Q-matrix) for a web tutoring system called ASSISTment.
The tagging of questions with skills allows for inference of skill knowledge given the observed student answers to questions in the tutor.
The picture to the right is a sample question from the tutor system. It is tagged with the WPI-106 skill “Venn-Diagram”.
1/15/2019

23. Dataset “Online” Dataset (2004-2005) “Test” Dataset
600 students from 8 classes, 2 schools
Each student answered at least 100 questions on the system (avg. 260).
“Test” Dataset
Massachusetts Standardized Test (MCAS) scores for each of the 600 students at the end of the school year.
Questions on the test were released by the state soon after the test was administered. These questions were then tagged with skills from our 106.
1/15/2019

24. Bayesian Prediction (v2)
1/15/2019

25. Bayesian Prediction (v2)
1/15/2019