How to interact with the system? What is ITS? An intelligent tutoring system (ITS), broadly defined, is any computer system that provides direct customized instruction or feedback to students, i.e. without the intervention of human beings How to interact with the system? The interface module is the way the student interacts with the ITS, usually through a graphical user interface

The ASSISTment System A web-based assessment system, designed to collect formative assessment data on student math skills. Students are tutored on items that they get incorrect. Currently, thousands of students use the system. http://www.assistment.org

Levels of interaction Scaffolding + hints represents the most interactive experience because students must answer scaffolding questions, i.e. learning by doing. Hints on demand are less interactive because students do not have to respond to hints, but they can get the same information while solving problems as in the scaffolding questions by requesting hints. Delayed feedback is the least interactive condition because students must wait until the end of the assignment to get any feedback.

What is Scaffolding? Scaffolding essentially means doing some of the work for the student who isn't quite ready to accomplish a task independently - In the ASSISTment system, scaffolding consists of breaking down a problem into steps based on knowledge components.

1 3 4 2 1 original question with 3 scaffolding questions

Research: Scaling, load-balancing, parallelization

Predicting Student Answers in the ASSISTment System Zach Pardos Neil Heffernan Brigham Anderson Cristina Heffernan 1/15/2019

Goal “To investigate the utility of fine grain math skill models within the Assistment tutoring system.” Predict end of year Test Scores Predict answers on the ASSISTment System 1/15/2019

Skill Tagging Skill to skill mapping Skill tagging (Q-matrix) for the 1, 5 and 39 skill models was derived from a mapping of WPI-106 skills to WPI-39 skills to WPI-5 skills. Skill Model to Skill Model mapping also constructed by subject matter experts Neil and Cristina Heffernan. 1/15/2019

Student Knowledge Models Graphical Representation: Example of 1 skill model Example of 5 skill model 1/15/2019

Bayesian Networks Bayesian Belief Network created from Skill Model Q-Matrix/DAG Guess & Slip Parameters Defined “Ad Hock” Gates used to simplify network, avoids exponential numbers of CPTs having to be defined. Network assumes a 10% guess and 5% slip parameter for all questions and all models. 1/15/2019

Tutor (Online) Network Bayesian Prediction Model prediction evaluation process (1 user at a time) Step 1 Enter user’s responses to questions as evidence in the Online network ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ 1 0 1 0 1 1 0 0 0 1 0 1 1 1 0 0 1 0 Tutor (Online) Network 1/15/2019

Tutor (Online) Network Bayesian Prediction Model prediction evaluation process (1 user at a time) Step 2 Infer user’s skill knowledge probabilities given the evidence ? ? ? ? ? ↓ ↓ ↓ ↓ ↓ Tutor (Online) Network Inference Result: Algebra: 0.40, Geometry: 0.33 Data Analysis & probability: 0.73 Number Sense: 0.88 Measurement: 0.79 1/15/2019

Bayesian Prediction Model prediction evaluation process (1 user at a time) Step 3 Enter skill inference probabilities as soft evidence in Test network 0.40 0.33 0.73 0.88 0.79 ↓ ↓ ↓ ↓ ↓ MCAS (Test) Network 1/15/2019

Bayesian Prediction Model prediction evaluation process (1 user at a time) Step 4 Predict probability of user answering test questions correctly given soft, probabilistic skill evidence MCAS (Test) Network ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ 0.40.. 0.37.. 0.33.. 0.33 0.56.. 0.73.. 0.88.. 0.88.. 0.88.. 0.79 Summation of predicted question probabilities = Predicted score on the test Predicted score = 16. Actual score = 18. Absolute Difference between predicted and actual score = 2. Error = 2/29 (total possible points) = 6.9% 1/15/2019

Bayesian Results Model Test Prediction Performance/Error Results: MAD is Mean Average Difference. The test is out of 29 points so an Error of 13.7% is equivalent to over or under predicting by 4 points. Predicting Test Scores Predicting ASSISTment Answers 1/15/2019

References To read these and other papers on ASSISTment and this dataset, please visit www.Assistment.org and click on the [paper] button. [1] Razzaq, Feng, Heffernan, Koedinger, Nuzzo-Jones, Junker, Macasek, Rasmussen, Turner & Walonoski (2007). Blending Assessment and Instructional Assistance. In Nadia Nedjah, Luiza deMacedo Mourelle, Mario Neto Borges and Nival Nunesde Almeida (Eds). Intelligent Educational Machines within the Intelligent Systems Engineering Book Series . pp.23-49. (see http://www.isebis.eng.uerj.br/). Springer Berlin / Heidelberg. [2] Pardos, Z. A., Heffernan, N. T., Anderson, B., Heffernan, C. (2007) The Effect of Model Granularity on Student Performance Prediction Using Bayesian Networks. In Ardissono, Brna & Mitroivc (Eds) User Modeling 2007; 10th International Conference. Springer. [3] Pardos, Z., Feng, M. & Heffernan, N. T. & Heffernan-Lindquist, C. (Submitted) Analyzing fine-grained skill models using bayesian and mixed effect methods. In Luckin & Koedinger (Eds) Proceedings of the 13th Conference on Artificial Intelligence in Education. IOS Press. 1/15/2019

Do Research at WPI CS Dept! References Thanks! Do Research at WPI CS Dept! 1/15/2019

Outline Goal Models Dataset/ASSISTment Methodology Results Future work 1/15/2019

Skill Models Skill Models WPI-1 WPI-5 WPI-39 WPI-106 Each skill model covers all of 8th grade Mathematics using a different number of skills and skill generality. 1/15/2019

Skill Models Model Skills WPI-1: “8th Grade Math” WPI-5: Data Analysis, Geometry, Measurement, Number Sense Operations, Algebra/Pattern Relations WPI-39: a subset of the WPI-5 skills (Geometry.understanding-and-applying-pythagorean-theorem). WPI-106: Created by Neil and Cristina Heffernan at Worcester Polytechnic Institute Skill names for the WPI-5 and WPI-39 come from The Massachusetts Department of Education. 1/15/2019

Skill Tagging Skills from the WPI-106 were tagged to around 300 math questions (Q-matrix) for a web tutoring system called ASSISTment. The tagging of questions with skills allows for inference of skill knowledge given the observed student answers to questions in the tutor. The picture to the right is a sample question from the tutor system. It is tagged with the WPI-106 skill “Venn-Diagram”. 1/15/2019

Dataset “Online” Dataset (2004-2005) “Test” Dataset 600 students from 8 classes, 2 schools Each student answered at least 100 questions on the system (avg. 260). “Test” Dataset Massachusetts Standardized Test (MCAS) scores for each of the 600 students at the end of the 2004-2005 school year. Questions on the test were released by the state soon after the test was administered. These questions were then tagged with skills from our 106. 1/15/2019

Bayesian Prediction (v2) 1/15/2019

Bayesian Prediction (v2) 1/15/2019