SimStudent: A Computational Model of Learning as a Research Toolbox for the Sciences of Learning Noboru Matsuda Human-Computer Interaction Institute Carnegie Mellon University

Research Questions Building a cognitive model is hard. Can machine-learning techniques help non-expert authors build a Cognitive Tutor? Would like to simulate students’ learning. Can machine-learning techniques help us build a computational model of learning with a cognitive fidelity? I heard that students learn by teaching others. Can we use the computational model of learning to study the theory of learning by teaching?

Solution: SimStudent Machine learning agent Fundamental technology Learns procedural skills, by Observing model solutions & solving problems Fundamental technology Programming by Demonstration Inductive Logic Programming Knowledge representation Production rules (Jess) Lau & Weld (1998). Blessing (1997).

SimStudent Projects Intelligent Authoring Building a Cognitive Tutor as a CTAT Plug-in Student Modeling and Simulation Controlled educational studies Error formation study Prerequisite conceptual knowledge study Teachable Peer Learner Learning by teaching

Authoring a Cognitive Tutor Example-Tracing Tutor Little programming A cognitive model specific to a particular problem Limited generalization by editing a behavior graph Model-Tracing Tutor Powerful student model Cognitive task analysis is hard Writing production rules is even more challenging Performing a task is relatively easy…

Next Generation Authoring Build a tutor GUI Teaching a solution SimSt. learning Production Rules Rule simplify-LHS: IF is-equation( Eq ), is-lhs( Eq, Lhs ), polynomial( Lhs ), all-var-terms( Lhs ) Then simplify( Lhs, S-lhs ), enter( S-lhs ) Rule simplify-LHS: IF is-equation( Eq ), is-lhs( Eq, Lhs ), polynomial( Lhs ), all-var-terms( Lhs ) Then simplify( Lhs, S-lhs ), enter( S-lhs ) Rule simplify-LHS: IF is-equation( Eq ), is-lhs( Eq, Lhs ), polynomial( Lhs ), all-var-terms( Lhs ) Then simplify( Lhs, S-lhs ), enter( S-lhs )

Demo Authoring by Tutoring

Example: Learning to subtract a constant term Learning to subtract a constant number First example Subtract the difference between 4 and 3… subtract 1 Subtract the coefficient of X… I see 3x, 1, x, and 4 in the equation. I wonder where the ‘1’ came from… Subtract the last term on the left-hand side… PSLC Summer School 2012 :: SimStudent :: Noboru Matsuda (CMU)

Example: Learning to subtract a constant term Learning to subtract a constant number First example Subtract the difference between 4 and 3… subtract 1 Subtract the coefficient of X… I see 3x, 1, x, and 4 in the equation. I wonder where the ‘1’ came from… Subtract the last term on the left-hand side… PSLC Summer School 2012 :: SimStudent :: Noboru Matsuda (CMU)

PSLC Summer School 2012 :: SimStudent :: Noboru Matsuda (CMU) Prior Knowledge Feature predicates 18 predicates isFractionTerm(X), isConstant(X), isPolynomial(X),… Operators 42 operators add(X,Y), coefficient(X), getFrstNumber(X), … PSLC Summer School 2012 :: SimStudent :: Noboru Matsuda (CMU)

Example: Stoichiometry Tutor

SimStudent Projects Intelligent Authoring Building a Cognitive Tutor as a CTAT Plug-in Student Modeling and Simulation Controlled educational studies Error formation study Prerequisite conceptual knowledge study Teachable Peer Learner Learning by teaching

Model of Incorrect Learning Identify errors students commonly make Weaken SimStudent’s background knowledge Let SimStudent make an induction error

Weak Prior Knowledge Hypothesis Multiple ways to make sense of examples Get a coefficient and divide Get a denominator and multiply “multiply by x” 3x=5 “divide by 3” 4/x=5 “divide by 4” Get a number and divide

Results: Learning Rate Step Score Estimate of correct (probability of each step right) Precision -> step score # training problems Steps Score = 0 (if there is no rule applicable) # correct rule applications / # all rule applications

Student Model A set of knowledge components (KCs) (Li et al. EDM2011) A Machine Learning Approach for Automatic Student Model Discovery A set of knowledge components (KCs) Encoded in intelligent tutors to model how students solve problems E.g. How to proceed given problems of the form Nv=N One of the key factors that affects automated tutoring systems in making instructional decisions Previous Approach: Require expert input Highly subjective Proposed Approach: Use a machine-learning agent, SimStudent, to acquire knowledge 1 Skill  1 KC Skill application for that step  KC for each step

Human-generated vs SimStudent KCs Human-generated Model SimStudent Comment Total # of KCs 12 21 # of Basic Arithmetic Operation KCs 4 13 Split into finer grain sizes based on different problem forms # of Typein KCs Approximately the same # of Other Transformation Operation KCs (e.g. combine like terms) Arithmetic op are like “add to both sides”, subtract from both sides, etc. 4x = 20 vs. –x = 5

Results Significance Test Human-generated Model SimStudent AIC 6529 6448 3-Fold Cross Validation RMSE 0.4034 0.3997 Significance Test SimStudent outperforms the human- generated model in 4260 out of 6494 steps p < 0.001 SimStudent outperforms the human-generated model across 20 runs of cross validation AFM Performance proportion = 0.655990145 and odds 1.906893465 Can you compute separately this ratio for the steps where the Q matrix is different from

SimStudent Projects Intelligent Authoring Building a Cognitive Tutor as a CTAT Plug-in Student Modeling and Simulation Controlled educational studies Error formation study Prerequisite conceptual knowledge study Teachable Peer Learner Learning by teaching

Learning by Teaching SimStudent

Learn more about SimStudents Project Web www.SimStudent.org Contact us Noboru Matsuda mazda@cs.cmu.edu