1. Intelligent Tutoring Systems

2. What is an ITS? Proposal:
Start with a bunch of exercises for a topic, where each exercise is given a difficulty rating and subtopic.
Start students with a series of exercises at the easiest rating (perhaps with training materials) for the first topic.
When the student has “proved proficiency” at the lowest rating, start giving them the next difficulty (perhaps with more training materials).
When they complete the various levels for the current subtopic, move to the next one.
Is this an ITS?

3. A Definition:
To be classified as “intelligent”, they must present “human-like” tutoring capabilities, that means to be able to adjust the content and delivery of the lesson to the students’ characteristics and needs by analysing and/or anticipating theirs responses and behaviours.
-Roger Nkambou

4. Q: Does a (human) tutor make much difference?

5. Effects of (Human) Tutoring
Two-sigma “problem”:
Bloom observed that students who receive good 1-on-1 tutoring perform two standard deviations better than students who receive traditional classroom instruction.
The ideal ITS would perform as well as a good human tutor, providing equivalent 1-on-1 instruction.
Today’s online courses are at the performance level of traditional classroom instruction.

6. Key problem “I am stuck, what do I do now?”
Online courses do no better than traditional lecture to solve this problem.
Can ITS do better?
Even if an ITS only handled the easy situations, that would still leave more time for a human instructor to handle the more difficult cases.
Work ongoing with ITS since the 1970s

7. Early Successes 1980’s: CMU LISP Tutor
students performed 43% better on final than traditional instruction
completed complex programming task 30% faster
Early 1990’s: Sherlock (jet aircraft troubleshooting procedures)
Students using ITS did significantly better than control group.
After 20 hours of instruction, performed at the level of technicians with 4 years of experience.
Many successful military training systems
Successful math instruction: Cognitive Tutors

8. Traditional Architecture

9. Primary Components
Expert Model (Domain Model): Subject matter knowledge
Student Model: Represents what the student does and does not know
Instructor Model (Tutoring Model): ITS strategy for how to teach
User Interface Model

10. Rule-based Approach
Contains domain expert’s subject matter knowledge and problem- solving strategies
Problem-solving strategy can be captured by rules
ITS can compare learner’s process to the expert model’s actions.
Ideally, can generate problems on-the-fly from the rules.
Expensive to build

11. Scenario-based Approach
A given training scenario incorporates the necessary knowledge.
A template task incorporates the necessary steps to solve the problem
Limits the necessary knowledge to the one problem type
A tutorial is then a series of such problem types, each with its knowledge and rules

12. Student Model Evaluates student performance
The key problem is deducing what misconception led to a particular wrong answer, and then addressing that misconception.
Example: Mistakes in addition
Fail to carry
Always carry
Mistakes in single-digit addition

13. Example: Tactical Action Officer

14. Instructor Model Encodes instructional process
Integrates information from the Student and Expert Models to make decisions
Attempts to decide the “next step”
Supply instruction (new content or hints)
Supply problems (of right difficulty)

15. ITS Systems
While the picture describe of the gross architecture is common, every ITS system is different in the details.
Typically an ITS works for only some tutoring strategies.
Each typically focuses on one model (expert, student, or instructor) over the others.

16. Rules
One approach to the instructor model is as a large body of production rules.
The Student Model stores values for probability that the student knows each rule.
When the student applies a rule to the current problem state, the probability value for that rule goes up.

17. Instructor Model: Guidance
A major issue is how much guidance to give the student.
Too much leads to hints-driven behavior, shallow learning
Too little leads to student not knowing what to do, making no progress.
Direct Instruction vs. Discovery Learning
Discovery learning is in favor
“Minimally guided discovery” or “pure discovery” is out of favor

18. Cognitive Tutor
Example of an experiment (Brunstein, Betts, and Anderson 2009). Also illustrates a cognitive math tutor.

19. Experiment

20. Results
Conditions: (a) Written instruction, (b) direct demonstration, (c) both, (d) neither (discovery)
For the first two problems in a set, discovery was the slowest
But for the remaining problems, discovery and direct demonstration were significantly faster than the others (total time)
This was true ONLY when there was significant practice during each section.
When the amount of practice was cut, the discovery treatment participants effectively could not continue past a certain point.
This represents a typical example of the type of design issues that ITS developers face.
In this case, I question if they got the right interpretation. This could be true that in isolation, discovery and lots of practice does best. But over the entire series, direct demonstration might do the best.

21. Development Process for ITS
Needs Assessment
Figure out the specific content pieces, their difficulty, etc.
Cognitive Task Analysis
Build computational model of the required problem solving knowledge
Lots of domain expert interviews for how to solve problems
Initial Tutor Implementation
Set up guidance system to support student working problems.
Evaluation

22. Are ITS used?
Recent evaluations of ITS have found some that have effect size comparable to one-on-one human tutors.
Algebra Cognitive Tutor
Such cognitive math tutors used in thousands of schools each year, with hundreds of thousands of students
If they have been studied so long, and work so well, why do they not make a bigger impact?
Does Khan Academy use ITS? Code Academy, etc?

23. Cost Issues
Development times are typically hours for 1 hour of instruction.
Not considered commercially viable at this ratio
This does not actually make sense to me. That cost does not seem at all excessive.

24. Pedagogical Limitations
Might not develop “deep” learning in students
Being able to request hints on demand might harm learning
Some students use a “hints driven” approach to get the exercises done
ITS does not ask questions of students to get them to explain their problem solving approach
The result is that they don’t learn the language, and so cannot communicate about the content.

25. Current research
Better dialog systems (Socratic Method, Discovery Driven)
Better detection of student emotional affect
Observation: The best learning comes with some --- but not too much --- tension or frustration.