1. Deriving Acquisition Principles from Tutoring Principles
Jihie Kim
Yolanda Gil
Information Sciences Institute University of Southern California

2. Our Previous work in KA
Interdependency based KA interface [Kim & Gil AAAI-99]
KA evaluation with various end users [Kim & Gil AAAI-00; Kim & Gil IUI-00]
KA evaluation methodology [Tallis, Kim & Gil JETAI-02]
KA interface to build process models [Kim & Gil IJCAI-01]
Analyzing KA tools in tutoring perspective [Kim & Gil CogSci-02]
Script-based knowledge acquisition [Tallis and Gil AAAI-99]
English-based editors [Blythe and Ramachandran KAW-99]
Capturing general principles [Blythe IJCAI-01]

3. Feedback from End Users
User’s comments :
“The system had to be taken by the hand”
"I do not really know whether there is a possibility of standardizing the entire [KA] process. But it would be better to document some of the [KA] processes which you think are standardized".
 Users Need More Proactive Guidance!

4. was/wasn’t understood
Proactive Learning
(Tele-operated
Robot)
passive
no feedback
WYGIWYI
~60’s …
(Autonomous
Robot)
Proactive
Plan and suggest
Ask for help
Highlight what
was/wasn’t understood …
now …

5. General Research Issues
How to turn a KA tool into a good student, how to help a user be a good teacher
Tutoring & educational literature
Assess competence and confidence in the new body of knowledge
Dialogue planning
Meta-level knowledge about KA tasks
Collaborative dialogue
User modeling
Utility of system’s interventions

6. Deriving Acquisition Principles from Tutoring Principles
SOFTWARE
USER ?
Instructional
System
Good
Tutoring
Principles
teaches
Acquisition
Tool
Good
Learning
Principles
teaches ?

7. 15 Tutoring and Learning Principles
Teaching/Learning principle
Tutoring literature
Start by introducing lesson topics and goals
Atlas-Andes, Meno-Tutor, Human tutorial dialog
Use topics of the lesson as a guide
BE&E, UMFE
Subsumption to existing cognitive structure
Human learning, WHY, Atlas-Andes
Immediate Feedback
SOPHIE, Auto-Tutor, Lisp tutor, Human tutorial dialog, human learning
Generate educated guesses
Human tutorial dialog, QUADRATIC, PACT
Keep on track
GUIDON, SHOLAR, TRAIN-Tutor
Indicate lack of understanding
Human tutorial dialog, WHY

8. Tutoring and Learning Principles (cont)
Teaching/Learning principle
Tutoring literature
Detect and fix “buggy” knowledge
SCHOLAR, Meno-Tutor, WHY, Buggy, CIRCSIM
Learn deep model
PACT, Atlas-Andes
Learn domain language
Atlas-Andes, Meno-Tutor
Keep track of correct answers
Atlas-Andes
Prioritize learning tasks
WHY
Limit the nesting of the lesson to a handful
Atlas
Summarize what was learned
EXCHECK, TRAIN-Tutor, Meno-Tutor
Provide overall assessment of learning knowledge
WEST, Human tutorial dialog

9. Tutoring and Learning Principles
Start by introducing lesson topics and goals
Advance organizer, Meno-Tutor, tutorial dialog
Use topics of the lesson as a guide
BE&E, UMFE
Subsumption to existing cognitive structure
human learning, WHY, Atlas-Andes
Immediate feedback
Tutoring: SOPHIE, Auto-tutor, LISP tutor, Human tutorial dialog, human learning
Generate educated guesses
Human tutorial dialog, QUADRATIC, PACT

10. Tutoring and Learning Principles (cont)
Keep on track
GUIDON, SCHOLAR, TRAIN-tutor
Indicate lack of understanding
WHY, tutorial dialogue
Detect and fix “buggy” knowledge
SCHOLAR, Meno-Tutor, WHY, Buggy, CIRCSIM
Learn deep models
PACT, Atlas-Andes
Learn domain language
Atlas-Andes, Meno-Tutor

11. Tutoring and Learning Principles (cont)
Keep track of correct answers
Atlas-Andes
Prioritize learning tasks
Why
Summarize what was learned
EXCHECK, TRAIN-tutor, Meno-tutor
Limit the nesting of the lesson to a handful
Atlas
Provide overall assessment of learned knowledge
WEST, Human tutorial dialog

12. Empty cells point to opportunities for future research!
Tutoring and Learning principles used in KA tools [Gil & Kim CogSci-02]
Tutoring/Learning principle
Assimilate
Instruction
Trigger
Goals
Propose
Strategies
Prioritize
Goals & Strats
Design
Presentation
Introduce topics & goals
EXPECT, SEEK2
Use topics of the lesson as a guide
SALT
SEEK2
EXPECT
SALT
Subsumption to existing cog. structure
PROTOS
TEIREISIAS
PROTOS, SALT
Immediate feedback
PROTOS
INSTRUCTO-SOAR
TEIREISIAS
EXPECT
Generate educated guesses
TEIREISIAS
EXPECT
Keep on track
Indicate lack of understanding
INSTRUCTO-SOAR
INSTRUCTO-SOAR
Detect and fix “buggy” K
TAQL
EXPECT,CHIMERA
Learn deep models
Learn domain language
Keep track of answers
SEEK2
Prioritize learned tasks
EXPECT
Summarize what is learned
Assess learned knowledge
KSSn
Empty cells point to opportunities for future research!

13. Viewing KA Activities as Lessons
1) SET UP LESSON AND CHECK BACKGROUND
2) ACCEPT AND RELATE NEW DEFINITIONS
3) TEST AND FIX
4) FIT WITH EXISTING KNOWLEDGE STRUCTURES:
5) ACHIEVE PROFICIENCY
6) REACH CLOSURE

14. Incorporating Tutoring Principles in Dialogue Planning
SET UP LESSON AND CHECK BACKGROUND:
Get the overall topic and purpose of the lesson.
Acquire any assumed prior knowledge before pursuing the lesson.
ACCEPT AND RELATE NEW DEFINITIONS:
Accept new definitions
Ensure that new knowledge is specific as possible.
Ask the user to be complete when enumerating items in terms of the elements and in terms of the significance of the order given.
Get all the information required when existing knowledge indicates it must be provided.
Make all new definitions consistent with existing knowledge.
Connect all new items with the topic of the lesson.
TEST AND FIX:
Test the new body of knowledge and generate tests for the aspects that have not been thoroughly tested.
Fix problems that result from self-checks or from user's indications.
Ensure user checks the reason for the answers, not just the answers themselves.
Confirm new answers that change in light of new knowledge over what the user had seen the answer to be earlier.

15. Incorporating Tutoring Principles in Dialogue Planning (cont)
FIT WITH EXISTING KNOWLEDGE STRUCTURES:
Establish identity of new objects by checking if existing objects appear to be the same.
Generalize definitions if analogous things exist and there could be plausible generalizations.
ACHIEVE PROFICIENCY:
Acquire domain terms to describe new knowledge.
Learn to reason/generate answers efficiently and with shorter explanations.
REACH CLOSURE:
Ensure that the purpose/topics of the lesson were covered and the test questions appropriately answered.

16. Competence and Confidence: Learning Awareness
Capable of assessing:
Competence: What is known, what is unknown
Confidence: What has been tested, what has been checked by the user
Steer the dialogue to improve KB in both counts

17. Awareness Annotations
Annotations to the new body of knowledge:
For each lesson: purpose, assumed background, sub-lessons, overall competence and confidence
For each k item: connection to lesson, relation to other items, identity wrt other items, possible analogies and generalizations, domain terminology details, competence, confidence
For each axiom of a k item: required information, generality, completeness, confidence
Annotations to the dialogue history:
For each user action: changes to the annotations to the new knowledge, acquisition goals achieved and/or activated, possible future KA strategies

18. Ongoing work: Developing KA interfaces based on the principles
SLICK (Skills for Learning to Interactively Capture Knowledge)
SLICK for SHAKEN [Clark et al K-CAP-01]
SLICK for EXPECT [Blythe et al IUI-01]
Will be tested by DARPA this summer

19. Bacterial Transcription: A process model in biology
An Example: Using SLICK to acquire biology concepts in SHAKEN
Bacterial Transcription: A process model in biology
Scenario called Bact-Txn1
Collide
Move-Through
Recognize
Make-Contact
Bacterial-Polymerase
Base-Pair
Promoter
object
base
subevent
next
path
first-subevent
Tangible-Entity
DNA-Melting
Bacterial-DNA
structural-part-of

20. SHAKEN current interface (courtesy of DARPA Rapid Knowledge Formation program)

21. Adding SLICK Interface to SHAKEN

22. Gral acquisition principle
Specific acquisition goal
Educated guesses

23. Awareness Annotations: 1) State

24. Awareness Annotations: 2) History
Shows user’s actions
and their effects in
accomplishing
acquisition goals
or raising new ones
User can view changes
to the state

25. BACKUP

26. Dialogue planning for ITS
Can build library of recipes for given domain
E.g. knowledge construction dialogues (Atlas)
Can use templates
E.g. templates for hint sequence
Anticipate all the bugs and corrections
System controls agenda
Students rarely introduce new topic or ask information-seeking questions
Not directly applicable for KA Tools

27. KA techniques used in building ITS [Murray 99]
Form-based data entry
Special-purpose pre-wired knowledge
Use default values
Use visualization tools
(e.g. curriculum network)
Some uses mechanisms to check accuracy, consistency, completeness,..
(e.g., objectives of the lesson is not covered by the lesson components)

28. KA techniques used in building ITS (cont)
Little knowledge reuse
use of program-by demonstration (limited application)
Difficulty: many diverse and interconnected types of information
Domain model
Teaching strategies
Interface
Student model

29. Our experience in KA
User activities in KA [Tallis, Kim, & Gil, JETAI-2002]
Challenging tasks for end users [Kim & Gil, AAAI-2000; Kim & Gil IUI-2000]
Understanding what pieces of knowledge are related and how
Starting KA tasks when the tool does not point out where to start.
Checking that they are making progress
Managing many errors and gaps
Wish list
More proactive guidance …

30. EXPECT Support for KA interfaces: Key Technologies
I don’t know the computer language…
An English-based editor [Blythe and Ramachandran KAW-99]
Where do I start?
Capture general principles, core theories (e.g. plan evaluation) [Blythe IJCAI-01]
There are many steps; users will be lost…
Script-Based Knowledge Acquisition [Tallis and Gil AAAI-99]
How do I know I am adding the right thing?
System derives and uses model of knowledge interdependencies to understand how different pieces of knowledge are related [Kim & Gil AAAI-99; Kim & Gil AAAI-00; Kim & Gil IJCAI-01] …

31. Designing Dialogue Ideas drawn from Tutoring strategies
Nature of good teacher-student interactions
Learning goals and teaching goals pursued at different points throughout the lesson
User Interaction techniques
Dialogue planning [Allen et al 2001]
Collaborative discourse theory [Sidner & Rich 2000]
Principles in mixed-initiative interfaces [Horvitz 1999]

32. Knowledge used in KA interfaces
General problem solving and task knowledge (e.g., SALT, TAQL)
Prior domain knowledge (EXPECT, INSTRUCTO-SOAR)
General background k (SHAKEN)
Example cases (INSTRUCTO-SOAR, PROTOS, SEEK2, SHAKEN, TEIREISIAS)
Underlying knowledge rep (CHIMAERA, KSSn, SEEK2, TAQL, TEIREISIAS)
Diagnosis and debugging (CHIMAERA, EXPECT, TEIREISIAS)