1. 1 USC INFORMATION SCIENCES INSTITUTE Interactive Knowledge Capture Interactive Knowledge Capture for Problem-Solving Systems Jim Blythe Yolanda Gil Jihie Kim www.isi.edu/ikcap

2. 2 USC INFORMATION SCIENCES INSTITUTE Interactive Knowledge Capture Outline Main elements of our approach Experiences and lessons from earlier work: monolithic systems New work on the Calo project: open systems

3. 3 USC INFORMATION SCIENCES INSTITUTE Interactive Knowledge Capture Acquiring procedure knowledge from users Each user can have unique requirements of a system Their requirements will change over time, perhaps frequently In travel planning, may be different for each trip Users need to be able to modify procedure representations in intelligent systems to address their needs Complements example-based learning

4. 4 USC INFORMATION SCIENCES INSTITUTE Interactive Knowledge Capture Example: adding procedure knowledge to a travel planning tool An assessment tool makes judgments about travel itineraries: e.g., the airline should be United or American e.g., the hotel should be within walking distance, unless I am renting a car Need to add procedure knowledge to tell the system to make a new kind of judgment: the hotel can cost up to 20% more than the government per diem rate for the city. or supporting procedures: to estimate driving time, divide the distance by 55

5. 5 USC INFORMATION SCIENCES INSTITUTE Interactive Knowledge Capture Expects Support for procedure KA: Key Technologies Where does the user start? An acquisition wizard guides the user to start the KA process through a dialog, based on problem-solving methods. KA takes many steps; users will be lost… The acquisition wizard manages the process from end to end. Users dont know the computer language. An English-based procedure editor –Users modify the English paraphrase of the formal representation. A search-based expression composer –Suggests valid reformulations of user sentences. How do users ensure all the needed information is added? An interdependency analyzer understands which pieces of knowledge are used to solve a problem.

6. 6 USC INFORMATION SCIENCES INSTITUTE Interactive Knowledge Capture Main elements of our approach Meta-reasoning: About how knowledge fragments are combined to solve a goal About missing information, effects of changes to KB About the dialog and context Putting knowledge in the users terms: Generate text descriptions to hide the syntax Browse-and-replace interface for procedures Search-based reformulations of free text Test knowledge on real examples

7. 7 USC INFORMATION SCIENCES INSTITUTE Interactive Knowledge Capture Outline Main elements of our approach Experiences and lessons from earlier work: monolithic systems using Expect New work on the Calo project: open systems

8. 8 USC INFORMATION SCIENCES INSTITUTE Interactive Knowledge Capture Acquisition wizard Dialog with user to start the process. Some questions use menus or text input. Others use the English editor to refine procedural knowledge

9. 9 USC INFORMATION SCIENCES INSTITUTE Interactive Knowledge Capture English-based Procedure editor NL description of method Alternatives for selected text fragment (multiply (obj (look-up (obj fsa-per-diem-hotel-rate) (for (r-city ?hotel)))) (by 1.2))

10. 10 USC INFORMATION SCIENCES INSTITUTE Interactive Knowledge Capture Test new knowledge immediately Each element is defined and checked using constraints Softools, developed under the DARPA AcT program

11. 11 USC INFORMATION SCIENCES INSTITUTE Interactive Knowledge Capture U: new port: Havana S: I need to know if it is an airport or a seaport U: seaport S: I need to know the location and the berths... r-location r-berths r-pols r-piers r-storage-area (evaluate (obj coa) (wrt logistics))... (r-location port)... (r-berths seaport) port airport seaport inland waterway seaport maritime seaport Domain Ontology Problem-Solving Methods Global Procedure Analysis: Test interdependencies, both rule-rule and rule-data INTERDEPENDENCIES Interdependencies guide Knowledge Acquisition

12. 12 USC INFORMATION SCIENCES INSTITUTE Interactive Knowledge Capture When combined, the tools use information from each other Application Acquisition wizard Acquisition analyzer Interdependency analyzer Procedure editor Relation/concept editor Instance editor Expression composer

13. 13 USC INFORMATION SCIENCES INSTITUTE Interactive Knowledge Capture The acquisition wizard Guides the user through the initial steps of adding new knowledge. Structures the knowledge to be added using default procedural knowledge. Questions are generated from a problem-solving theory.

14. 14 USC INFORMATION SCIENCES INSTITUTE Interactive Knowledge Capture Problem-solving theory for plan evaluation A hierarchy of generic types of plan judgments with default procedural knowledge. DEFINED: check that the value is less than the maximum value ASK USER: compute a maximum value for each object judgment global judgmentlocal judgmentbounds checkextensional check completeness judgment hotel cost judgment upper boundlower boundpositivenegative Warn if the value is too large?

15. 15 USC INFORMATION SCIENCES INSTITUTE Interactive Knowledge Capture Benefits of integration: The acquisition wizard and the method editor Each component receives information from the other that helps the user: The wizard provides to the editor: An initial version of the method, with the correct capability An expectation of the result type of the method The editor provides to the wizard: A more detailed method result type Used to help classify the new task in the ontology

16. 16 USC INFORMATION SCIENCES INSTITUTE Interactive Knowledge Capture bounds check upper boundlower bound Warn if the value is too large?

17. 17 USC INFORMATION SCIENCES INSTITUTE Interactive Knowledge Capture The expression composer Assists users in formalizing informal statements by suggesting composite expressions using terms the system understands. Uses an ontology of terms and relations, and synonyms derived from WordNet with extensions. Makes breadth-first forward search, matching keywords

18. 18 USC INFORMATION SCIENCES INSTITUTE Interactive Knowledge Capture Expression composer example User types: max staging post landing Tool suggests: find the maximum of the landing distance available of the runways of the forward staging post. find objectof landing-distance-available runways ?forward-staging-post maximum Function call reformulation Typed variable combining queries Information element

19. 19 USC INFORMATION SCIENCES INSTITUTE Interactive Knowledge Capture Integrated with the method editor Constructs compound terms in KB that include user terms and have desired type Anytime breadth-first search through space of terms

20. 20 USC INFORMATION SCIENCES INSTITUTE Interactive Knowledge Capture Outline Main elements of our approach Experiences and lessons from earlier work: monolithic systems New work on the Calo project: open systems

21. 21 USC INFORMATION SCIENCES INSTITUTE Interactive Knowledge Capture Tailor: interactive task acquisition for the Calo office assistant A personal assistant that manages everyday tasks. Learns from experience, interacts with the user naturally through several channels. Large DARPA-funded project run by SRI. Goal2 ACT8 sleeping Fact1 ACT2 normal Goal3 ACT3 sleeping Intention Graph Cue: (TEST (source vision new)) ACT2 Process Library Process Execution (source commanders-vision new) (ACHIEVE (satisfy Incoming-INs)) New Facts & Goals External World 1 2 3 4 5 6 7 8 Cue: (ACHIEVE (satisfy source)) ACT1 Facts & Goals (ACHIEVE (derive IN-list source)) ACT1 current Spark is Calos task manager. (A PRS-based system developed by Myers et al.)

22. 22 USC INFORMATION SCIENCES INSTITUTE Interactive Knowledge Capture Approach and challenges in acquiring knowledge within CALO Carry key elements forward: Explicit reasoning about knowledge dependencies and acquisition context Expressing the procedures in the users terms Earlier work used Expect, a monolithic system using custom problem-solving system, language, methods, ontology,.. Now use Sparks performance element and procedures, ontologies provided by several groups.

23. 23 USC INFORMATION SCIENCES INSTITUTE Interactive Knowledge Capture Example: purchasing a laptop Spark manages the workflow as the user purchases a laptop. Choose a model, find bids, get authorization, track purchase… During the process, Spark finds that the order cannot be completed, because a manager who must authorize the purchase is not available. User should be able to tell the system You dont need authorization when the cost is less than $2000

24. 24 USC INFORMATION SCIENCES INSTITUTE Interactive Knowledge Capture Initial solution Make direct analysis of the Spark procedures Encapsulates a model of Sparks behaviour Requires additions to language, e.g. types, purpose of subtasks Combine the procedure analysis with the expression composer to help interpret user sentences. Dialog is currently implicit Currently provide follow-up questions based on analysis.

25. 25 USC INFORMATION SCIENCES INSTITUTE Interactive Knowledge Capture Expression composer suggests valid condition User can explore modifications based on different assumptions Response to You dont need authorization when the cost is less than $2000 Automatic generation of text from Spark procedure definitions

26. 26 USC INFORMATION SCIENCES INSTITUTE Interactive Knowledge Capture Tailor: summary and future work Currently allow users to modify existing procedures, using global analysis and expression composer. Next: Support defining new tasks, integrate with advice for Spark. Model dialog: build templates for supporting theory, working with Allen/Ferguson on dialog model.

27. 27 USC INFORMATION SCIENCES INSTITUTE Interactive Knowledge Capture Summary Our work allows users to add procedure knowledge to both custom and pre-existing intelligent systems. Combination of global analysis and term reformulation can allow users and the system to reach shared understanding. Interactive knowledge capture complements example-based learning to allow intelligent systems to adapt. Evaluation strategies: Ablation user studies with fixed and free tasks. Good results with Expect & Constable, still to do with Tailor.