1 USC INFORMATION SCIENCES INSTITUTE EXPECT TEMPLE: TEMPLate Extension Through Knowledge Acquisition Yolanda Gil Jim Blythe Information Sciences Institute University of Southern California {gil,

2 USC INFORMATION SCIENCES INSTITUTE EXPECT Acquiring Planning Knowledge Problem: SOF users need to add knowledge to these planning tools  ROEs, commander’s guidance  Plan evaluation/critiquing criteria  Highlight the information that is important to them  Add/extend templates Approach: provide knowledge acquisition tools to adapt and extend pre-existing planning knowledge  Exploit ontologies and background knowledge so users don’t have to start from scratch  KA Scripts guide the user through multiple steps  Users manipulate English paraphrases of internal representations Benefits:  Users can extend the tool’s baseline knowledge for the operation

3 USC INFORMATION SCIENCES INSTITUTE EXPECT Prototype for adding plan critiques: Expect’s PSM Tool Questions formulated based on background knowledge User adds detailed knowledge through English paraphrases

4 USC INFORMATION SCIENCES INSTITUTE EXPECT The next 100 days Allow users to specify and customise “sentinels” that check for new information and alert planners when needed.  Our tools generate Java. Extend ontologies and background knowledge to handle SOF domain. Integrate with one of the jumpstart applications, probably the travel planning tool, using InterAcT.

5 USC INFORMATION SCIENCES INSTITUTE EXPECT Backup slides Description of approach, tools and experiment from HPKB project.

6 USC INFORMATION SCIENCES INSTITUTE EXPECT Key Technologies Guiding users through knowledge acquisition scripts [Tallis and Gil 99] that capture typical dialogues that users follow to enter new knowledge step by step Exploiting domain-independent background knowledge about plan evaluation and critiquing [Blythe & Gil 99] that use background knowledge about plan evaluation and critiquing to guide the dialog An English-based editor [Blythe & Ramachandran 99] that lets the user add or modify internal knowledge by manipulating its English paraphrase, without having to see or understand the internal formal representation

7 USC INFORMATION SCIENCES INSTITUTE EXPECT Architecture of TEMPLE (Server) Constraint Acquisition UI (Client) Constraint wizard Constraint viewer Acq Domain knowledge Domain constraints Domain methods Domain models and templates

8 USC INFORMATION SCIENCES INSTITUTE EXPECT Evaluation and Critiquing Knowledge Plan ontology (PLANET) Ontology of critiques Submethods for checking plan resources Submethods for checking plan structure Reused knowledge (ontologies and methods) Domain-specific critiques Domain-specific submethods Domain-specific plan critiquing and evaluation system Domain-specific knowledge Ontology of resources

9 USC INFORMATION SCIENCES INSTITUTE EXPECT An Ontology of Plan Evaluation Criteria Captures general knowledge of how to evaluate plans with respect to standard norms of plan development Ill-formed description Statement critique Link critique Structure critique Complete statement Correct statement Clear statement ako Correct link ako Does the purpose of supporting effort X support the main effort? isa Does have sufficient combat power to accomplish its mission? isa Complete plan ako... isaDoes the COA include a statement of the reserve forces?

10 USC INFORMATION SCIENCES INSTITUTE EXPECT KA Scripts Helps the user add new critiques by using a background theory of plan evaluation and critiquing. KA Scripts guide the dialog with the user about the new critique (wizard-type interaction). The tool creates some of the needed methods for the critiques, helps the user to create new ones (by suggesting initial templates), and ensures consistency with existing knowledge.

11 USC INFORMATION SCIENCES INSTITUTE EXPECT English-Based Editor Generates automatically English paraphrases of problem-solving fragments, and presents alternative text to replace parts of the paraphrase based on the ontologies and background knowledge NL description of method Alternatives for selected text fragment

12 USC INFORMATION SCIENCES INSTITUTE EXPECT First experiment: An ablation test on the PSM-Based KA Scripts Hypothesis: the PSM-Based KA Scripts significantly reduce the expertise and the effort required to add a new critique to the knowledge base. KA tasks: add two new critiques to the EXPECT COA critiquer (a completeness check and a resource check) Knowledge (and tool) ablation experiment: Two tasks done using PSM-Based KA Scripts, two tasks done without Subjects: four Army officers, previously trained on EXPECT’s language for a day

13 USC INFORMATION SCIENCES INSTITUTE EXPECT Sample Tasks Given to Subjects Simple critique: Add a critique to check if the COA has a security statement. Complex critique: Add a critique to check if each task in the COA has sufficient force ratio. To compute force ratio, divide the sum of combat powers of the Blue units assigned to the task by the sum of combat powers of the Red units acted on by the task. (Two other comparable tasks were also used)

14 USC INFORMATION SCIENCES INSTITUTE EXPECT Quantitative results: what users could do Users could complete more tasks using the PSM- based KA scripts LEGEND: indicates total tasks

15 USC INFORMATION SCIENCES INSTITUTE EXPECT Quantitative results: speed improvements Time reduction using the PSM-based KA Scripts Time in minutes

16 USC INFORMATION SCIENCES INSTITUTE EXPECT Axiom acquisition rates: Experiment with PSM-Based KA Scripts Adding small amounts of new knowledge Adding larger amounts of new knowledge with PSM-Based KA Scripts 2.12 ax/min 1.26 ax/min with ablated version 1.1 ax/min N/A (users were not able to do tasks)

17 USC INFORMATION SCIENCES INSTITUTE EXPECT Summary Using the PSM-Based KA Scripts significantly reduced the time taken to add a critique Using the PSM-Based KA Scripts, all four subjects successfully added simple critiques to the EXPECT critiquer; three of them successfully added more complex critiques. Without the PSM-Based KA Scripts, three out of four subjects successfully added simple critiques and two added more complex critiques. Comments on the tool usability were positive in all cases.

18 USC INFORMATION SCIENCES INSTITUTE EXPECT Second experiment with PSM-Based KA Scripts and English-Based Editor Hypothesis: the combination of the PSM-Based KA Scripts and English-based editor allows a user with very little training to add new critiques. Single subject usability test: A subject was briefed in COA critiquer and the KA interface (but not about EXPECT) for 20min and asked to add two critiques using the tool KA tasks: add two new critiques to the EXPECT COA critiquer (a completeness check and a resource check), same used in the previous experiment Subject: an Army officer with no EXPECT training

19 USC INFORMATION SCIENCES INSTITUTE EXPECT Results The subject was able to add two new critiques of both low and medium complexity. The time taken was comparable to that for the other four subjects that had previous training in Expect: Time in minutes

20 USC INFORMATION SCIENCES INSTITUTE EXPECT EXPECT: A User-Centered Framework for Developing KBSs Method instantiator Method instantiator Knowledge Base Domain ontologies and factual knowledge Problem solving methods Domain dependent KBS compiler KBS compiler Knowledge-Based System Interdependency Model (IM) EXPECT Ontologies and Method libraries Plans (PLANET) Evaluations and Critiques Evaluation PSMs Resources (OZONE) COA ontologies CYC/Sensus Upper Instrumentation KA Strategies KA tools EMeD KA Scripts PSMTool

21 USC INFORMATION SCIENCES INSTITUTE EXPECT EXPECT: A User Centered Approach for Knowledge-Based Planning Tools Knowledge acquisition technology that can guide users to specify planning knowledge and develop planning tools  Expressive representations –Loom/Powerloom KR&R –EXPECT’s language to represent problem solving knowledge  Powerful reasoners –Loom/Powerloom pattern classifier & reasoners –abstract problem solving through partial evaluation ex: how to move from a to another  Explicit models of planning knowledge and plan reasoners: –PLANET ontology of plans, OZONE resource ontology –plan evaluation and planning methods  Expectation-based knowledge acquisition tools –Derive interdependencies between individual knowledge fragments –KA Scripts to guide users in completing complex modifications