IL Kickoff Meeting June 20-21, 2006 DARPA Integrated Learning POIROT Project 1 Learning Hierarchical Task Networks by Analyzing Expert Traces Pat Langley Tolga Konik Negin Nejati Institute for the Study of Learning and Expertise Palo Alto, California

IL Kickoff Meeting June 20-21, 2006 DARPA Integrated Learning POIROT Project 2 Formulation of the Learning Task Given: A set of domain operators with known effects A worked out problem solution that consists of The goal to be achieved in the problem A sequence of operator instances that achieves the goal A related sequence of intermediate problem states Find: A hierarchical task network that Reproduces the solution to the training problem Generalizes well to related problems in the domain

IL Kickoff Meeting June 20-21, 2006 DARPA Integrated Learning POIROT Project 3 The I CARUS Architecture ConceptualMemory BeliefMemory Goal/IntentionMemory ConceptualInference SkillExecution Perception Environment PerceptualBuffer Skill Learning MotorBuffer Skill Retrieval and Selection Skill Memory

IL Kickoff Meeting June 20-21, 2006 DARPA Integrated Learning POIROT Project 4 Representing Long-Term Structures Conceptual clauses: A set of relational inference rules with perceived objects or defined concepts in their antecedents; Skill clauses: A set of executable skills that specify: a head that indicates a goal the skill achieves; a single (typically defined) precondition; a set of ordered subgoals or actions for achieving the goal. These define a specialized class of hierarchical task networks in a syntax very similar to Nau et al.s SHOP2 formalism. Beliefs, goals, and intentions are instances of these structures. I CARUS encodes two forms of general long-term knowledge:

IL Kickoff Meeting June 20-21, 2006 DARPA Integrated Learning POIROT Project 5 Representing Concepts (Axioms) ((in-rightmost-lane ?self ?clane) :percepts ((self ?self) (segment ?seg) (line ?clane segment ?seg)) :relations ((driving-well-in-segment ?self ?seg ?clane) (last-lane ?clane) (not (lane-to-right ?clane ?anylane))) ) ((driving-well-in-segment ?self ?seg ?lane) :percepts ((self ?self) (segment ?seg) (line ?lane segment ?seg)) :relations ((in-segment ?self ?seg) (in-lane ?self ?lane) (aligned-with-lane-in-segment ?self ?seg ?lane) (centered-in-lane ?self ?seg ?lane) (steering-wheel-straight ?self)) ) ((in-lane ?self ?lane) :percepts ((self ?self segment ?seg) (line ?lane segment ?seg dist ?dist)) :tests ((> ?dist -10) (<= ?dist 0)) ) Primitive Concepts Nonprimitive Concepts

IL Kickoff Meeting June 20-21, 2006 DARPA Integrated Learning POIROT Project 6 ((in-rightmost-lane ?self ?line) :percepts ((self ?self) (line ?line)) :start ((last-lane ?line)) :subgoals ((driving-well-in-segment ?self ?seg ?line)) ) ((driving-well-in-segment ?self ?seg ?line) :percepts ((segment ?seg) (line ?line) (self ?self)) :start ((steering-wheel-straight ?self)) :subgoals ((in-segment ?self ?seg) (centered-in-lane ?self ?seg ?line) (aligned-with-lane-in-segment ?self ?seg ?line) (steering-wheel-straight ?self)) ) ((in-segment ?self ?endsg) :percepts ((self ?self speed ?speed) (intersection ?int cross ?cross) (segment ?endsg street ?cross angle ?angle)) :start ((in-intersection-for-right-turn ?self ?int)) :actions(( steer 1)) ) Primitive Skill Clauses Nonprimitive Skill Clauses Representing Skills (Methods)

IL Kickoff Meeting June 20-21, 2006 DARPA Integrated Learning POIROT Project 7 Each concept is defined in terms of other concepts and/or percepts. Each skill is defined in terms of other skills, concepts, and percepts. concepts skills I CARUS organizes both concepts and skills in a hierarchical manner. Hierarchical Structure of Memory

IL Kickoff Meeting June 20-21, 2006 DARPA Integrated Learning POIROT Project 8 Hierarchical Structure of Memory For example, the skill highlighted here refers directly to the highlighted concepts. I CARUS interleaves its long-term memories for concepts and skills. concepts skills

IL Kickoff Meeting June 20-21, 2006 DARPA Integrated Learning POIROT Project 9 Basic I CARUS Processes Concepts are matched bottom up, starting from percepts. Skill paths are matched top down, starting from intentions. I CARUS matches patterns to recognize concepts and select skills. concepts skills

IL Kickoff Meeting June 20-21, 2006 DARPA Integrated Learning POIROT Project 10 Impasse-Driven Analytical Learning Skill Hierarchy Reactive Execution Analytical Learning Experts Primitive Skill Sequence … Effects of Primitive skills Learned Skills If Impasse Problem ? Initial State Goal

IL Kickoff Meeting June 20-21, 2006 DARPA Integrated Learning POIROT Project 11 Learning HTNs by Trace Analysis concepts primitive skills

IL Kickoff Meeting June 20-21, 2006 DARPA Integrated Learning POIROT Project 12 Skill Chaining concepts primitive skills Learning HTNs by Trace Analysis

IL Kickoff Meeting June 20-21, 2006 DARPA Integrated Learning POIROT Project 13 Concept Chaining concepts primitive skills Learning HTNs by Trace Analysis

IL Kickoff Meeting June 20-21, 2006 DARPA Integrated Learning POIROT Project 14 unstack C B on B A hand-empty putdown C putdownable C unstackable B A clear A unstack B A clear B unstackable C B A B C A B CAC B A B C Constructing an Explanation concepts primitive skills

IL Kickoff Meeting June 20-21, 2006 DARPA Integrated Learning POIROT Project 15 A B C unstack C B on B A hand-empty putdown C putdownable C unstackable B A clear A unstack B A clear B unstackable C B A B C A B CAC B concepts primitive skills From an Explanation to an HTN

IL Kickoff Meeting June 20-21, 2006 DARPA Integrated Learning POIROT Project 16 on ?y ?x hand-empty putdown ?z putdownable ?z unstackable ?y ?x clear ?x clear ?y unstackable ?z ?y unstack ?z ?y A B C A B CAC B A B C unstack ?y ?x From an Explanation to an HTN concepts primitive skills

IL Kickoff Meeting June 20-21, 2006 DARPA Integrated Learning POIROT Project 17 Key Ideas of the Approach Constrained form of hierarchical task networks Each skill clause/method has a goal as its head Each method has one (possibly defined) precondition The resulting semi-lattice makes learning tractable Learning involves analyzing the expert trace Explanation draws on a form of goal regression Each step in the explanation becomes an HTN method Similar to explanation-based learning for planning but retains the explanation structure

IL Kickoff Meeting June 20-21, 2006 DARPA Integrated Learning POIROT Project 18 Related Research Nonincremental, knowledge-lean approaches Behavioral cloning (Sammut, 1996; Urbancic & Bratko 1994) Relational induction from traces (e.g., Reddy & Tadepalli, 1997) Incremental, knowledge-intensive approaches Explanation-based learning (e.g., Shavlik, 1989; Mooney, 1990) Derivational analogy (e.g., Veloso & Carbonell, 1993) Programming by demonstration (e.g., Lau et al., 2003)

IL Kickoff Meeting June 20-21, 2006 DARPA Integrated Learning POIROT Project 19 Plans for Future Research Extend framework to use and learn partial-order skills Augment approach to use known subtasks during learning Extend method to learn skills with negated goals and subgoals Modify approach to handle partially observable traces Extend system to learn skills with uncertain outcomes

IL Kickoff Meeting June 20-21, 2006 DARPA Integrated Learning POIROT Project 20 End of Presentation