1. Collaborative Learning of Hierarchical Task Networks from Demonstration and Instruction
Anahita Mohseni-Kabir, Sonia Chernova and Charles Rich
Worcester Polytechnic Institute

2. Project Objectives and Contributions
Main Goal: Learning complex procedural tasks from human demonstration and instruction in the form of hierarchical task networks and applying it to car maintenance domain
Project Contributions:
Unified system that integrates hierarchical task networks (HTNs) and collaborative discourse theory into the learning from demonstration
Learning task model from a small number of demonstrations
Generalization techniques
Integration of mixed-initiative interaction into the learning process through question asking

3. Related Work Collaborative Discourse Theory
Disco (ANSI/CEA-2018 standard) (Grosz and Sidner, 1986 and Rich et al., 2001)
Learning from Demonstration
Mix LfD and planning (Nicolescu and Mataric, 2003)
Integrate HTN and LfD (Rybski et al., 2007)
Learn from Instruction (Mohan and Laird, 2011)
Learn the HTN from task’s traces (Garland et al., 2001)
Segmentation (Niekum et al., 2012)
Active learning (Cakmak and Thomaz, 2012)

4. System Architecture Primitive actions
Primitive and Non-primitive actions
Task model visualization
Questions and answers
Primitive actions

5. Task Structure Learning
Task Hierarchy
Top-Down
Bottom-Up
Mix of Top-Down
and Buttom-Up
Temporal Constraints
Single demonstration
Data flow

6. System Overview

7. Generalization Input Generalization Merging multiple demonstrations
Part/whole generalization
Type generalization
Merging multiple demonstrations
Ontology

8. System Overview

9. Question Asking Question Type Repeated steps Grouping steps
Example
Repeated steps
Should I(robot) execute UnscrewStud on other objects of type Stud of LFhub?
Grouping steps
Should I add a new task with Unscrew and PutDown as its steps?
Applicability condition of alternative recipes
What is the applicability condition of Rotate’s recipe with these steps?
New task name
What is the best name that describes this new task?
Input of a task
Please specify the input of Unscrew.
Execution of one of the alternative recipes
Should I achieve Rotate by executing recipe1 or recipe2?

11. Performance Tire rotation task
Six primitive actions: Unscrew, Screw, Hang, Unhang, PutDown and PickUp
Complete execution of two recipes of tire rotation requires 128 steps
Complete teaching of the HTN (two recipes) on average requires 26 demonstration interactions
E.g., 15 demonstrations, 11 instructions, 11 question responses

12. Conclusion and Future Work
Make the interaction as natural as possible by making the UI and robot look like a unified system
Do user study and use the real robot instead of the simulation
Learn applicability conditions and pre/postconditions of the tasks
Failure detection and recovery
This work is supported in part by ONR contract N , in collaboration with Dmitry Berenson, Jim Mainprice , Artem Gritsenko, and Daniel Miller.

13. References
Barbara J. Grosz and Candace L. Sidner. Attention, intentions, and the structure of discourse. Comput. Linguist., 12(3):175–204, July 1986.
Charles Rich, Candace L Sidner, and Neal Lesh. Collagen: applying collaborative discourse theory to human-computer interaction. AI Magazine, 22 (4):15, 2001.
Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and Autonomous Systems, 57(5):469–483, 2009.
Paul E Rybski, Kevin Yoon, Jeremy Stolarz, and Manuela M Veloso. Interactive robot task training through dialog and demonstration. In ACM/IEEE Int. Conf. on Human-Robot Interaction, pages 49–56, 2007.

14. References
Scott Niekum, Sarah Osentoski, George Konidaris, and Andrew G Barto. Learning and generalization of complex tasks from unstructured demonstrations. In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, pages 5239–5246, 2012.
Maya Cakmak and Andrea L Thomaz. Designing robot learners that ask good questions. In ACM/IEEE International Conference on Human-Robot Interaction, pages 17–24. ACM, 2012.
Monica N Nicolescu and Maja J Mataric. Natural methods for robot task learning: Instructive demonstrations, generalization and practice. In AAMAS, pages 241–248, 2003.

15. Merging