1. DARPA ITO/MARS Project Update Vanderbilt University A Software Architecture and Tools for Autonomous Robots that Learn on Mission K. Kawamura, M. Wilkes, R. A. Peters II, D. Gaines Vanderbilt University Center for Intelligent Systems http://shogun.vuse.vanderbilt.edu/CIS/IRL/ 12 January 2000

2. Vanderbilt MARS Team Kaz Kawamura, Professor of Electrical & Computer Engineering. MARS responsibility - PI, Integration Dan Gaines, Asst. Professor of Computer Science. MARS responsibility - Reinforcement Learning Alan Peters, Assoc. Professor of Electrical Engineering. MARS responsibility - DataBase Associative Memory, Sensory EgoSphere Mitch Wilkes, Assoc. Professor of Electrical Engineering. MARS responsibility - System Status Evaluation Jim Baumann, Nichols Research MARS responsibility - Technical Consultant Sponsoring Agency Army Strategic Defense Command

3. IMPACT: NEW IDEAS:GRAPHIC: SCHEDULE: Learning with a DataBase Associative Memory Sensory EgoSphere Attentional Network Robust System Status Evaluation Mission-level interaction between the robot and a Human Commander. Enable automatic acquisition of skills and strategies. Simplify robot training via intuitive interfaces - program by example. A Software Architecture and Tools for Autonomous Mobile Robots That Learn on Mission Year 1Year 2 IMA agents and schema Learning algorithms Test Demo Final Demo Demo III COMM LEARNING CMDR SQUAD 1 SQUAD 2 SQUAD N... SELF ENVIR IMA 

4. Project Goal Develop a software control system for autonomous mobile robots that can: 1.accept mission-level plans from a human commander, 2.learn from experience to modify existing behaviors or to add new behaviors, and 3.share that knowledge with other robots.

5. Project Approach Use IMA, to map the problem to a set of agents. Develop System Status Evaluation (SSE) for self diagnosis and to assess task outcomes for learning. Develop learning algorithms that use and adapt prior knowledge and behaviors and acquire new ones. Develop Sensory EgoSphere, behavior and task descriptions, and memory association algorithms that enable learning on mission.

6. MARS Project: The Robots ISAC HelpMate ATRV-Jr.

7. Communications Agent Act./Learning Agent Commander Agent Squad Agent 1 Squad Agent 2 Squad Agent n... Self Agent Environment Agent IMA The IMA Software Agent Structure of a Single Robot

8. Robust System Status Analysis Timing information from communication between components and agents will be used. Timing patterns will be modeled. Deviations from normal indicate “discomfort.” Discomfort measures will be combined to provide system status information.

9. What Do We Measure? Visual Servoing Component –error vs. time Arm Agent –error vs. time, proximity to unstable points Camera Head Agent –3D gaze point vs. time Tracking Agent –target location vs. time Vector Signals/Motion Links –log when data is updated

11. Commander Interface

14. Obstacle Avoidance

15. Planning/Learning Objectives Integrated Learning and Planning –learn skills, strategies and world dynamics –handle large state spaces –transfer learned knowledge to new tasks –exploit a priori knowledge Combine Deliberative and Reactive Planning –exploit predictive models and a priori knowledge –adapt given actual experiences –make cost-utility trade-offs

16. Overview of Approach

17. Example: Different Terrains

18. Generate Abstract Map Nodes selected based on learned action models Each node represents a navigation skill

19. Generate Plan in Abstract Network Plan makes cost-utility trade-offs Plans updated during execution

20. Action Model Learning –adapted MissionLab to allow experimentation (terrain conditions) –using regression trees to build action models Plan Generation –developed prototype Spreading Activation Network –using to evaluate potential of SAN for plan generation Planning/Learning Status

21. Role of ISAC in MARS Inspired by the structure of vertebrate brains a fundamental human-robot interaction model sensory attention and memory association learning sensory-motor coordination (SMC) patterns learning the attributes of objects through SMC ISAC is a testbed for learning complex, autonomous behaviors by a robot under human tutelage.

22. System Architecture A A A A A A A A Human Agent Robot Human Robot Self Agent Software System IMA Primitive Agent Hardware I/O

23. Next Up: Peer Agent We are currently developing the peer agent. The peer agent encapsulates the robot’s understanding of and interaction with other (peer) robots.

24. System Architecture: High Level Agents human agent self agent peer agent peer agent environment agent object agent object agent Due to the flat connectivity of IMA primitives, all high level agents can communicate directly if desired.

25. Robot Learning Procedure The human programs a task by sequencing component behaviors via speech and gesture commands. The robot records the behavior sequence as a finite state machine (FSM) and all sensory-motor time- series (SMTS). Repeated trials are run. The human provides reinforcement feedback. The robot uses Hebbian learning to find correlations in the SMTS and to delete spurious info.

26. Robot Learning (cont’d) The robot extracts task dependent SMC info from the behavior sequence and the Hebbian-thinned data. SMC occurs by associating sensory-motor events with behaviors nodes in the FSMs. The FSM is transformed into a spreading activation network (SAN). The SAN becomes a task record in the database associated memory (DBAM) and is subject to further refinements.

27. Human Agent: Human Detection

28. Human Agent: Recognition

29. Human Agent: Face Tracking

30. Schedule