1. Cognitive Control within a Parallel Distributed MultiAgent-Based Cognitive Robot Architecture: From Body-Sensor Integration to Body-Mind Integration

2. New Challenge Integration of Body and Mind Integration of Body and Mind  Cognitive Robot †  Conscious Robot  † K. Kawamura, D. Noelle, et al., A Multi-Agent Approach to Self-Reflection for Cognitive Robots”, the 11 th International Conference on Advanced Robotics, Coimbra, Portugal, June 30 – July 3, 2003.

3. MultiAgent-Based Cognitive Architecture for ISAC

4. Compound Agents Compound Agents  Self Agent – Representation of Robot’s Self  Human Agent – Representation of Humans  Memory Structures  STM – Sensory EgoSphere  LTM – Procedural memory  WM – under construction

5. Four Key Cognitive Abilities We assume that there are four key cognitive abilities cognitive robots must have in order to become effective members of human society: 1. Self-awareness   System Health Monitoring   Task Monitoring 2. Awareness of others   Intention detection   Mutual Spatial & Timing Control 3. Cognitive Control   Executive Controller   Working Memory 4. Self-reflection   Ability to reflect experiences gained   Ability to think ahead Self Agent Human Agent Mental Experimentation Agent Central Executive Controller

6. Self Agent Self Agent with memory structures

7. Levels of Monitoring and Action within the Self Agent

8. Levels of Interaction within the Human Agent

9. New Architecture for ISAC

10. † STM: Sensory EgoSphere (SES) † Purpose of the SES Spatio-temporal method for storing sensory events Spatio-temporal method for storing sensory events Integration of multiple sensory modalities Integration of multiple sensory modalities Short-term memory of events Short-term memory of events Egocentric, topological mapping of locations Egocentric, topological mapping of locations R.A. Peters II, K.A. Hambuchen, K. Kawamura, and D.M. Wilkes, "The Sensory Ego-Sphere as a Short- Term Memory for Humanoids", Proc. IEEE-RAS Int'l. Conf. on Humanoid Robots, Waseda University, Tokyo, Nov 22-24, 2001, pp. 451-459.

11. SMC-STM Experiment - Object Learning from Human Pointing ISAC is taught to reach 5 objects on a table

12. SMC-STM Experiment - Object Learning from Human Pointing Data Flow for the human-guided motion generation and behavior derivation demo

13. SMC-STM Experiment - Object Learning from Human Pointing   Robot learns objects upon human instruction   Scans table for human pointing finger   Registers object on Sensory EgoSphere (SES)   Robot retrieves position of objects at human direction   Parses request   Retrieves object information from SES   Looks and/or points to object of interest

14. Navigation based on Egocentric representations SES represents the current perception of the robot LES represents the expected state of the world Comparison of these provides the best estimate direction towards a desired region LES SES SES- and LES-Based Navigation K. Kawamura, R.A. Peters II, D.M. Wilkes, A.B. Koku and A. Sekman, “Towards Perception-Based Navigation using EgoSphere”, Proc. of the International Society of Optical Engineering Conference (SPIE), October 28-20, 2001. Novel Approach: Range-free perception-based navigation

15. Demo

16. PM stores motion skills such as how to reach to a point PM is a combination of primitive motions which are generated by several motion generation techniques Interpolation method we are using is called the Spatio-Temporal Isomap LTM database currently contains Primitive Memory (PM) Long-Term Memory (LTM)

17. Behavior Generation and Learning Basic behaviors are derived from the motion taught by human Basic behaviors are derived from the motion taught by human Spatio-temporal Isomap is used to derive primitive and meta-level behaviors Spatio-temporal Isomap is used to derive primitive and meta-level behaviors Behaviors are stored in Procedural Memory(PM) in Long Term Memory of robot Behaviors are stored in Procedural Memory(PM) in Long Term Memory of robot † Isomap Method † for Behavior-Based Motion Generation † † J. B. Tenenbaum, V. de Silva, and J. C. Langford,”A global geometric framework for nonlinear dimensionality reduction”, Science, 290(5500), pp. 2319–2323, 2000.

18. Behavior Generation and Learning (cont’d) Structure of LTM database including steps to generate behaviors

19. Verbs and Adverbs Method of generating a wide variety of behaviors from a few exemplar motions First used in animation community º Similar to primitive motion modules  º C. Rose, M. F. Cohen and B. Bodenheimer, “Verbs and adverbs: multidimensional motion interpolation”, IEEE Computer Graphics and Applications, 18:5, Sept-Oct 1998, 32–40.

20. Verbs and Adverbs (con’t) ISAC performing a reaching motion using the verbs and adverbs technique. ISAC preparing to grasp Barney

21. Verbs and Adverbs (con’t)   Verbs   General description of motions   Joint angles as function of time   Adverbs   Parameters of a verb   A verb can have any number of adverbs   Adverbs are independent of each other   Can be subjective values such as sad or happy

22. Working Memory (WM)   Maintains transient information that is critical for decision-making in current context   Allows one to remember a phone number - retaining it only long enough to dial it   Of particular interest are the lateral portions of the prefrontal cortex (PFC) PFC Schematic diagram of the visual DA model; mechanisms in this example detect shape or color to drive the desired visual response 1. PFC: http:www. driesen.com/prefrontal_cortex.htm 2. A. Cohen and U. Feintuch, “The dimensional-action system: a distinct visual system,” Common Mechanisms in Perception and Action: Attention and Performance XIX. W. Prinz and B. Hommel, Eds. New York: Oxford University Press, 2002.

23. Cognitive Control and the CEC   Cognitive Control in the human brain:   The anterior cingulated cortex (ACC) is responsible for executive or cognitive control in our brain   ACC helps focus attention and select actions   Central Executive Controller (under construction) executes tasks based on:   Attention system supervision   Memory management   Context processing   Non-routine behavior selection/control

24. Goal Oriented Perception-Action Mapping

25. Cognitive Experiment Demonstrates how the CEC, STM, WM and LTM work together to carry out a cognitive task

26. Cognitive Control and the CEC The Central Executive Controller (CEC), currently under construction, is responsible for the cognitive control during action decision A basic CEC is designed to provide the following functions:   Controls task contexts via gating   Allows reactive processes to process tasks whenever possible   Provides deliberation when reactive processes cannot handle the task by themselves   Allows access to self reflection when impasses arise

27. Cognitive Experiment - Stroop Experiment REDBLUE REDYELLOW GREENYELLOW YELLOWRED BLUEGREEN GREENBLUE A word and color are shown, then the subject must name the color in which a word is written

28. Modified Stroop Experiment RED BLUE RED   Robot is told to choose a particular color/ word (e.g., red “BLUE”)   SES maintains different representations of the environment for color and word (Note: Color recognition is VERY sensitive to environmental lighting conditions)   These are held in WM after passing through the attention system

29. RED BLUE RED Modified Stroop Experiment (Cont’d) “Point to red BLUE” Send attention parameters Attention system filters data Human CEC Attention Store filtered data WM Retrieve motions CEC Perform Action CEC Word BLUE Color red Point to BLUE

30. Self-Reflection   Self Reflection here is defined as a cognitive activity of a robot to:   Reflect on past experiences   Control intentional actions   We have not yet finalized the exact specifications nor the roles of the Self Reflection Agent into a multi-agent architecture. Possible roles could include:   Reflective process may inspect the LTM database   Modify this database to generate new behaviors

31. Reflective Processes in ISAC A Proposal   We propose to take a multi-strategy approach: reflection for an impasse and reflection on behavior.   Three proposed agents, interacting with/within the Self Agent, implement ISAC’s reflective capabilities:   The Anomaly Detection Agent (ADA), during normal operation, gathers information about the distribution of inputs and outputs for each agent   The Mental Experimentation Agent (MEA) is invoked upon impasse. It conducts a search through the space of control parameters for those which are expected to resolve the impasse   The Central Executive Controller (CEC) Agent

32. Knowledge Base facts & rules facts & rules about the domain about reasoning processes Working Memory goals & intentions plans relevant information about past states/potential future states Reflective Processes Representations of the Current State Actions to be Taken Now Reflective Process

33. Concept for The Anomaly Detection Agent (ADA) Passively record the input/output distributions of each agent, conditioned on the current intention. This enables the identification of agents operating outside of their normal operating range. Self Agent ADA Observed Agent InputsOutputs Current Intention

34. Concept for The Mental Experimentation Agent (MEA)   When the Self Agent recognizes an impasse, it activates the MEA   The MEA searches for control signals which will resolve the impasse   Agents are run in “simulation mode” in order to determine the likely effects of control signal changes   The search is guided to suspected agents by the ADA MEA Suspected Agent InputsOutputs Control

35. Conclusion During the past decade, we have seen major advances in the integration of body and sensor. The next grand challenge will be in the integration of body and mind. An adaptive working memory, an executive controller, and neuron-base brain processing may be key to this integration.

37. ISAC: System Information Interaction competencies for a personal robot HardwareSoftware OBSERVATIONAL Presence of Person: Infrared (IR) Passive IR motion detector array Digital I/O Sound: Event LocalizationCondenser MicrophonesMatlab Speech: Detection/RecognitionHandheldMicrosoft Speech Recognition Engine 4.0 Vision: Face and GestureSony Color CCD CamerasVisual C++ routines, some with Intel Libraries DEMONSTRATIVE / RESPONSIVE Speech: SynthesisPC SpeakersAT&T Natural Voices Engine Motor behaviors: HeadDirected Perception PTU-46-70IMA wrapper for serial port Motor behaviors: ArmsPneumatic MusclesVisual C++ routines and control

38. Intelligent Machine Architecture   Hardware or Resource Agent   interfaces to sensor or actuator hardware   Behavior or Skill Agent   encapsulates basic behaviors or skills   Environment Agent   provides an abstraction for dealing with objects in the robot’s environment   Sequencer Agent   performs a sequence of operations   Multi-Type Agent   combines the functionality of at least two of the first four agent types Right Camera Left Camera Pan-Tilt Head Left Arm Left Hand Right Arm Right Hand Block Bean bag Legend Hardware Agent Environment Agent Spoon Cup IMA is a multi-agent based robot control architecture developed at Vanderbilt. Atomic agents (e.g. hardware, behavior,environment) are the building blocks of IMA.