Presentation is loading. Please wait.

Presentation is loading. Please wait.

CogRIC Workshop Adaptive Working Memory: From Computational Neuroscience Model To Robot Control Module David C. Noelle Assistant Professor of Computer.

Similar presentations


Presentation on theme: "CogRIC Workshop Adaptive Working Memory: From Computational Neuroscience Model To Robot Control Module David C. Noelle Assistant Professor of Computer."— Presentation transcript:

1 CogRIC Workshop Adaptive Working Memory: From Computational Neuroscience Model To Robot Control Module David C. Noelle Assistant Professor of Computer Science Assistant Professor of Psychology Vanderbilt University david.noelle@vanderbilt.edu August 17, 2006

2 CogRIC Workshop Adaptive Working Memory: From Computational Neuroscience Model To Robot Control Module David C. Noelle Assistant Professor of Computer Science Assistant Professor of Cognitive Science University of California, Merced david.noelle@vanderbilt.edu August 17, 2006

3 CogRIC Workshop Adaptive Working Memory: From Computational Neuroscience Model To Robot Control Module David C. Noelle Assistant Professor of Computer Science Assistant Professor of Cognitive Science University of California, Merced david.noelle@vanderbilt.edu August 17, 2006

4 David C. Noelle, Ph.D. david.noelle@vanderbilt.edu people.vanderbilt.edu/~david.noelle Adaptive Working Memory Project Funded by NSF ITR program (EIA-0325641) www.cecs.missouri.edu/~skubic/WM/ Joshua Phillips Kaz Kawamura Mitch Wilkes Marge Skubic Jim Keller Julia High Will Dodd Palis Ratanaswasd Mert Tugcu Sam Blisard Bob Luke Stephen Gordon

5 David C. Noelle, Ph.D. david.noelle@vanderbilt.edu people.vanderbilt.edu/~david.noelle Julia High Will Dodd Palis Ratanaswasd Mert Tugcu Sam Blisard Bob Luke Stephen Gordon Adaptive Working Memory Project Funded by NSF ITR program (EIA-0325641) www.cecs.missouri.edu/~skubic/WM/ Joshua Phillips Kaz Kawamura Mitch Wilkes Marge Skubic Jim Keller

6 David C. Noelle, Ph.D. david.noelle@vanderbilt.edu people.vanderbilt.edu/~david.noelle Working Memory

7 David C. Noelle, Ph.D. david.noelle@vanderbilt.edu people.vanderbilt.edu/~david.noelle Working Memory Working memory systems are those that actively maintain transient information that is critical for successful decision-making in the current context. A working memory system can be viewed as a relatively small cache of task relevant information that is strategically positioned to efficiently influence behavior.

8 David C. Noelle, Ph.D. david.noelle@vanderbilt.edu people.vanderbilt.edu/~david.noelle Working Memory In The Brain A number of brain regions are implicated as important components of the human working memory system. One important region is dorsolateral portions of prefrontal cortex. Working memory is exhibited in delay period activity. Cells have been found which encode for locations, visual features, and association rules.

9 David C. Noelle, Ph.D. david.noelle@vanderbilt.edu people.vanderbilt.edu/~david.noelle Modeling Collaborators Todd Braver Jonathan Cohen Randy O'Reilly

10 David C. Noelle, Ph.D. david.noelle@vanderbilt.edu people.vanderbilt.edu/~david.noelle Active Maintenance How are high neural firing rates sustained over a delay? Mutual excitation of neurons. Dense recurrent connections in prefrontal cortex. Stripe sets. Attractor network computational models.

11 David C. Noelle, Ph.D. david.noelle@vanderbilt.edu people.vanderbilt.edu/~david.noelle Controlling PFC Updating How does PFC know when to actively maintain its current working memory contents? How does it know when to abandon a given working memory chunk in favor of a new one? The dynamics of recurrent attractor networks are insufficient to meet the simultaneous constraints of (1) active maintenance in the face of distraction and (2) rapid updating when needed. A dynamic gating mechanism is needed.

12 David C. Noelle, Ph.D. david.noelle@vanderbilt.edu people.vanderbilt.edu/~david.noelle Mesolimbic Dopamine (DA) System

13 David C. Noelle, Ph.D. david.noelle@vanderbilt.edu people.vanderbilt.edu/~david.noelle Dopamine (DA) Cells (Schultz, Apicella, and Ljungberg, 1993)

14 David C. Noelle, Ph.D. david.noelle@vanderbilt.edu people.vanderbilt.edu/~david.noelle Temporal Difference Learning DA neurons seem to encode for change in expected reward. This is equivalent to the key variable, called temporal difference error, in a powerful reinforcement learning algorithm called temporal difference (TD) learning. The brain may learn to produce rewarding motor sequences using a neural implementation of TD learning (Montague, Dayan, and Sejnowski, 1996). There are extensive DA projections to PFC. If TD learning is used to learn when to produce a given overt action, perhaps it can be used to learn when to produce a covert action – like updating working memory (Braver and Cohen, 2000).

15 David C. Noelle, Ph.D. david.noelle@vanderbilt.edu people.vanderbilt.edu/~david.noelle Computational Cognitive Neuroscience Models Healthy performance on frontal tasks. Prolonged frontal developmental period. Monkey lesion data. Human frontal damage patient performance. Autistic peformance. (Rougier, Noelle, Braver, Cohen, and O'Reilly, 2005)

16 David C. Noelle, Ph.D. david.noelle@vanderbilt.edu people.vanderbilt.edu/~david.noelle Robotic Working Memory The highly limited capacity of working memory, along with its tight coupling with deliberation mechanisms, might alleviate the need for costly memory searches. Information needed to fluently perform the current task is temporarily kept handy in the working memory store. Could robot control systems benefit from the inclusion of a working memory system? Can computational neuroscience models of the working memory mechanisms of the human brain shed light on the design of a robotic working memory system?

17 David C. Noelle, Ph.D. david.noelle@vanderbilt.edu people.vanderbilt.edu/~david.noelle Potential Uses Focus attention on the most relevant features of the current task. Guide perceptual processes by limited the perceptual search space. Provide a focused short-term memory to prevent the robot from being confused by occlusions. Provide robust operation in the presence of distracting irrelevant events.

18 David C. Noelle, Ph.D. david.noelle@vanderbilt.edu people.vanderbilt.edu/~david.noelle Adaptive Working Memory Hand Coding – For relatively routine and well understood tasks, designers may hand code procedures for the identification of useful chunks. Learning – If the robot is expected to flexibly respond in novel task situations, or even acquire new tasks, it would be beneficial to have a means to learn when to store a particular chunk in working memory. How does the working memory system know when a given chunk of information should be actively maintained in working memory? The central focus of this project is on assessing the utility of adaptive working memory mechanisms for robot control.

19 David C. Noelle, Ph.D. david.noelle@vanderbilt.edu people.vanderbilt.edu/~david.noelle The Working Memory Toolkit Memory traces, or chunks, are pointers to arbitrary C++ data structures. The adaptive working memory toolkit (WMtk) requires the user to specify: the capacity of the working memory a function which extracts features from chunks a function which provides relevant features of the current system state a function which provides instantaneous external reward information The toolkit provides a function for examining the contents of working memory, returning chunk pointers.

20 David C. Noelle, Ph.D. david.noelle@vanderbilt.edu people.vanderbilt.edu/~david.noelle On Each Time Step... The robot control system making use of the WMtk suggests candidate chunks for retention by the working memory. A component of the TD learning algorithm, called the adaptive critic, is used to estimate the expected future reward of retaining various combinations of chunks. The collection of chunks with the highest expected future reward value are remembered (with high probability). The amount of instantaneous external reward received on this time step, along with the estimates of the adaptive critic on this time step and on the previous time step, are used to compute the TD error – the change in expected future reward. This error signal is used to train the adaptive critic.

21 David C. Noelle, Ph.D. david.noelle@vanderbilt.edu people.vanderbilt.edu/~david.noelle Delayed Saccade Task

22 David C. Noelle, Ph.D. david.noelle@vanderbilt.edu people.vanderbilt.edu/~david.noelle Robotic Delayed Saccade Task

23 David C. Noelle, Ph.D. david.noelle@vanderbilt.edu people.vanderbilt.edu/~david.noelle Task Structure Three kinds of goal working memory chunks: Stay fixated on the object that you are looking at. Look at the last location of the crosshair. Look at the last location of the target. The robot control system obeys any goal chunks in working memory (resolving look at conflicts at random). If no chunks are being actively maintained, the system looks at a randomly selected object or, when there are no displayed objects, at a random location on the screen. Note that remembering all chunks will often lead to failure.

24 David C. Noelle, Ph.D. david.noelle@vanderbilt.edu people.vanderbilt.edu/~david.noelle Results The robot successfully learns the task...... after about 4000 trials.

25 David C. Noelle, Ph.D. david.noelle@vanderbilt.edu people.vanderbilt.edu/~david.noelle Revisiting The Neuroscience Dopamine cells sometimes fire in a way that does not reflect a change in expected future reward. Specifically, they often fire in novel situations. If the dopamine signal is seen as TD error, this suggests that brain treats novel situations as if they were more predictive of reward than is warranted by experience. This has been implemented in the WMtk through the incorporation of an optimistic critic – the TD algorithm is initialized to predict high future reward for novel combinations of chunks. With this modification...

26 David C. Noelle, Ph.D. david.noelle@vanderbilt.edu people.vanderbilt.edu/~david.noelle Optimistic Critic Results The robot successfully learns the task...... after about 300 trials. An improvement by more than a factor of ten!

27 David C. Noelle, Ph.D. david.noelle@vanderbilt.edu people.vanderbilt.edu/~david.noelle Other Preliminary Successes Mitch Wilkes and his students have used the WMtk to allow a physical mobile robot to...... learn which percepts to approach in order to produce the largest amount of forward motion down a hallway.... learn which percepts reliably identify a specific target location, where reward is received. Marge Skubic and her students have used the WMtk to allow a simulated robot to learn which motor program to retain as a goal chunk, given the current sensory state, so as to solve a water maze problem.

28 David C. Noelle, Ph.D. david.noelle@vanderbilt.edu people.vanderbilt.edu/~david.noelle Summary Computational cognitive neuroscience models of the interactions between the prefrontal cortex and the midbrain dopamine system have been successful at accounting for a variety of working memory phenomena. The basic structure of these models, involving the use of a reinforcement learning algorithm to learn, from experience, what should be retained in working memory and what can be safely forgotten, has been abstracted into an open source software library called the Working Memory Toolkit. By attending to nuances of biology, the adaptive learning capabilities of the toolkit have been greatly improved.

29 David C. Noelle, Ph.D. david.noelle@vanderbilt.edu people.vanderbilt.edu/~david.noelle Questions?

30 David C. Noelle, Ph.D. david.noelle@vanderbilt.edu people.vanderbilt.edu/~david.noelle The End

31 David C. Noelle, Ph.D. david.noelle@vanderbilt.edu people.vanderbilt.edu/~david.noelle Extra Slides

32 David C. Noelle, Ph.D. david.noelle@vanderbilt.edu people.vanderbilt.edu/~david.noelle PFC Stripes & Thalamic Loops

33 David C. Noelle, Ph.D. david.noelle@vanderbilt.edu people.vanderbilt.edu/~david.noelle A More Complex Network


Download ppt "CogRIC Workshop Adaptive Working Memory: From Computational Neuroscience Model To Robot Control Module David C. Noelle Assistant Professor of Computer."

Similar presentations


Ads by Google