Outline 1) Goal-Directed Feature Learning (Weber & Triesch, IJCNN 2009)‏ Task 4.1 Visual processing based on feature abstraction 2) Emergence of Disparity Tuning (Franz & Triesch, ICDL 2007)‏ Task 4.3 Learning of attention and vergence control 3) From Exploration to Planning (Weber & Triesch, ICANN 2008)‏ Task 6.4 Learning hierarchical world models for planning

Outline 1) Goal-Directed Feature Learning (Weber & Triesch, IJCNN 2009)‏ Task 4.1 Visual processing based on feature abstraction 2) Emergence of Disparity Tuning (Franz & Triesch, ICDL 2007)‏ Task 4.3 Learning of attention and vergence control 3) From Exploration to Planning (Weber & Triesch, ICANN 2008)‏ Task 6.4 Learning hierarchical world models for planning

reinforcement learning input s action a weights

actor go right? go left? simple input go right! complex input reinforcement learning input (state space)‏

sensory input reward action complex input scenario: bars controlled by actions, ‘up’, ‘down’, ‘left’, ‘right’; reward given if horizontal bar at specific position

need another layer(s) to pre-process complex data P(a=1) = softmax(Q s)‏a action s state I input Q weight matrix W weight matrix position of relevant bar encodes v = a Q s feature detector s = softmax(W I)‏ E = (0.9 v(s’,a’) - v(s,a)) 2 = δ 2 d Q ≈ dE/dQ = δ a s d W ≈ dE/dW = δ Q s I + ε minimize error: learning rules:

SARSA with WTA input layer

memory extension model uses previous state and action to estimate current state

RL action weights feature weights data learning the ‘short bars’ data reward action

short bars in 12x12 average # of steps to goal: 11

RL action weights feature weights input reward 2 actions (not shown)‏ data learning ‘long bars’ data

WTA non-negative weights SoftMax non-negative weights SoftMax no weight constraints

models’ background: - gradient descent methods generalize RL to several layers Sutton&Barto RL book (1998); Tesauro (1992;1995) - reward-modulated Hebb Triesch, Neur Comp 19, (2007), Roelfsema & Ooyen, Neur Comp 17, (2005); Franz & Triesch, ICDL (2007)‏ - reward-modulated activity leads to input selection Nakahara, Neur Comp 14, (2002) - reward-modulated STDP Izhikevich, Cereb Cortex 17, (2007), Florian, Neur Comp 19/6, (2007); Farries & Fairhall, Neurophysiol 98, (2007);... - RL models learn partitioning of input space e.g. McCallum, PhD Thesis, Rochester, NY, USA (1996)‏

unsupervised learning in cortex reinforcement learning in basal ganglia state space actor Doya, 1999

Discussion - may help reinforcement learning work with real-world data... real visual processing!

Outline 1) Goal-Directed Feature Learning (Weber & Triesch, IJCNN 2009)‏ Task 4.1 Visual processing based on feature abstraction 2) Emergence of Disparity Tuning (Franz & Triesch, ICDL 2007)‏ Task 4.3 Learning of attention and vergence control 3) From Exploration to Planning (Weber & Triesch, ICANN 2008)‏ Task 6.4 Learning hierarchical world models for planning

Representation of depth How to learn disparity tuned neurons in V1?

Reinforcement learning in a neural network after vergence: input at a new disparity if disparity is zero  reward

Attention-Gated Reinforcement Learning Hebbian-like weight learning: (Roelfsema, van Ooyen, 2005) ‏

Six types of tuning curves (Poggio, Gonzalez, Krause, 1988)‏ Measured disparity tuning curves

All six types of tuning curves emerge in the hidden layer! Development of disparity tuning

Discussion - requires application... use 2D images from 3D space... open question as to the implementation of the reward... learning of attention?

Outline 1) Goal-Directed Feature Learning (Weber & Triesch, IJCNN 2009)‏ Task 4.1 Visual processing based on feature abstraction 2) Emergence of Disparity Tuning (Franz & Triesch, ICDL 2007)‏ Task 4.3 Learning of attention and vergence control 3) From Exploration to Planning (Weber & Triesch, ICANN 2008)‏ Task 6.4 Learning hierarchical world models for planning

Reinforcement Learning leads to a fixed reactive system that always strives for the same goal value actor units task: in exploration phase, learn a general model to allow the agent to plan a route to any goal

Learning actor state space randomly move around the state space learn world models: ● associative model ● inverse model ● forward model

Learning: Associative Model weights to associate neighbouring states use these to find any possible routes between agent and goal

Learning: Inverse Model weights to “postdict” action given state pair use these to identify the action that leads to a desired state Sigma-Pi neuron model

Learning: Forward Model weights to predict state given state-action pair use these to predict the next state given the chosen action

Planning

goal actor units agent

Planning

Discussion - requires embedding... learn state space from sensor input... only random exploration implemented - tong... hand-designed planning phases... hierarchical models?