Journal club Marian Tsanov Reinforcement Learning

Predicting Future Reward – Temporal Difference Learning Actor-Critic Learning Sarsa learning Q - learning TD error: Where V is the current value function implemented by the critic.

Actor-critic methods are TD methods that have a separate memory structure to explicitly represent the policy independent of the value function. The policy structure is known as the actor, because it is used to select actions, and the estimated value function is known as the critic, because it criticizes the Actions made by the actor.

Why is Trial-To-Trial Variability needed for reinforcement learning ? In reinforcement learning there is no „supervisor“ which tells a neural circuit what to do with its input. Instead, a neural circuit has to try out different ways of processing the input, until it finds a successful (i.e., rewarded) one. This process is called exploration.

trial-by-trial learning rule known as the Rescorla-Wagner rule Here e is the learning rate, which can be interpreted in psychological terms as the associability of the stimulus with the reward.

Dayan and Abbott, 2000

Foster, Morris and Dayan, 2000

The prediction error δ plays an essential role in both the Rescorla-Wagner and temporal difference learning rules – biologically implemented by VTA.

Actor – critic model and reinforcement learning circuits Barnes et al. 2005

In a search for a critic – the striato-nigral problem Proposed critic – striosomes of dorsal striatum Proposed actor – matriomes of dorsal striatum

r DS SMC DP Dorsal striatum /coincidence detector/ hebbian w target SNc VP PPTN critic circuit LH actor circuit Prefrontal cortex Amigdala Hippocampus VS Lateral Hypothalamus /sensory-driven reward/ Ventral Striatum (Nucleus Accumbens) Ventral Pallidum Dorsal Pallidum /action/ pedunculopontine tegmental nucleus Dopamine Substantia nigra Sensory Motor Cortex t LCAC tt

Evidence for interregional learning systems interaction DeCoteau et al Schultz et al SNc Striosome

Q-learning

Need for multiple critics/actors Adaptive State Aggregation R R

Neuronal activity when shifting the cue modality Could current models explain this data? plain Q-learning? SARSA? a single actor-critic?

First Phase (tones) SWITCH BETWEEN ACTORS Actors used in the end (mean ± se) N = 3 (2.64 ± 0.05) N = 4 (2.96 ± 0.06) (mean ± se) N = 2 (0.76 ± 0.12) N = 3 (0.64 ± 0.09) N = 4 (0.62 ± 0.09) (mean ± se) N = 2 (1.06 ± 0.07) N = 3 (1.06 ± 0.1) N = 4 (1.05 ± 0.09) Second Phase (textures)

4 ACTORS

IF THE CTX/STRIATUM CAN TRACK THE PERFORMANCE OF THE ACTORS, AFTER THE TRANSFER THERE MIGHT BE AN INITIAL BIAS FOR THE BEFORE USED ACTORS (HERE WE IMPLEMENTED RANDOMLY). THEN THE PERFORMANCE SHOULD BE CLOSER TO THE RESULTS WITH N=2, EVEN IF MORE ACTORS ARE AVAILABLE

Motivation: How is the knowledge transferred to the second cue? What is a “State” in reinforcement learning? A place where you are free to choose your next action in to other states The representation of environment should change State aggregation The Knowledge Transfer Problem State aggregation and sequence learning

DS SMC DP (action) coincidence detector DS SMC DP (action) coincidence detector DS SMC DP (action) coincidence detector hebbian w target STDP w target Sequence learning Theta dependent STDP plasticity DeCoteau et al. 2007

A B A B Unsupervised Theta Dependent STDP

SMCLC/AC before learning after learning (audio cue) actors aggregation in dorsal striatum

Algorithm Adaptive combination of states Knowledge Transfer: Keep the learned states Number of activated states.

State aggregation The Knowledge Tranfer Effect Trail Number – Average Runing Step No Aggregation State Aggregation

State Number Reduction

Conclusion: State Aggregation - Link to Learned Actor Multiple Motor Layer and higher level decision making State Aggregation: change to abstract states of Motion Selector Sequential Learning Learning pattern generator