Learning, Volatility and the ACC Tim Behrens FMRIB + Psychology, University of Oxford FIL - UCL.

B Trials Into Past Reward History Weight (β) CON i-1i-2i-3i-4i-5i-6i-7i-8 Kennerley, et al., Nature Neuroscience, 2006

ACCs B Trials Into Past Reward History Weight (β) CON i-1i-2i-3i-4i-5i-6i-7i-8 Kennerley et al. Nature Neuroscience, 2006

Monkeys will sacrifice food opportunities to look at other monkeys ACC G Rudebeck,et al. Science 2005

Interest in other individuals is reduced after ACC gyrus lesion ACC G Rudebeck,et al. Science 2005

Anatomy - Differences in connections between ACCs and ACCg. Connections unique to the sulcus are mainly with motor regions: Primary motor cortex Premotor cortex Parietal motor areas Spinal Cord ACCs has information about our own actions

Anatomy - Differences in connections between ACCs and ACCg. Connections unique to the gyrus are mainly with regions that process emotional and biological stimuli: Periacqueductal grey hypothalamus STS/STG Insula/Temporal pole connections are stronger to the gyrus ACCg has access to information about other agents.

Anatomy - shared connections between ACCs and ACCg. Some shared connections Orbitofrontal cortex Amydala Ventral striatum ACCg and ACCs are strongly interconnected Both regions have access to and influence over reward and value processing.

ACC Sulcus and learning about your actions.

ACCs B Trials Into Past Reward History Weight (β) CON i-1i-2i-3i-4i-5i-6i-7i-8 Kennerley et al. Nature Neuroscience, 2006

Kennerly et al. Nat Neurosci 2006Sugrue et al. Science 2005 Trials Into Past Reward History Weight (β) CON i-1i-2i-3i-4i-5i-6i-7i-8 What determines the integration length?

Kennerly et al. Nat Neurosci 2006Sugrue et al. Science 2005 VOLATILE Reward probabilities change approximately every 25 trials STABLE Reward probabilities change only after hundreds of trials Trials Into Past Reward History Weight (β) CON i-1i-2i-3i-4i-5i-6i-7i-8

Reinforcement learning We need to continually re-appraise the value of an action based each new experience.  prediction (V t ) outcome  x  new prediction (V t+1 )

Updating beliefs on the basis of new information 14 V t+1 =V t +(  x  The learning rate is the weight given to the current information The prediction error is the information available from this event

The learning rate and the value of information. V t+1 =V t +(  x  The learning rate should represent the value of the current information for guiding future beliefs.

 =0.01  =0.4  =0.1 Relationship with integration length

stable Behrens et al., Nature Neuroscience, 2007

Behrens, Woolrich, Walton, Rushworth, Nature Neuroscience, 2007 V t+1 =V t +  x 

changes in reward estimates occur throughout the task… Behrens, Woolrich, Walton, Rushworth, Nature Neuroscience, 2007 …as do change in volatility estimates

Decide Monitor x Volatility Behrens et al., Nature Neuroscience, 2007

ACC effect size predicts learning rate across subjects Behrens, Woolrich, Walton &Rushworth Nat Neurosci 2007

ACC Gyrus and learning about your social partners.

Interest in other individuals is reduced after ACC gyrus lesion ACC G Rudebeck et al. Science 2005

Rudebeck et al., Science, 2006

25 Learning about other agents Behrens, Hunt, Woolrich, Rushworth Nature 2008

Sources of information Probability that confederate advice is good Probability that correct colour is blue Value of action information Value of social information Behrens, Hunt, Woolrich, Rushworth Nature 2008

Social information is integrated over time - behaviour

Reward Prediction Error Reward -Expectation V t+1 =V t +(  x  Outcome Time Effect size Behrens, Hunt, Woolrich, Rushworth Nature 2008

Prediction error on a social partner. Lie event -Lie prediction V t+1 =V t +(  x  Outcome Time Effect size Behrens, Hunt, Woolrich, Rushworth Nature 2008

The value of information and the ACC 30 Value of reward information Value of social information V t+1 =V t +(  x 

Combining Information to drive behaviour V t+1 =V t +(  x 

32 Conclusions ACC codes a learning signal when information is observed. This signal predicts the speed of learning. Learning from our own and others’ actions are processed in parallel in ACCs and ACCg. The outputs of these parallel learning processes are combined in the reward system.

33 Acknowledgments Matthew Rushworth Mark Woolrich Laurence Hunt Mark Walton 33