Tutorial Session: The Bayesian brain, surprise and free-energy Value-learning and perceptual learning have been an important focus over the past decade, attracting the concerted attention of experimental psychologists, neurobiologists and the machine learning community. Despite some formal connections; e.g., the role of prediction error in optimizing some function of sensory states, both fields have developed their own rhetoric and postulates. In work, we show that perception is, literally, an integral part of value learning; in the sense that it is necessary to integrate out dependencies on the inferred causes of sensory information. This enables the value of sensory trajectories to be optimized through action. Furthermore, we show that acting to optimize value and perception are two aspects of exactly the same principle; namely the minimization of a quantity (free energy) that bounds the probability of sensations, given a particular agent or phenotype. This principle can be derived, in a straightforward way, from the very existence of biological agents, by considering the probabilistic behaviour of an ensemble of agents belonging to the same class. Put simply, we sample the world to maximise the evidence for our existence. Predictive Coding: Whatever Next? University of Edinburgh, January 19th, 2010

“Objects are always imagined as being present in the field of vision as would have to be there in order to produce the same impression on the nervous mechanism” - Hermann Ludwig Ferdinand von Helmholtz Thomas Bayes Geoffrey Hinton Richard Feynman From the Helmholtz machine and the Bayesian Brain to Action and self-organization Hermann Haken

Overview Ensemble dynamics Entropy and equilibria Free-energy and surprise The free-energy principle Action and perception Generative models Perception Birdsong and categorization Simulated lesions Action Active inference Reaching Policies Control and attractors The mountain-car problem

Particle density contours showing Kelvin-Helmholtz instability, forming beautiful breaking waves. In the self- sustained state of Kelvin-Helmholtz turbulence the particles are transported away from the mid-plane at the same rate as they fall, but the particle density is nevertheless very clumpy because of a clumping instability that is caused by the dependence of the particle velocity on the local solids-to-gas ratio (Johansen, Henning, & Klahr 2006) temperature pH falling transport Self-organization that minimises an ensemble density to ensure a limited repertoire of states are occupied (i.e., ensuring states have a random attracting set).

How can an active agent minimise its equilibrium entropy? This entropy is bounded by the entropy of sensory signals (under simplifying assumptions) Crucially, because the density on sensory signals is at equilibrium, it can be interpreted as the proportion of time each agent entertains them (the sojourn time). This ergodic argument means that entropy is the path integral of surprise experienced by a particular agent: This means agents minimise surprise at all times. But there is one small problem… Agents cannot access surprise; however, they can evaluate a free-energy bound on surprise, which is induced with a recognition density q :

Overview Ensemble dynamics Entropy and equilibria Free-energy and surprise The free-energy principle Action and perception Generative models Perception Birdsong Simulated lesions Action Active inference Reaching Polices Control and attractors The mountain-car problem

Action External states in the world Internal states of the agent (m) Sensations The free-energy principle Action to minimise a bound on surprisePerception to optimise the bound

The free-energy rests on expected Gibb’s energy and can be evaluated, given a generative model comprising a likelihood and prior: So what models might the brain use? The generative model

Processing hierarchy Backward (nonlinear) Forward (linear) lateral Ensemble dynamics Entropy and equilibria Free-energy and surprise The free-energy principle Action and perception Generative models Perception Birdsong Simulated lesions Action Active inference Reaching Polices Control and attractors The mountain-car problem

Hierarchical (deep) dynamic models

Structural priors Dynamical priors Likelihood and empirical priors Hierarchal form Gibb’s energy: a simple function of prediction error Prediction errors

Synaptic gain Synaptic activity Synaptic efficacy Activity-dependent plasticity Functional specialization Attentional gain Enabling of plasticity Attention and salience Perception and inferenceLearning and memory The recognition density and its sufficient statistics Mean-field approximation: Laplace approximation:

Backward predictions Forward prediction error Perception and message-passing Synaptic plasticitySynaptic gain David Mumford

The free-energy principle and infomax The infomax principle requires the mutual information between sensory data and their conditional representation to be maximal, under prior constraints on the representations If the recognition density is a point mass In short, the infomax principle is a special case of the free-energy principle that obtains when we discount uncertainty and represent sensory data with point estimates of their causes. Alternatively, the free-energy is a generalization of the infomax principle that covers probability densities on the unknown causes of data. Horace Barlow

Overview Ensemble dynamics Entropy and equilibria Free-energy and surprise The free-energy principle Action and perception Generative models Perception Birdsong and categorization Simulated lesions Action Active inference Reaching Polices Control and attractors The mountain-car problem

Synthetic song-birds SyrinxVocal centre Time (sec) Frequency Sonogram

prediction and error time hidden states time Backward predictions Forward prediction error Causal states time (bins) Recognition and message passing stimulus time (seconds)

Perceptual categorization Frequency (Hz) Song A time (seconds) Song B Song C

Generative models of birdsong: sequences of sequences Syrinx Neuronal hierarchy Time (sec) Frequency (KHz) sonogram Kiebel et al

Frequency (Hz) percept Frequency (Hz) no structural priors time (seconds) Frequency (Hz) no dynamical priors LFP (micro-volts) LFP LFP (micro-volts) LFP peristimulus time (ms) LFP (micro-volts) LFP Simulated lesion studies: a model for false inference in psychopathology?

Ensemble dynamics Entropy and equilibria Free-energy and surprise The free-energy principle Action and perception Generative models Perception Birdsong Simulated lesions Action Active inference Reaching Polices Control and attractors The mountain-car problem

prediction From reflexes to action action dorsal root ventral horn True dynamics Generative model

From reflexes to action Jointed arm Movement trajectory Descending sensory prediction error visual input proprioceptive input

Overview Ensemble dynamics Entropy and equilibria Free-energy and surprise The free-energy principle Action and perception Generative models Perception Birdsong Simulated lesions Action Active inference Reaching Polices Control and attractors The mountain-car problem

Cost-functions, priors and policies with attractors At equilibrium we have: This means maxima of the equilibrium density must have negative divergence. We can exploit this to ensure maxima lie in A, where cost increases dissipation Adriaan Fokker Max Planck

True equations of motion position height The mountain car problem positionhappiness The cost-function Policy (expected equations of motion) The environment

With cost (i.e., exploratory dynamics) Exploring & exploiting the environment a

Using just the free-energy principle and a simple gradient ascent scheme, we have solved a benchmark problem in optimal control theory using a handful of learning trials. Note that we use reinforcement learning or dynamic programming. Adaptive policies and trajectories

Infomax and the redundancy minimisation principle Maximisation of the mutual information between sensations and representations Probabilistic neuronal coding Encoding a recognition density in terms of conditional expectations and uncertainty The Bayesian brain hypothesis Minimising the difference between a recognition density and the conditional density on sensory causes Predictive coding and hierarchical inference Minimisation of prediction error with recurrent message passing Perceptual leaning and memory Optimisation of synaptic efficacy to represent causal structure in the sensorium Associative plasticity Optimisation of synaptic efficacy Optimal control and value learning Optimisation of a free-energy bound on surprise or value Model selection and evolution Optimising the agent’s model and priors through neurodevelopment and natural selection The free-energy principle Minimisation of the free-energy of sensations and the representation of their causes Attention and biased competition Optimisation of synaptic gain representing the precision (salience) of predictions Exploration and exploitation Policies as prior expectations on motions Computational motor control Minimisation of sensory prediction error

Perception and Action: The optimisation of neuronal and neuromuscular activity to suppress prediction errors (or free- energy) based on generative models of sensory data. Learning and attention: The optimisation of synaptic gain and efficacy over seconds to hours, to encode the precisions of prediction errors and causal structure in the sensorium. This entails suppression of free-energy over time. Neurodevelopment: Model optimisation through activity- dependent pruning and maintenance of neuronal connections that are specified epigenetically Evolution: Optimisation of the average free-energy (free-fitness) over time and individuals of a given class (e.g., conspecifics) by selective pressure on the epigenetic specification of their generative models. time-scale process

Thank you And thanks to collaborators: Jean Daunizeau Lee Harrison Stefan Kiebel James Kilner Klaas Stephan And colleagues: Peter Dayan Jörn Diedrichsen Paul Verschure Florentin Wörgötter