A possible representation of reward in the learning of saccades Cornelius Weber and Jochen Triesch Frankfurt Institute for Advanced Studies Johann Wolfgang Goethe Universität Frankfurt am Main, Germany Presentation at the EpiRob 2006, September, Paris

Contents saccade learning: supervised or reward-driven? separate control of horizontal & vertical saccades

development emergence supervised learning unsupervised & reinforcement learning explorationimitation learn from environment genetic description or instructor Learning Signals specific generic

Saccades in the adult are quite inaccurate undershoot overshoot Data taken from: A. Lewis, R. Garcia and L. Zhaoping (2003) The distribution of visual objects on the retina: connecting eye movements and cone distributions. Journal of Vision, 3,

performance vectorial error Saccade Learning Signal Specific? Generic?

SC: superior colliculus LLBN: long-lead burst neuron EBN: excitatory burst neuron VI: abducens nucleus NPH/MVN: cells in nucleus prepositus hypoglossi or medial vestibular nucleus IBN: inhibitory burst neuron OPN: omnipause neuron La: latch neurons Tr: trigger signal Figure source: D. Sparks (2002) The brainstem control of saccadic eye movements. Nat Rev Neurosci, 3: Saccade control downstream of the SC

Figure source: M.A. Frens and A.J. Van Opstal (1997). Monkey superior colliculus activity during short-term saccadic adaptation. Brain Research Bulletin 43(5): Site of plasticity adaptation is downstream exact error signal unknown pre-saccadic activation patch looks like adaptation fields SC

Figure source: E.A. Vessel (2004) Behavioral and Neural Investigation of Perceptual Effect.

Sensory neuron responses are modulated by reward in V1 of adult rat M.G.Shuler, M.F.Bear (2006) Reward Timing in the Primary Visual Cortex. Science 311, in the inferior colliculus (IC) of adult monkey R.R.Metzger, N.T.Greene, K.K.Porter, J.M. Groh (2006) Effects of Reward and Behavioral Context on Neural Activity in the Primate Inferior Colliculus. J Neurosci 26(28),

Foveal stimuli are magnified on the SC retinaSC

Vectorial error vs. Reward signal

Figure source: F. Robinson, C. Noto, S. Bevans (2003) Effect on visual error size on saccade adaptation in monkey. J Neurophysiol, 90: Constant-sized error allows no feedback of learning progress (Robinson, 2003) target shifts 1 o backward relative to saccade endpoint

Figure source: F. Robinson, C. Noto, S. Bevans (2003) Effect on visual error size on saccade adaptation in monkey. J Neurophysiol, 90: Constant-sized error allows no feedback of learning progress

Figure source: F. Robinson, C. Noto, S. Bevans (2003) Effect on visual error size on saccade adaptation in monkey. J Neurophysiol, 90: Constant-sized error allows no feedback of learning progress error size gain change

Oblique saccades are a “sum” of horizontal and vertical components Figure source: lecture "Modelling of sensorimotor systems" by S. Glasauer, Ludwig-Maximilians-Universität München

NIC: interstitial nucleus of Cajal riMLF: rostral interstitial nucleus of the medial longi- tudinal fasciculus MRF: midbrain reticular formation PPRF: paramedian pontine reticular formation NPH: nucleus prepositus hypoglossi Med. RF: medullary reticular formation III: oculomotor nucleus IV: trochlear nucleus VI: abducens nucleus Figure source: D. Sparks (2002) The brainstem control of saccadic eye movements. Nat Rev Neurosci, 3: Horizontal and vertical control circuits are separate

Visual field topography in the SC

Model architecture vectorial error for horizontal saccades success-based learning for vertical saccades Model assumption

Algorithm for vectorial error based learning (horizontal) a SC mhmh D = a i - a c ∆w h ≈ D a SC m h

Algorithm for performance reward based learning (vertical) ∆w v ≈ T a SC m v T = a post - a pre mvmv a SC

Learnt weights and model errors

Conclusion Vectorial error for horizontal saccades simple implementation specific to brain sub-system redoing Robinson (2003) experiment for vertical saccades could tell * * Performance-based reward possible for vertical saccades more generic Two possible implementations of feedback for learning:

Figure source: “Attention and Eye Movement in young Infants: Neural Control and Development” by J.E. Richards and S.K. Hunter; and Eye Movement in young Infants.ppt

Figure source: J.J. Hopp and A.F. Fuchs (2004) The characteristics and neuronal substrate of saccadic eye movement plasticity. Progress in Neurobiology, 72: Sub-cortical and cortical visual systems secondary visual system, mature at birth primary visual system, functional after 2 months of age cerebellum serves both