1. Motor adaptation and the timescales of memory Reza Shadmehr Johns Hopkins School of Medicine Ali Ghazizadeh Maurice Smith Konrad Koerding Haiyin Chen Dave ZeeWilsaan Joiner Jun Izawa Tushar Rane

3. Duhamel et al. Science 255, 90-92 (1992) The brain predicts the sensory consequences of motor commands

4. musclesMotor commands force Body part State change Sensory system Proprioception Vision Audition Measured sensory consequences Forward model Predicted sensory consequences Integration Belief What we sense depends on what we predicted Wolpert et al. (1995)

5. 5 10 5 Eye Position (deg) Saccade adaptation: gain decrease McLaughlin 1967

6. 5 10 5 Eye Position (deg) McLaughlin 1967 Saccade adaptation: gain decrease

7. Kojima et al. (2004) J Neurosci 24:7531. _ Result 1: After changes in gain, monkeys exhibit recall despite behavioral evidence for washout. ++ Savings: when adaptation is followed by de-adaptation, motor system still exhibits recall Saccade gain = Target displacement Eye displacement

8. Result 2: Following changes in gain and a period of darkness, monkeys exhibit a “jump” in memory. + _ + Offline learning: with passage of time and without explicit training, the motor system still appears to learn Kojima et al. (2004) J Neurosci 24:7531.

9. Adaptation as concurrent learning in multiple systems: A fast learning system that forgets quickly A slow learning system that hardly forgets Smith et al. PLOS Biology, 2006 prediction Prediction error Learning

10. Savings: de-adaptation may not erase adaptation Task reversal period re-adaptation Trial number Smith et al. PLOS Biology, 2006

11. Offline learning: Passage of time has asymmetric affects on the fast and slow systems Smith et al. PLOS Biology, 2006 Task reversal period “dark” period re-adaptation Trial number Slow state Fast state -

12. Spontaneous recovery is also observed in reach adaptation Trial number Perturbation force Trial number 0 1 Performance relative to goal Task reversal period Error clamp period Smith et al. PLOS Biology 2006 Errors clamped to zero

13. 1. Perturbations that can affect the motor plant have multiple time scales. Some perturbations are fast: muscles recover from fatigue quickly. Some perturbations are slow: recovery from disease may be slow. 2.Faster perturbations are more variable (have more noise). 3.The error that we observe is due to a contribution from all possible perturbations. 4.The problem of learning is one of credit assignment: when I observe an error, what is the time-scale of this perturbation? The learner’s view about the cause of motor errors Koerding, Tenenbaum, Shadmehr, unpublished

14. A Slow change fast change The Bayesian learner’s interpretation of motor error Context perturbation State of the various potential causes of error Disease state Fatigue state

15. Savings: de-adaptation does not washout the adapted system Simulation Koerding, Tenenbaum, Shadmehr, unpublished Spontaneous recovery

16. Characteristics of long-term motor memory Data from Robinson et al. J Neurophysiol 2006 Bayesian Learner Koerding, Tenenbaum, Shadmehr, unpublished

17. Motor system is disturbed by processes that have various timescale (fatigue vs. disease). Credit assignment of error depends on uncertainty regarding what is the timescale of the disturbance. Prediction: When there are actions but the sensory consequences cannot be observed, states decay at various rates, but uncertainty grows. Increased uncertainty encourages learning. Bayesian learner Adapting with uncertainty

18. Adapting with uncertainty: two predictions Sensory deprivation  Faster subsequent rate of learning. Example: A subject that spends a bit of time in the dark will subsequently learn faster than a subject that spends that time with the lights on. Why: In the dark, uncertainty about state of the motor system increases. Longer inter-stimulus interval  Better retention. Example: A subject that trains on n trials with long ITI will show less forgetting than one that trains on the same n trials with short ITI. Why: events that take place spaced in time will be interpreted as having a long timescale.

19. Ali Ghazizadeh Maurice Smith Konrad Koerding Fast and slow adaptive processes arose because disturbances to the motor system have various timescales (fatigue vs. disease). When faced with error, the brain faces a credit assignment problem: what is the timescale of the disturbance? To solve this problem, the brain likely keeps a measure of uncertainty about the timescales. A prediction error causes changes in multiple adaptive systems. Some are highly responsive to error, but rapidly forget. Others are poorly responsive to error but have high retention. This explains savings and spontaneous recovery. Summary

20. 1.Internal models are supposed to help us control our movements in real-time. What are these fast and slow systems learning and how does that learning affect real-time control of movements? 2.Can we say anything about the neural structures that might be responsible for computing internal models? What are some of the holes in these ideas?

21. Body + environment State change Sensory system Proprioception Vision Audition Measured sensory consequences Forward model Predicted sensory consequences Integration Belief about state of body and world Goal selector Motor command generator Emo Todorov: Motor command generator as an optimal controller

22. Signal dependent motor noise Signal dependent sensory noise Actual state of the system (eye state, target state, etc.) What we can observe about the state of the system Motor command generator as a stochastic optimal controller Todorov (2005) Cost to minimize Feedback control policy Body + environment State change Sensory system Measured sensory consequences Forward model Predicted sensory consequences Integration Belief about state of body and world Goal specification Motor command generator Belief about state

23. eye velocity deg/sec 00.050.10.150.20.25 0 100 200 300 400 500 Time (sec) Body + environment State change Sensory system Measured sensory consequences Forward model Predicted sensory consequences Integration Belief about state of body and world Goal specification Motor command generator 51015304050 Saccade size The mathematical framework allows one to produce detailed trajectory of movements. In the target jump paradigm, error is a difference between predicted and actual sensory consequences of oculomotor commands. Therefore, the forward model must adapt. But if that adaptation is not precisely matched by the motor command generator, the result will be sub-optimal saccades. Prediction error

24. The direct and indirect output pathways from the superior colliculus (SC) Direct pathway SC  brainstem Indirect pathway SC  cerebellum  brainstem

25. Cross-axis saccade adaptation Equal rates of learning in the controller and the forward model saccades remain straight Learning in the forward model only saccades become curved Body + environment State change Forward model Predicted sensory consequences Belief about state of body and world Goal specification Motor command generator Increased training T1 T2 fixation

26. Cross-axis saccade adaptation: Experiment design (In complete darkness, with search coil lenses on the eyes) Chen, Joiner, Zee, Shadmehr (unpublished)

27. Characteristics of primary saccades during adaptation T1 T2 15 o 5o5o Chen, Joiner, Zee, Shadmehr (unpublished)

28. Curvature of primary saccades quantified through chord slopes Chen, Joiner, Zee, Shadmehr (unpublished)

29. The observation that saccades become curved, and therefore sub- optimal, is a reflection of a neural system that adaptively computes sensory consequences of motor commands, and corrects the motor commands as they are produced. The forward model (indirect pathway) appears to adapt much more quickly than the controller (direct pathway). Saccade curvature suggests that errors cause rapid adaptation in the forward model Body + environment State change Sensory system Forward model Predicted sensory consequences Integration Belief about state of body and world Goal specification Motor command generator Prediction error

30. Haiyin Chen Dave Zee In saccades and reaching, performance is guided by internal models that adapt at multiple timescales: A fast learning system that has poor retention. A slow learning system that hardly forgets. The observation that saccades become curved, and therefore sub-optimal, is a reflection of a neural system that adaptively computes sensory consequences of motor commands, and corrects the motor commands as they are produced. The forward model (indirect pathway) appears to adapt much more quickly than the controller (direct pathway). Summary: Wilsaan Joiner

31. 1.If learning of forward models (indirect pathway) is faster than the controller (direct pathway), the result is a sub-optimal system. Most of our movements appear optimal. What guides learning in the direct pathway so that we eventually become optimal? 2.If we learn as a Bayesian, we keep a measure of uncertainty about what we know. Does the uncertainty in the internal model affect our control policies (direct pathway)? What are some of the holes in these ideas?

32. Raymond Clarence Ewry (USA) Gold Medal, 1908 Olympics Cornelius Johnson (USA) Gold Medal, 1936 Olympics Dick Fosbury (USA) Gold Medal, 1968 Olympics Body + environment State change Sensory system Forward model Predicted sensory consequences Integration Belief about state of body and world Goal specification Motor command generator (control policy) Prediction error Learning in the direct pathway: finding a better control policy in the high jump task

33. N=6

34. The optimal control policy To maximize probability of arriving at target in time, I should minimize my motor commands near the end of the movement. Over compensate for the forces early, let the robot bring you back. Predicted trajectories under the optimal control policy Accuracy of model Izawa, Rane, Donchin, Shadmehr (unpublished)

35. Null field Izawa, Rane, Donchin, Shadmehr (unpublished)

36. In performing an action, the motor commands that we generate should depend on our confidence (uncertainty) in our models.

37. Stochastic optimal control with model uncertainty Jun Izawa Tushar Rane Traditional stochastic optimal control

38. Izawa, Rane, Donchin, Shadmehr (unpublished work) Stochastic optimal control with model uncertainty: Predictions

39. Izawa, Rane, Donchin, Shadmehr (unpublished work) People learn policies that depend on their model uncertainty: Overcompensate only if you are certain of the world N=6 High certainty Low certainty High certainty Low certainty

40. Jun Izawa Motor control is about solving two distinct problems: Learning a control policy (direct pathway). Learning a forward model (indirect pathway). Motor learning is at multiple timescales: A fast learning system that has poor retention. A slow learning system that hardly forgets. The forward model (indirect pathway) adapts much more quickly than the controller (direct pathway). Overview: Computational problem of motor control Maurice Smith Haiyin Chen

41. 1.In saccade adaptation, nothing happened to the body; it was the target that was behaving strangely. When there is error, how does the brain distinguish between changes in the body vs. changes in the world? This is a second credit assignment problem. 2.What is the error signal that guides learning of control policies? 3.Are the direct and indirect pathways computational pathways or neural pathways? What are some of the holes in these ideas?

42. thalamus Motor cortex Deep cerebellar nuclei Pons DBS: deep brain stimulation Inf. Olive Reversible disruption of cerebellar pathways in humans Cerebellar cortex Corticospinal tract Sherwin Hua

43. Deep Brain Stimulation 1.5 mm electrode is implanted in the thalamus and connected via subcutaneous wires to a stimulator. The subcutaneous stimulator and battery. Parameter settings can be adjusted via an external device. Fred Lenz

44. Stimulation of VL thalamus improves tremor but impairs adaptation Chen et al. Cerebral Cortex, 2006 Stimulation voltage Bipolar stim Unipolar stim Tremor during reaching

45. Movement onset Thoroughman & Shadmehr, J Neurosci, 1999 EMG patterns during reach adaptation

46. Neural correlates of motor learning in the VL thalamus

48. Adaptation level was low Behavioral performance

49. Sites attempted recording ………………..105 Sites successfully recorded units ………. 58 (55%) Units with more than 60 trials …………… 61 –Vim………………….35 –Vim-Vop border……12 –Voa/Vop…………… 14 Single units ……………………………….. 16 (26%) Movement related units …………………. 36 (59%) –Vim………………….21 –Vim-Vop border……5 –Voa/Vop……………10 Units showed direction selectivity ………. 18 (50%) –Vim………………….11 –Vim-Vop border……1 –Voa/Vop…………….6 Recording sites and neural responses

50. targetVmaxstophold/wait Adaptation induces change in firing pattern before movement onset

51. 1.The cerebellum appears to be a critical structure for motor adaptation. Is this the place where forward models are formed? 2.Speculation: cerebellar cortex may represent the “fast system”, with the cerebellar nuclei representing the “slow system”. Prediction: cerebellar patients may learn slowly, but they will also forget slowly. 3.Learning control policies depends on reward prediction errors. Is the basal ganglia the structure crucial for learning control policies? 4.Challenge ahead: To look for behavior and neural signatures of control policies and forward models in healthy individuals and patients with motor disorders. Conclusion and speculations

52. The neural basis of motor adaptation Cerebellar degeneration impaired adaptation of reaching Huntington’s disease (HD) patients showed no deficit in adaptation Smith and Shadmehr, J Neurophysiology 2005

53. early null3 Visual rotation adaptation