Motor Control. Beyond babbling Three problems with motor babbling: –Random exploration is slow –Error-based learning algorithms are faster but error signals.

Slides:



Advertisements
Similar presentations
Bayesian Learning & Estimation Theory
Advertisements

Visual feedback in the control of reaching movements David Knill and Jeff Saunders.
Introductory Control Theory I400/B659: Intelligent robotics Kris Hauser.
Computer vision: models, learning and inference Chapter 8 Regression.
CSCI 347 / CS 4206: Data Mining Module 07: Implementations Topic 03: Linear Models.
Institute for Theoretical Physics and Mathematics Tehran January, 2006 Value based decision making: behavior and theory.
1. Algorithms for Inverse Reinforcement Learning 2
The loss function, the normal equation,
Classification and Prediction: Regression Via Gradient Descent Optimization Bamshad Mobasher DePaul University.
Introduction to Control: How Its Done In Robotics R. Lindeke, Ph. D. ME 4135.
Dynamics of Articulated Robots Kris Hauser CS B659: Principles of Intelligent Robot Motion Spring 2013.
Visual Recognition Tutorial
CSCE 641: Forward kinematics and inverse kinematics Jinxiang Chai.
Quantifying Generalization from Trial-by-Trial Behavior in Reaching Movement Dan Liu Natural Computation Group Cognitive Science Department, UCSD March,
CSCE 641: Forward kinematics and inverse kinematics Jinxiang Chai.
Machine Learning CUNY Graduate Center Lecture 3: Linear Regression.
Discriminative Training of Kalman Filters P. Abbeel, A. Coates, M
Near-Optimal Decision-Making in Dynamic Environments Manu Chhabra 1 Robert Jacobs 2 1 Department of Computer Science 2 Department of Brain & Cognitive.
Support Vector Machines
Biological motor control Andrew Richardson McGovern Institute for Brain Research March 14, 2006.
Computational aspects of motor control and motor learning Michael I. Jordan* Mark J. Buller (mbuller) 21 February 2007 *In H. Heuer & S. Keele, (Eds.),
CS Bayesian Learning1 Bayesian Learning. CS Bayesian Learning2 States, causes, hypotheses. Observations, effect, data. We need to reconcile.
Classification and Prediction: Regression Analysis
CSC2535: 2013 Advanced Machine Learning Lecture 3a: The Origin of Variational Bayes Geoffrey Hinton.
Radial Basis Function Networks
Optimality in Motor Control By : Shahab Vahdat Seminar of Human Motor Control Spring 2007.
Definition of an Industrial Robot
A kinematic cost Reza Shadmehr. Subject’s performanceMinimum jerk motion Flash and Hogan, J Neurosci 1985 Point to point movements generally exhibit similar.
The free-energy principle: a rough guide to the brain? K Friston Summarized by Joon Shik Kim (Thu) Computational Models of Intelligence.
Simple Linear Regression Models
Optimization-Based Full Body Control for the DARPA Robotics Challenge Siyuan Feng Mar
A Shaft Sensorless Control for PMSM Using Direct Neural Network Adaptive Observer Authors: Guo Qingding Luo Ruifu Wang Limei IEEE IECON 22 nd International.
Computer vision: models, learning and inference Chapter 19 Temporal models.
EM and expected complete log-likelihood Mixture of Experts
1  (x) f(x,u) u x f(x,  (x) x Example: Using feed-forward, what should be canceled?
20/10/2009 IVR Herrmann IVR: Introduction to Control OVERVIEW Control systems Transformations Simple control algorithms.
T. Bajd, M. Mihelj, J. Lenarčič, A. Stanovnik, M. Munih, Robotics, Springer, 2010 ROBOT CONTROL T. Bajd and M. Mihelj.
Optimal Therapy After Stroke: Insights from a Computational Model Cheol Han June 12, 2007.
20/10/2009 IVR Herrmann IVR:Control Theory OVERVIEW Control problems Kinematics Examples of control in a physical system A simple approach to kinematic.
Statistical learning and optimal control: A framework for biological learning and motor control Lecture 4: Stochastic optimal control Reza Shadmehr Johns.
BCS547 Neural Decoding. Population Code Tuning CurvesPattern of activity (r) Direction (deg) Activity
ECE-7000: Nonlinear Dynamical Systems Overfitting and model costs Overfitting  The more free parameters a model has, the better it can be adapted.
Reasons to be careful about reward A flow (policy) cannot be specified with a scalar function of states: the fundamental theorem of vector calculus – aka.
Reaching and Grasping: what problems does the brain have to solve?
Model Predictive Impedance Control MPIC
Kinematic Redundancy A manipulator may have more DOFs than are necessary to control a desired variable What do you do w/ the extra DOFs? However, even.
Chapter 7. Learning through Imitation and Exploration: Towards Humanoid Robots that Learn from Humans in Creating Brain-like Intelligence. Course: Robots.
1  The Problem: Consider a two class task with ω 1, ω 2   LINEAR CLASSIFIERS.
Robust Estimation With Sampling and Approximate Pre-Aggregation Author: Christopher Jermaine Presented by: Bill Eberle.
Neural Networks Presented by M. Abbasi Course lecturer: Dr.Tohidkhah.
ECE 8443 – Pattern Recognition ECE 8423 – Adaptive Signal Processing Objectives: Normal Equations The Orthogonality Principle Solution of the Normal Equations.
EEE502 Pattern Recognition
CSC321 Lecture 5 Applying backpropagation to shape recognition Geoffrey Hinton.
(Example) Class Presentation: John Desmond
6. Population Codes Presented by Rhee, Je-Keun © 2008, SNU Biointelligence Lab,
Motor learning through the combination of primitives. Mussa-Ivaldi & Bizzi Phil.Trans. R. Soc. Lond. B 355:
MURI High- Level Control Biomimetic Robots - ONR Site Visit - August 9, 2000 Human Computational Modeling PurposePurpose: to understand arm impedance.
Visuo-Motor Relationships: Plasticity and Development.
Computacion Inteligente Least-Square Methods for System Identification.
Motor System In Autism. Introduction  Motor system  Structure  Process  Challenge  Motor studies of autism  Studies  Sensory system  State estimation.
Ch 7. Computing with Population Coding Summarized by Kim, Kwonill Bayesian Brain: Probabilistic Approaches to Neural Coding P. Latham & A. Pouget.
Summary of points for WS04 – from Weiss & Jeannerod (1998)
PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 1: INTRODUCTION.
Biointelligence Laboratory, Seoul National University
CSC2535: Computation in Neural Networks Lecture 11 Extracting coherent properties by maximizing mutual information across space or time Geoffrey Hinton.
Deep Feedforward Networks
The loss function, the normal equation,
Mathematical Foundations of BME Reza Shadmehr
The Organization and Planning of Movement Ch
Chapter 4 . Trajectory planning and Inverse kinematics
Presentation transcript:

Motor Control

Beyond babbling Three problems with motor babbling: –Random exploration is slow –Error-based learning algorithms are faster but error signals are available in sensory coordinates only –Real arms have two many degrees of freedom

Degrees of freedom 11 22 11 22 Average position

General Learning Principles W X YYdYd Error This works if we have equations for this term…

General Learning Principles W X* YX Error ? ?

Forward Modeling Cortex: Inverse Model Cortex: Inverse Model Desired Position of the hand Reaching Motor command Arm Hand Displacement

Forward Modeling Inverse Model Inverse Model Desired Sensory Change Motor command Plant Sensory Change

Inverse Model Inverse Model Desired Sensory Change:  S d Motor command (M) Arm Sensory Change: :  S -  S d -  S) Learning starts with desired change, not a spontaneous movement. Forward Modeling

Inverse Model Inverse Model Desired Sensory Change:  S d Motor command (M) Arm Sensory Change: :  S - Major problem: How do we compute ?  S d -  S)

Forward Modeling Inverse Model Inverse Model Forward Model Forward Model Desired Sensory Change: :  S d Predicted Sensory Change: :  S p Motor command Plant Sensory Change:  S -  S d -  S p )

Forward Modeling We can’t compute but we can use instead,

Forward Modeling This works as long as: This means that the forward model only needs to be approximately correct

Forward Modeling Inverse Model Inverse Model Forward Model Forward Model Desired Sensory Change: :  S* Predicted Sensory Change:  S p Motor command Plant Sensory Change:  S -  S-  S p ) Training the forward model with motor babbling using prediction error

Forward Modeling Inverse Model Inverse Model Forward Model Forward Model Desired Sensory Change: :  S d Predicted Sensory Change: :  S p Motor command -  S d -  S p ) Off line training of inverse model with predicted performance error

Forward Modeling Inverse Model Inverse Model Forward Model Forward Model Desired Sensory Change: :  S d Predicted Sensory Change: :  S p Motor command Plant Sensory Change:  S -  S d -  S) On line training of inverse models using performance error

Forward Modeling Forward models can be used to predict the consequences of motor commands The prediction can be used to drive a linear inverse model (e.g. Jacobian) since they are not subject to long delays The prediction can be subtracted from current sensory input to compute the prediction error. Errors can be used to improve prediction.

Modular Approach Sensorimotor transformations are context dependent, e.g., the force required to move an object depends on the mass of the objects. A central controller for all contexts might not be feasible and would take too many resources Alternative: use a family of controllers and mix them smoothly across contexts.

Responsibility Estimator Modular Approach X XXX + Inverse Model 1 or

Modular Approach The weights of the inverse model are adjusted according to:

Responsibility Estimator Modular Approach Forward Model 1 X X XX + utut To compute the responsibility, use the forward models Favors the model which makes the best prediction at the previous time step

Modular Approach The weights of the forward model are adjusted according to:

Modular Approach We can also add a responsibility predictor. where y t are the sensory cues used for the predictions.

Modular Approach The overall responsibility is computed according to: plays the role of a likelihood function, while is the prior.

Controlling a trajectory The inverse problem (inverse dynamics) Ex: Controlling the trajectory of a spaceship

Controlling a trajectory The inverse problem (inverse dynamics) Ex: a multi-joint arm (  (t): joint torques)

Controlling a trajectory The inverse problem (inverse dynamics) –Feedback models –Feedforward models –Equilibrium point models

Controlling a trajectory The inverse problem: feedback models Generate force according to the difference between desired position and actual position (easy to compute in linear systems). Unstable because of sensory delays Works better if the “actual position” is internally estimated by a forward model Adapts on the fly to change of context Popular model of the oculomotor system

Controlling a trajectory The inverse problem: feedforward models Estimating torque and its differentials from a desired trajectory is a nonlinear mapping Use a basis function network to implement the mapping Subject to the curse of dimensionality Unable to adapt to change of context (unless you add context units…)

Controlling a trajectory The inverse problem (inverse dynamics) Ex: a multi-joint arm (  (t): joint torques)

Controlling a trajectory Equilibrium models Basic idea: the motor system only specifies the muscle length that maintain the arm at the desired location (in other words, it only specifies the joint coordinates).

Controlling a trajectory Equilibrium models Monkeys can reach accurately without proprioceptive or visual feedback If the arm is moved to the end point right at the onset of a movement, it goes back to the starting point and resume its trajectory. This implies that trajectory are specified by moving the equilibrium point

Controlling a trajectory Equilibrium models How do you control trajectories? The motor system may specify the trajectory of the equilibrium point (virtual trajectory). Unless muscles are very stiff, the actual trajectory will end up being very different. This makes it difficult to control trajectories very precisely.

Controlling a trajectory There is an infinite number of trajectories between two points. How do we choose one? Do you have to specify one? –Minimum Jerk (Flash & Hogan) –Maximum accuracy (Harris & Wolpert) –Optimal control (Todorov & Jordan)

Controlling a trajectory Minimum Jerk Goal: find  (t) minimizing C. The solution can be found using calculus of variations.

Controlling a trajectory Minimum Jerk It’s unclear how Jerk is computed in the brain. No principled explanation for why the brain would minimize such a quantity.

Controlling a trajectory Maximum accuracy Hypothesis: trajectory are optimized to maximize accuracy Assumption: motor commands are corrupted by noise with standard deviation proportional to mean (this is different from Poisson noise!!) Question: for a fixed duration and amplitude of a movement, what’s the optimal control signal?

Controlling a trajectory Maximum accuracy Large control signals lead to fast but inaccurate movements Small control signals lead to accurate but slow movements. Goal: select the control signal that leads to maximum accuracy for a given duration.

Controlling a trajectory Maximum accuracy w t : white noise with mean zero and variance ku t 2

Controlling a trajectory Maximum accuracy Goal: minimize cov[x t ] under the constraint that E[x t ] is equal the desired location for several time steps after the end of the movement. Quadratic problem.

Controlling a trajectory Maximum accuracy Eye Movements

Controlling a trajectory Maximum accuracy Arm Movements

Controlling a trajectory Optimal motor control (Todorov-Jordan) Experts show a lot of variance in their movements but high accuracy on end points Indeed, there are directions in motor space that induce no variance in end points, because of the large number of degrees of freedom

Controlling a trajectory Optimal motor control (Todorov-Jordan) Choose trajectory with maximum accuracy and minimum effort

Controlling a trajectory x1x1 x2x2 1 St solution: bring X to a value X* such that x 1 *=(X 0 *) 1, x 2 *=(X 0 *) 2 (X 0 *) 2 (X 0 *) 1 Line where the constraint is verified

Controlling a trajectory x1x1 x2x2 (X 0 *) 2 (X 0 *) 1 Additional noise 1 St solution: bring X to a value X* such that x 1 *=(X 0 *) 1, x 2 *=(X 0 *) 2

Controlling a trajectory x1x1 x2x2 2nd solution: To minimize effort, go to the closest point such that x 1 *+x 2 *=X 0 *

Controlling a trajectory x1x1 x2x2 2nd solution: To minimize effort, go to the closest point such that x 1 *+x 2 *=X 0 *

Controlling a trajectory x1x1 x1x1 x2x2 x2x2

Optimal motor control (Todorov-Jordan) X 1 +X 2 -X* Task error X 1 -X 2 More task error Less variability in solutions

Open question How does the nervous system compute the solutions to those optimization problems? Is it done off line (i.e., does the CNS specify a trajectory?) or on line? Attractor nets?