1. Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 10. MNS Model 1 1 Michael Arbib: CS564 - Brain Theory and Artificial Intelligence University of Southern California, Fall 2001 Lecture 10. The Mirror Neuron System Model (MNS) 1 Reading Assignment: Schema Design and Implementation of the Grasp-Related Mirror Neuron System Erhan Oztop and Michael A. Arbib

2. Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 10. MNS Model 1 2 Visual Control of Grasping in Macaque Monkey F5 - grasp commands in premotor cortex Giacomo Rizzolatti AIP - grasp affordances in parietal cortex Hideo Sakata A key theme of visuomotor coordination: parietal affordances (AIP) drive frontal motor schemas (F5)

3. Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 10. MNS Model 1 3 Mirror Neurons Rizzolatti, Fadiga, Gallese, and Fogassi, 1995: Premotor cortex and the recognition of motor actions Mirror neurons form the subset of grasp-related premotor neurons of F5 which discharge when the monkey observes meaningful hand movements made by the experimenter or another monkey. F5 is endowed with an observation/execution matching system

4. Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 10. MNS Model 1 4 F5 Motor Neurons F5 Motor Neurons include all F5 neurons whose firing is related to motor activity.  We focus on grasp-related behavior. Other F5 motor neurons are related to oro- facial movements. F5 Mirror Neurons form the subset of grasp-related F5 motor neurons of F5 which discharge when the monkey observes meaningful hand movements. F5 Canonical Neurons form the subset of grasp-related F5 motor neurons of F5 which fire when the monkey sees an object with related affordances.

5. Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 10. MNS Model 1 5 What is the mirror system (for grasping) for? Action recognition Understanding (assigning meaning to other’s actions) Associative memory for actions Mirror neurons: The cells that selectively discharge when the monkey executes particular actions as well as when the monkey observes an other individual executing the same action. Mirror neuron system (MNS): The mirror neurons and the brain regions involved in eliciting mirror behavior. Interpretations:

6. Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 10. MNS Model 1 6 Computing the Mirror System Response The FARS Model: Recognize object affordances and determine appropriate grasp. The Mirror Neuron System (MNS) Model: We must add recognition of  trajectory and  hand preshape to  recognition of object affordances and ensure that all three are congruent. There are parietal systems other than AIP adapted to this task.

7. Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 10. MNS Model 1 7 Further Brain Regions Involved cIPS Spatial coding for objects, analysis of motion during interaction of objects and self-motion 7b (PF): Rostral part of the posterior parietal lobule Mainly somatosensory Mirror-like responses STS: Superior Temporal Sulcus Detection of biologically meaningful stimuli (e.g.hand actions) Motion related activity (MT/MST part) cIPS Axis and surface orientation cIPS: caudal intraparietal sulcus 7a (PG): caudal part of the posterior parietal lobule

8. Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 10. MNS Model 1 8 cIPS cell response Surface orientation selectivity of a cIPS cell Sakata et al. 1997 cIPS

9. Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 10. MNS Model 1 9 Key Criteria for Mirror Neuron Activation When Observing a Grasp a) Does the preshape of the hand correspond to the grasp encoded by the mirror neuron? b) Does this preshape match an affordance of the target object? c) Do samples of the hand state indicate a trajectory that will bring the hand to grasp the object? Modeling Challenges: i) To have mirror neurons self-organize to learn to recognize grasps in the monkey’s motor repertoire ii) To learn to activate mirror neurons from smaller and smaller samples of a trajectory.

10. Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 10. MNS Model 1 10 Initial Hypothesis on Mirror Neuron Development The development of the (grasp) mirror neuron system in a healthy infant is driven by the visual stimuli generated by the actions (grasps) performed by the infant himself. The infant (with maturation of visual acuity) gains the ability to map other individual’s actions into his internal motor representation. [In the MNS model, the hand state provides the key representation for this transfer.] Then the infant acquires the ability to create (internal) representations for novel actions observed. Parallel to these achievements, the infant develops an action prediction capability (the recognition of an action given the prefix of the action and the target object)

11. Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 10. MNS Model 1 11 The Mirror Neuron System (MNS) Model

12. Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 10. MNS Model 1 12 Implementing the Basic Schemas of the Mirror Neuron System (MNS) Model using Artificial Neural Networks (Work of Erhan Oztop)

13. Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 10. MNS Model 1 13 Opposition Spaces and Virtual Fingers The goal of a successful preshape, reach and grasp is to match the opposition axis defined by the virtual fingers of the hand with the opposition axis defined by an affordance of the object (Iberall and Arbib 1990)

14. Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 10. MNS Model 1 14 Hand State Our current representation of hand state defines a 7-dimensional trajectory F(t) with the following components F(t) = (d(t), v(t), a(t), o 1 (t), o 2 (t), o 3 (t), o 4 (t)): d(t): distance to target at time t v(t): tangential velocity of the wrist a(t): Aperture of the virtual fingers involved in grasping at time t o 1 (t): Angle between the object axis and the (index finger tip – thumb tip) vector [relevant for pad and palm oppositions] o 2 (t): Angle between the object axis and the (index finger knuckle – thumb tip) vector [relevant for side oppositions] o 3 (t), o 4 (t): The two angles defining how close the thumb is to the hand as measured relative to the side of the hand and to the inner surface of the palm.

15. Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 10. MNS Model 1 15 Curve recognition The general problem: associate N-dimensional space curves with object affordances A special case: The recognition of two (or three) dimensional trajectories in physical space Simplest solution: Map temporal information into spatial domain. Then apply known pattern recognition techniques. Problem with simplest solution: The speed of the moving point can be a problem! The spatial representation may change drastically with the speed Scaling can overcome the problem. However the scaling must be such that it preserves the generalization ability of the pattern recognition engine. Solution: Fit a cubic spline to the sampled values. Then normalize and re- sample from the spline curve. Result:Very good generalization. Better performance than using the Fourier coefficients to recognize curves.

16. Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 10. MNS Model 1 16 Curve recognition Spatial resolution: 30 Network input size: 30 Hidden layer size: 15 Output size: 5 Training : Back-propagation with momentum.and adaptive learning rate Sampled points Point used for spline interpolation Fitted spline Curve recognition system demonstrated for hand drawn numeral recognition (successful recognition examples for 2, 8 and 3).

17. Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 10. MNS Model 1 17 STS hand shape recognition Model Matching Precision grasp Hand Configuration Classification Step 2: The feature vector generated by the first step is used to fit a 3D-kinematics model of the hand by the model matching module. The resulting hand configuration is sent to the classification module. Color Coded Hand Feature Extraction Step 1 of hand shape recognition: system processes the color-coded hand image and generates a set of features to be used by the second step

18. Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 10. MNS Model 1 18 STS hand shape recognition 1: Color Segmentation and Feature Extraction Preprocessing Color Expert (Network weights) Training phase: A color expert is generated by training a feed-forward network to approximate human perception of color. Features Actual processing: The hand image is fed to the augmented segmentation system. The color decision during segmentation is done by consulting color expert. NN augmented segmentation system

19. Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 10. MNS Model 1 19 STS hand shape recognition2: 3D Hand Model Matching A realistic drawing of hand bones. The hand is modelled with 14 degrees of freedom as illustrated. ClassificationGrasp Type Result of feature extraction Feature Vector Error minimization The model matching algorithm minimizes the error between the extracted features and the model hand.

20. Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 10. MNS Model 1 20 Virtual Hand/Arm and Reach/Grasp Simulator A power grasp and a side grasp A precision pinch

21. Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 10. MNS Model 1 21 Power grasp time series data +: aperture; *: angle 1; x: angle 2;  : 1-axisdisp1;  :1-axisdisp2;  : speed;  : distance.

22. Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 10. MNS Model 1 22 Core Mirror Circuit Hand state Mirror Neurons (F5mirror) Association (7b) Neurons Mirror Feedback Object affordance Mirror Neuron Output Motor Program (F5 canonical) Hand shape recognition & Hand motion detection Hand-Object spatial relation analysis Object affordance - hand state association Object Affordances Action recognition (Mirror Neurons) Motor program Motor execution Mirror Feedback Integrate temporal association Motor program F5canonical F5mirror

23. Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 10. MNS Model 1 23 Connectivity pattern 7b7b Mirror Feedback Object affordance (AIP) 7a7a STS F5mirror Motor Program (F5canonical)

24. Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 10. MNS Model 1 24 A single grasp trajectory viewed from three different angles The wrist trajectory during the grasp is shown by square traces, with the distance between any two consecutive trace marks traveled in equal time intervals. How the network classifies the action as a power grasp. Empty squares: power grasp output; filled squares: precision grasp; crosses: side grasp output

25. Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 10. MNS Model 1 25 Power and precision grasp resolution (a) (b) Power Grasp Mirror Neuron Precision Pinch Mirror Neuron Note that the modeling yields novel predictions for time course of activity across a population of mirror neurons.

26. Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 10. MNS Model 1 26 Research Plan Development of the Mirror System  Development of Grasp Specificity in F5 Motor and Canonical Neurons  Visual Feedback for Grasping: A Possible Precursor of the Mirror Property Recognition of Novel and Compound Actions and their Context  The Pliers Experiment: Extending the Visual Vocabulary  Recognition of Compounds of Known Movements  From Action Recognition to Understanding: Context and Expectation