Broca’s area Pars opercularis Motor cortexSomatosensory cortex Sensory associative cortex Primary Auditory cortex Wernicke’s area Visual associative cortex Visual cortex [Adapted from Neural Basis of Thought and Language Jerome Feldman, Spring 2007, Language categories and concepts

Categories & Prototypes Three ways of examining the categories we form:  relations between categories (e.g. basic-level category)  internal category structure (e.g. radial category)  instances of category members (e.g. prototypes) Furniture SofaDesk leather sofa fabric sofa L-shaped desk Reception disk Basic-Level Category Superordinate Subordinate

Basic-level -- Criteria Perception –  overall perceived shape  single mental image  fast identification

Concepts are not categorical

Radial Structure of Mother The radial structure of this category is defined with respect to the different models Central Case Stepmother Adoptive mother Birth mother Natural mother Foster mother Biological mother Surrogate mother Unwed mother Genetic mother

Action Words- an fMRI study Somatotopic Representation of Action Words in Human Motor and Premotor Cortex  Olaf Hauk, Ingrid Johnsrude,and Friedemann Pulvermuller*  Medical Research Council, Cognition and Brain Sciences Unit Cambridge, United Kingdom  Neuron, Vol. 41, 1–20, January 22, 2004, Copyright  2004 by Cell Press

Traditional theory Unified meaning center in the left temporal lobe.  Connected to Wernicke’s area  Experiments on highly imageable words/nouns. Vocalization and grammar associated with frontal lobe  Connected to Broca’s area

Do action words activate the motor cortex Given: Cortical representations of the face, arm, and leg are discrete and somatotopically organized in the motor and premotor cortex Hypothesis: Words referring to actions performed with the face, arm, or leg would activate premotor networks.  neurons processing the word form and those processing the referent action should frequently fire together and thus become more strongly linked, resulting in word-related networks overlapping with motor and premotor cortex in a somatotopic fashion. Experiment: An fMRI study with word stimuli from different effectors (face, arm, or leg). ROI based on movements (face, arm, leg)

Movement vs. Actions

Neural Evidence for category structure Are there specific regions in the brain to recognize/reason with specific categories? No, but there are specific circuits distributed over relevant regions of the brain.

What are schemas?  Regularities in our perceptual, motor and cognitive systems  Structure our experiences and interactions with the world.  May be grounded in a specific cognitive system, but are not situation-specific in their application (can apply to many domains of experience)

Basis of Image schemas Perceptual systems Motor routines Social Cognition Image Schema properties depend on  Neural circuits  Interactions with the world

Image schemas Trajector / Landmark (asymmetric)  The bike is near the house  ? The house is near the bike Boundary / Bounded Region  a bounded region has a closed boundary Topological Relations  Separation, Contact, Overlap, Inclusion, Surround Orientation  Vertical (up/down), Horizontal (left/right, front/back)  Absolute (E, S, W, N) LM TR bounded region boundary

Spatial schemas TR/LM relation Boundaries, bounded region Topological relations Orientational Axes Proximal/Distal

TR/LM -- asymmetry The cup is on the table ?The table is under the cup. The skateboard is next to the post. ?The post is next to the skateboard.

Topological Relations Separation Contact Coincidence: - Overlap - Inclusion - Encircle/surround

Orientation Vertical axis -- up/down up down above below upright

Relative frame of reference front back left?? right??

Language and Spatial Schemas People say that they look up to some people, but look down on others because those we deem worthy of respect are somehow “above” us, and those we deem unworthy are somehow “beneath” us. Much of our language is rich with such spatial talk. Concrete actions such as a push or a lift clearly imply a vertical or horizontal motion, but so too can more abstract concepts. Metaphors: Arguments can go “back and forth,” and hopes can get “too high.”

Simulation-based language understanding Analysis Process Semantic Specification “Harry walked into the cafe.” Utterance CAFE Simulation Belief State General Knowledge Constructions construction W ALKED form self f.phon  [wakt] meaning : Walk-Action constraints self m.time before Context.speech-time self m..aspect  encapsulated

Simulation specification A simulation specification consists of: - schemas evoked by constructions - bindings between schemas

Language and Thought We know thought (our cognitive processes) constrains the way we learn and use language Does language also influence thought? Benjamin Whorf argues yes Psycholinguistics experiments have shown that linguistics categories influence thinking even in non-linguistics task Language Thought cognitive processes

Natural Theory of Language (NTL) Basic concepts and words derive their meaning from embodied experience. Abstract and theoretical concepts derive their meaning from metaphorical maps to more basic embodied concepts. Structured Connectionist Models can capture both of these processes nicely. Grammar extends this by Constructions: pairings of form with embodied meaning.

Simulation-based language understanding “Harry walked to the cafe.” SchemaTrajectorGoal walkHarrycafe Analysis Process Simulation Specification Utterance Simulation Cafe Constructions General Knowledge Belief State

19 Konvens, The ICSI/Berkeley Neural Theory of Language Project

Background: Primate Motor Control Relevant requirements (Stromberg, Latash, Kandel, Arbib, Jeannerod, Rizzolatti)  Should model coordinated, distributed, parameterized control programs required for motor action and perception.  Should be an active structure.  Should be able to model concurrent actions and interrupts. Model  The NTL project has developed a computational model based on that satisfies these requirements (x- schemas).  Details, papers, etc. can be obtained on the web at

Active representations Representation based on stochastic Petri nets captures dynamic, parameterized nature of actions Many inferences about actions derive from what we know about executing them Generative model: action, planning, recognition, language. Walking: bound to a specific walker with a direction or goal consumes resources (e.g., energy) may have termination condition (e.g., walker at goal ) ongoing, iterative action walker =Harry goal =home energy walker at goal

Somatotopy of Action Observation Foot Action Hand Action Mouth Action Buccino et al. Eur J Neurosci 2001

Language Development in Children 0-3 mo: prefers sounds in native language 3-6 mo: imitation of vowel sounds only 6-8 mo: babbling in consonant-vowel segments 8-10 mo: word comprehension, starts to lose sensitivity to consonants outside native language mo: word production (naming) mo: word combinations, relational words (verbs, adj.) mo: grammaticization, inflectional morphology 3 years – adulthood: vocab. growth, sentence-level grammar for discourse purposes

food toys misc. people sound emotion action prep. demon. social Words learned by most 2-year olds in a play school (Bloom 1993)

Learning Spatial Relation Words Terry Regier A model of children learning spatial relations. Assumes child hears one word label of scene. Program learns well enough to label novel scenes correctly. Extended to simple motion scenarios, like INTO. System works across languages. Mechanisms are neurally plausible.

Learning Verb Meanings David Bailey A model of children learning their first verbs. Assumes parent labels child’s actions. Child associates action with word Program learns well enough to: 1) Label novel actions correctly 2) Obey commands using new words (simulation) System works across languages Mechanisms are neurally plausible.

Motor Control (X-schema) for SLIDE

Parameters for the SLIDE X-schema

Training Results David Bailey English 165 Training Examples, 18 verbs Learns optimal number of word senses (21) 32 Test examples : 78% recognition, 81% action All mistakes were close lift ~ yank, etc. Learned some particle CXN,e.g., pull up Farsi With identical settings, learned senses not in English

Embodied Construction Grammar (Bergen, Chang & Paskin 2000) Assumptions from Construction Grammar  Constructions are form-meaning pairs (Lakoff 1987, Goldberg 1995)  Constructions vary in degree of specificity and level of description (morphological, lexical, phrasal, clausal) Constructions evoke and bind semantic schemas Additional influences  Cognitive Grammar (Langacker 1987)  Frame Semantics (Fillmore 1982)  Structured Connectionism (Feldman 1987)

Conclusion Language acquisition and use is a hallmark of being human  Language seems to rely on fine-grained aspects of embodied (sensory- motor and social cognition) primitives and brain-like computation (massively parallel, distributed, spreading activation, temporal binding).  Understanding requires imaginative simulation! Sensory-Motor imagination and simulation is crucial in interpretation!