1. The ICSI/Berkeley Neural Theory of Language Project
Graduate Students
Leon Barrett (CS)
*Johno Bryant (CS)
*Nancy Chang (CS)
Ellen Dodge (Ling)
Michael Ellsworth (Ling)
Joshua Marker (Ling)
*Eva Mok (CS)
Shweta Narayan (Ling)
*Steve Sinha (CS)
Alumni
Terry Regier (UCB Ling)
David Bailey (Google)
Andreas Stolcke (ICSI, SRI)
Dan Jurafsky (Stanford Ling)
Olya Gurevich (Powerset)
Benjamin Bergen (U. Hawaii Ling)
Carter Wendelken (UCB)
Srini Narayanan (ICSI, UCB)
Gloria Yang (UTD)
Principal investigators
Jerome Feldman (UCB,ICSI)
George Lakoff (UCB Ling)
Srini Narayanan (UCB,ICSI)
Lokendra Shastri (now India)
Affiliated faculty
Chuck Fillmore (ICSI)
Eve Sweetser (UCB Ling)
Rich Ivry (UCB Psych)
Lisa Aziz-Zadeh (USC)

2. Unified Cognitive Science
Neurobiology
Psychology
Computer Science
Linguistics
Philosophy
Social Sciences
Experience
Take all the Findings and Constraints Seriously

3. Constrained Best Fit in Nature
inanimate animate
physics
lowest energy state
chemistry
molecular fit
biology
fitness, MEU
Neuroeconomics
vision
threats, friends
language
errors, NTL
framing, compromise
society, politics

4. Brains ~ Computers 1000 operations/sec 100,000,000,000 units
10,000 connections/
graded, stochastic
embodied
fault tolerant
evolves
learns
1,000,000,000 ops/sec
1-100 processors
~ 4 connections
binary, deterministic
abstract, disembodied
crashes frequently
explicitly designed
is programmed

5. Fast Brain ~ Slow Neurons
Mental Connections are Active Neural Connections
There is No Erasing in the Brain

6. Constraints on Connectionist Models
100 Step Rule
Human reaction times ~ 100 milliseconds
Neural signaling time ~ 1 millisecond
Simple messages between neurons
Long connections are rare
No new connections during learning
Developmentally plausible

7. Connectionist Models in Cognitive Science
Structured
PDP
Hybrid
Neural
Conceptual
Existence
Data Fitting
Fast Mapping Skill Learning
Not discussed in meeting

8. Triangle nodes and McCullough-Pitts Neurons?B C A A B C

9. Representing concepts using triangle nodes

10. Functionalism
In fact, the belief that neurophysiology is even relevant to the functioning of the mind is just a hypothesis. Who knows if we’re looking at the right aspects of the brain at all. Maybe there are other aspects of the brain that nobody has even dreamt of looking at yet. That’s often happened in the history of science. When people say that the mental is just the neurophysiological at a higher level, they’re being radically unscientific. We know a lot about the mental from a scientific point of view. We have explanatory theories that account for a lot of things. The belief that neurophysiology is implicated in these things could be true, but we have very little evidence for it. So, it’s just a kind of hope; look around and you see neurons: maybe they’re implicated.
Noam Chomsky 1993, p.85

11. Embodiment Alan Turing (Intelligent Machines,1948)
Of all of these fields, the learning of languages would be the most impressive, since it is the most human of these activities. This field, however, seems to depend rather too much on the sense organs and locomotion to be feasible.
Alan Turing (Intelligent Machines,1948)
Continuity Principle of the American Pragmatists

12. The ICSI/Berkeley Neural Theory of Language Project
ECG
Learning early constructions (Chang, Mok)
VERY brief overview, to situate my project:
The general tactic is to view complex cognitive phenomena as having more than one level of analysis, here 5 levels. (Cog/ling level that cogsci people study; computational level with relatively standard CS rep’ns; neural networks with structure, etc.)
Importantly, this reduction or abstraction is constrained in that structures or representations used at one level should have equivalent translations or implementations at the more concrete or biologically inspired levels. [Caveat: it’s the NTL project; “neural” is the goal, not necessarily the status quo. Any input welcome.]
e.g. SHRUTI (binding via temporal synchrony)
Most of the previous work has been at the top several levels; my research fits in there too.
I will give a brief illustration of the general approach and show how that leads naturally to the model I’ll be describing (learning early constructions).

13. Ideas from Cognitive Linguistics
Embodied Semantics (Lakoff, Johnson, Sweetser, Talmy
Radial categories (Rosch 1973, 1978; Lakoff 1985)
mother: birth / adoptive / surrogate / genetic, …
Profiling (Langacker 1989, 1991; cf. Fillmore XX)
hypotenuse, buy/sell (Commercial Event frame)
Metaphor and metonymy (Lakoff & Johnson 1980, …)
ARGUMENT IS WAR, MORE IS UP
The ham sandwich wants his check.
Mental spaces (Fauconnier 1994)
The girl with blue eyes in the painting really has green eyes.
Conceptual blending (Fauconnier & Turner 2002, inter alia)
workaholic, information highway, fake guns
“Does the name Pavlov ring a bell?” (from a talk on ‘dognition’!)

14. Simulation-based language understanding
“Harry walked to the cafe.”
Utterance
Constructions
Analysis Process
General Knowledge
Simulation Specification
Schema Trajector Goal
walk Harry cafe
Belief State
Cafe
Simulation

15. Psycholinguistic evidence
Embodied language impairs action/perception
Sentences with visual components to their meaning can interfere with performance of visual tasks (Richardson et al. 2003)
Sentences describing motion can interfere with performance of incompatible motor actions (Glenberg and Kashak 2002)
Sentences describing incompatible visual imagery impedes decision task (Zwaan et al. 2002)
Simulation effects from fictive motion sentences
Fictive motion sentences describing paths that require longer time, span a greater distance, or involve more obstacles impede decision task (Matlock 2000, Matlock et al. 2003)

16. Neural evidence: Mirror neurons
Gallese et al. (1996) found “mirror” neurons in the monkey motor cortex, activated when
an action was carried out
the same action (or a similar one) was seen.
Mirror neuron circuits found in humans (Porro et al. 1996)
Mirror neurons activated when someone:
imagines an action being carried out (Wheeler et al. 2000)
watches an action being carried out (with or without object) (Buccino et al. 2000)

17. Active representations
Many inferences about actions derive from what we know about executing them
Representation based on stochastic Petri nets captures dynamic, parameterized nature of actions
Used for acting, recognition, planning, and language
walker at goal
Walking:
bound to a specific walker with a direction or goal
consumes resources (e.g., energy)
may have termination condition (e.g., walker at goal)
ongoing, iterative action
Certain words are closely associated with biological phenomena.
Simple representation based on petri nets. Action/event representation has in common with motor control the need to refer to
process states and transitions;
and resource consumption/production; parameters.
Part of learning/understanding words like “push”, “walk” clearly involves grounded knowledge about how to perform the action, as well as quite complex/concrete inferences based on execution. (e.g....)
Note: these representations might be parameterized: “shove”, “walk slowly”, “walk home”
and NOTE: this model of action is accurate cross-linguistically, even if some specific conditions on word meaning varies from language to language.
This may seem complex, but in fact VERY early children seem to have no problem performing and understanding words like this. And more complicated ones too!
energy
walker=Harry
goal=home

18. Learning Verb Meanings David Bailey
A model of children learning their first verbs.
Assumes parent labels child’s actions.
Child knows parameters of action, associates with word
Program learns well enough to:
1) Label novel actions correctly
2) Obey commands using new words (simulation)
System works across languages
Mechanisms are neurally plausible.

19. System Overview

20. Learning Two Senses of PUSH
Model merging based on Bayesian MDL

21. NTL Manifesto Basic Concepts are Grounded in Experience
Sensory, Motor, Emotional, Social,
Abstract and Technical Concepts map by Metaphor to more Basic Concepts
Neural Computation models all levels

22. Simulation based Language Understanding
Discourse & Situational Context
Constructions
Utterance
Analyzer:
incremental,
competition-based, psycholinguistically plausible
Semantic Specification:
image schemas, frames, action schemas
Simulation

23. Pragmatics Semantics Syntax Morphology Phonology Phonetics
“Harry walked into the cafe.”
Phonology
Semantics
Pragmatics
Morphology
Syntax
Phonetics

24. UTTERANCE Pragmatics Semantics Syntax Morphology Phonology Phonetics
“Harry walked into the cafe.”
UTTERANCE
Phonology
Semantics
Pragmatics
Morphology
Syntax
Phonetics

25. Embodied Construction Grammar
Embodied representations
active perceptual and motor schemas (image schemas, x-schemas, frames, etc.)
situational and discourse context
Construction Grammar
Linguistic units relate form and meaning/function.
Both constituency and (lexical) dependencies allowed.
Constraint-based
based on feature unification (as in LFG, HPSG)
Diverse factors can flexibly interact.
common schema representations for concepts and constructional meaning
dynamic, cognitively motivated representations (image schemas, x-schemas, frames, etc.)

26. Embodiment and Grammar Learning
Paradigm problem for Nature vs. Nurture
The poverty of the stimulus

27. Embodiment and Grammar Learning
Paradigm problem for Nature vs. Nurture
The poverty of the stimulus
The opulence of the substrate
Intricate interplay of genetic and environmental, including social, factors.

28. Embodied Construction Grammar ECG (Formalizing Cognitive Linguisitcs)
Linguistic Analysis
Computational Implementation
Test Grammars
Applied Projects – Question Answering
Map to Connectionist Models, Brain
Models of Grammar Acquisition