1. Situation Models and Embodied Language Processes
Franz Schmalhofer
University of Osnabrück / Germany
Memory and Situation Models
Computational Modeling of Inferences
What Memory and Language are for
Neural Correlates
Integration of Behavioral Experiments and Neural Correlates (ERP; fMRI) by Formal Models

2. Theories of Knowledge
Prior to the twentieth century: knowledge was assumed to be perceptual
Past several decades: fields of cognition and perception have diverged.
Perceptual approaches viewed as untenable for conceptual systems.
Logic, statistics, programming languages have inspired amodal theories different from perceptional characteristics

3. Symbol Grounding
Illustration for computational theories of language understanding:
The Chinese room (Searle, 1980)
- getting Chinese input symbols
- manipulation of symbols only according to their
shapes ( no meanings/no „understanding“ )
returning Chinese symbols as output
The symbol grounding problem (Harnad, 1990)
cognition cannot be just symbol manipulation

4. Perceptual Symbol Systems (Barsalou, 1999)
WRONG: Perceptual systems pick up information from the environment and pass it on to separate systems that support the various cognitive functions. (i.e. language, memory, and thought)
CORRECT: Cognition is inherently perceptual, sharing systems with perception at both the cognitive and the neural levels.
No divergence between cognition and perception

5. How we got the wrong ideas
Behaviorist attacks on mentalism. (Watson)
Similar attacks from ordinary language philosophy (Wittgenstein)
Continuing attacks after the cognitive revolution.
Development of logic, programming languages, statistical representation
Contributions of amodal symbol systems :
Formalizable, runnable, applicable
Highlight important computational properties of productivity, proposition, structure, etc.

6. Assumption: Cognition is Grounded in Perception
A common representational system underlies perception and cognition.
From Aristoteles to Locke to Kant, theorists over the last 2000 years have viewed cognition as imagistic in nature.
Image-based theories disappeared as behaviorists and language philosophers began to avoided to talk about mental states.

7. How is cognition grounded in perception?
During perceptual experience, association areas in the brain capture bottom-up patterns of activation in sensory-motor areas and in a top-down manner, association areas partially reactivate sensory-motor areas to implement perceptual symbols.
The storage and reactivation of perceptual symbols operates at the level of perceptual components--not at the level of holistic perceptual experiences.
Use of selective attention, schematic representations of perceptual components are extracted from experience and stored in memory

8. Properties of amodal symbol system
As amodal symbols are transduced from perceptual states, they enter into larger representational structures.
In turn, these structures support all of the higher cognitive functions, including knowledge, memory, language and thought.
Across the cognitive sciences, standard theories of knowledge adopt the assumptions, so called amodal symbol systems.
Information in the physical world produces neural states in perceptual systems.
A transduction process takes these states as input, produces descriptions of them in a completely new representation language.

9. Amodal symbol representations
4. Amodal symbol system
Perceptual states are transduced into a completely new representational system that describes these states amodally.
The internal structure of these symbols is unrelated to the perceptual state that produced them.

10. Problems with amodal symbol system
No account for the relation between cognition and perception.
No empirical evidence that amodal systems exist.
Transduction process that maps perceptual states into amodal symbols remains unclear.
Converse of transduction problem : no account how perceptual states map to amodal symbols.
Too powerful : amodal symbol systems are unconstrained, offer little insight into the phenomena.

11. Resurgence of perceptual symbol system
Theorists develop perceptual views that are provocative, powerful.
It provides a natural account of the relation between cognition and perception.
An obvious account exists of how perceptual symbols are implemented in the brain.
No need for a major leap in evolution
Provide natural mechanisms for representing space and time
Make clear a priori predictions that are falsifiable
Growing empirical evidence for perceptual symbols

12. Basic assumption of perceptual views
States arise in sensory-motor systems during contact with the physical world.
traditionally viewed as conscious states
but will be viewed here primarily as neural states
these sensory-motor states are stored in memory to some extent (utilizing sensory-motor systems)
stored perceptual states later support higher cognitive processes
during memory, language, and thought
may establish reference back into the physical world

13. Representation in Perceptual Symbol Systems
Subsets of perceptual states in sensory-motor systems are extracted.
The internal structure of these symbols is therefore
modal and
analogically related to the perceptual state that produced them.

15. Core assumptions about perceptual symbols
Perceptual symbols are schematic.
Perceptual symbols are multimodal
Perceptual symbols enter into simulation competence.
Perceptual symbols are productive.
Perceptual symbols represent situation components.
Perceptual symbols also represent abstract concepts

16. Experiments
Spivey, M.J. et al. (2000) Eye movements during comprehension of spoken scene descriptions.
Zwaan, R.A., Stanfield, R. Y. & Yaxley, R. H. (2002) Do language comprehenders routinely represent the shapes of objects. Psychological Science, 13,
Glenberg A. M. & Kaschak M. P. (2002) Grounding language in action. Psychonomic Bulletin & Review, 9,

17. Zwaan (2004) The immersed experiencer
C = construal
T = time
S = spatial region (personal, action, vista)
P = perspective
F = focal entity
R = relation
B = background entity
f = feature

18. Embodied Language Comprehension
Taylor and Tversky (1992): “language is a surrogate for experience”
Goal of language comprehension: creation of an embodied mental model
In the brain words activate experiences with their referents (Pulvermüller)
Perceptual simulation of a described object or situation; construction of a situation model 4)
Language comprehension is based upon the same mechanisms as the comprehension or perception of the environment: it depends on the successive transformation of conceptualizations that are patterns of possible action. Language comprehension requires further suppression of the real environment.
Tayler and Tversky: language is a surrogate for experience. For language to be a useful surrogate it must make use of the same sort of embodied representations that we use to understand the environment. Language does this: Understanding language is based on creating embodied conceptualizations of situations the language is describing. A mental model / situation model derived from language incorporates constraints and experiences like those that are derived from the real world. Constraints imposed upon a mental model of a situation are looser – an airplane described by a sentence can not grasp all the details of the plane that we perceive of it in the real world. Conceptualizations derived from language do not constrain action as efficiently as conceptualization derived from the environment. Nevertheless in understanding the situation described in a text passage we need to combine “mesh” the different patterns of action derived from each descriptive part / word to get a coherent and meaningful representation of the situation. in understanding language we create an embodied model of the situation in which the meshed patterns of possible actions constrain and modify further conceptualizations and make predictions about upcoming events possible. 5)
Goal of language comprehension: creation of a conceptualizaton of meshed patterns of action. Interpretation of a word, phrase involves the meshing of actions derived from the specific word or phrase with patterns of actions that are derived from previous text  embodied mental model
“In short, the basic assumption is reading or hearing a word activates experiential representations of words (lexical, grammatical, phonological, motoric, tactile) as well as associated experiential representations of their referents –motor, perceptual, and emotional representations, and often combinations of these (see also Sadoski & Paivio, 2001). These traces can be activated by verbal input and as such enable the reconstitution of experience. In this sense, then, comprehension is the vicarious experience of the described events through the integration and sequencing of traces from actual experience cued by the linguistic input “ Zwaan 2003 Immersed Experiencer

19. Visual field
Fovea
Para-fovea
Plateau
Periphery
Temporal monocular

20. From the Retina to V1 nasal retinal fibres temporal retinal fibres LGN
major relay station between the eye(s) and the visual cortex
part of the thalamus; cells are arranged in layers; each layer receiving input from either the L eye or the R eye
temporal retinal fibres

21. ‘Vision for Perception’
Visual Pathways
‘Vision for Action’
Goodale & Milner (1992)
streams differ on the basis what the vis. Info is USED for!
Ventral path - vis. Perception of objects
dorsal path - visually-guided actions directed at objects
‘Vision for Perception’
Goodale & Humphrey (1998)

22. Perceptual Experience and Perceptual States
Perceptual experiences activate association areas in the brain which capture
bottom-up patterns of activation in sensory-motor areas.
top-down patterns, association areas partially reactivate sensory-motor areas to implement perceptual symbols.
Perceptual States:
Arise in sensory-motor systems with two components
Unconscious neural representation of physical input
Optional conscious experience
Related perceptual symbols combine to form simulators that allow “the cognitive system to construct specific simulations of an entity or event in its absence

23. Perceptual symbols
Are patterns that rise in hierarchical feature maps of sensory-motor systems during perception and action.
May or may not be topographical; captured by association areas (Damasio, 1989)
From local, to poly-sensory and to frontal
Tuned for specific combinations of features
Activating an associative pattern reinstates
not necessarily complete nor veridical but partial records of the neural states that underlie perception
Is dynamic, not discrete
not necessarily representative of specific individuals
potentially indeterminate

25. Perceptual symbols are NOT
like physical pictures
entire perceptual states
If they were, the componential character of conceptual system would not be feasible
mental images or conscious experience
states in neural systems

26. Simulators
Perceptual symbols of the same category are integrated together in a single system (simulators).
A simulator
An organized system of perceptual symbols that can produce simulations of a category in the absence of physical exemplars.
Typically contains perceptual symbols extracted from many members of category on many modalities.
Composed of frames, the simulations that the frame produces.

27. Simulators and Simulations
Specific runs of a simulator that reenact the multimodal experience of a category
Utilize a small subset of the information in a simulator
Infinite many simulations is possible.
Simulations are always partial and sketchy, never complete.
A simulator goes beyond a simple empirical collection of sense impressions.
A huge set of simulations that include the range of experience associated with a category
i.e. the representation of a type, not a token

29. Simulations as
vehicles.
This figure is showing
some partial frame
for car, it illustrates
how the frame has
changed dynamically.

30. Categorization by construal
When an entity is categorized, the best fitting simulator is found
The simulator runs a simulation that provides a good fit to the entity
Some examples
Hearing a bark and simulating a dog
Seeing a growling dog and simulating the experience of being attacked
Smelling food and simulating what it is

31. Concepts and Offline Conceptualization
Concepts and Simulators
A concept is equivalent to a simulator.
If we have an appropriate simulator of something, then it can be said that we understand the concept.
Goal of human learning is to establish simulators.
Offline conceptualizing during memory, language and thought
Simulations provide inferences about likely properties of entities in their absence
The simulations run in sensory-motor systems and they can be mapped later to perceived entities
E.g. remembering one’s parking spot, finding the referent of a linguistic description.

33. Summary I: Perceptual Symbol Systems
Perceptual states arise in the sensory-motor system.
A subset of the state is extracted by selective attention and stored in long-term memory.
This perceptual memory can function as a symbol entering into symbol manipulation.
Collections of the perceptual symbols comprise our conceptual representations.
The structure of a perceptual symbol corresponds (at least somewhat) to the perceptual state that produced it.
Note: this does not claim that a perceptual symbol corresponds to the physical world.

34. Summary II: Perceptual Symbol Systems
A very different approach to knowledge in the form of perceptual symbol systems.
Perceptual states are not transduced into a completely new representational language, instead subsets of perceptual states are extracted to function symbolically and support the higher cognitive functions.
Reference: Barsalou, L. W. (1999). “Perceptual Symbol Systems.” Behavioral and Brain Sciences 22: ( ).