Situation Models and Embodied Language Processes Franz Schmalhofer University of Osnabrück / Germany Memory and Situation Models Computational Modeling of Inferences What Memory and Language are for Neural Correlates Integration of Behavioral Experiments and Neural Correlates (ERP; fMRI) by Formal Models

Theories of Knowledge Prior to the twentieth century: knowledge was assumed to be perceptual Past several decades: fields of cognition and perception have diverged. Perceptual approaches viewed as untenable for conceptual systems. Logic, statistics, programming languages have inspired amodal theories different from perceptional characteristics

Symbol Grounding Illustration for computational theories of language understanding: The Chinese room (Searle, 1980) - getting Chinese input symbols - manipulation of symbols only according to their shapes ( no meanings/no „understanding“ ) returning Chinese symbols as output The symbol grounding problem (Harnad, 1990) cognition cannot be just symbol manipulation

Perceptual Symbol Systems (Barsalou, 1999) WRONG: Perceptual systems pick up information from the environment and pass it on to separate systems that support the various cognitive functions. (i.e. language, memory, and thought) CORRECT: Cognition is inherently perceptual, sharing systems with perception at both the cognitive and the neural levels. No divergence between cognition and perception

How we got the wrong ideas Behaviorist attacks on mentalism. (Watson) Similar attacks from ordinary language philosophy (Wittgenstein) Continuing attacks after the cognitive revolution. Development of logic, programming languages, statistical representation Contributions of amodal symbol systems : Formalizable, runnable, applicable Highlight important computational properties of productivity, proposition, structure, etc.

Assumption: Cognition is Grounded in Perception A common representational system underlies perception and cognition. From Aristoteles to Locke to Kant, theorists over the last 2000 years have viewed cognition as imagistic in nature. Image-based theories disappeared as behaviorists and language philosophers began to avoided to talk about mental states.

How is cognition grounded in perception? During perceptual experience, association areas in the brain capture bottom-up patterns of activation in sensory-motor areas and in a top-down manner, association areas partially reactivate sensory-motor areas to implement perceptual symbols. The storage and reactivation of perceptual symbols operates at the level of perceptual components--not at the level of holistic perceptual experiences. Use of selective attention, schematic representations of perceptual components are extracted from experience and stored in memory

Properties of amodal symbol system As amodal symbols are transduced from perceptual states, they enter into larger representational structures. In turn, these structures support all of the higher cognitive functions, including knowledge, memory, language and thought. Across the cognitive sciences, standard theories of knowledge adopt the assumptions, so called amodal symbol systems. Information in the physical world produces neural states in perceptual systems. A transduction process takes these states as input, produces descriptions of them in a completely new representation language.

Amodal symbol representations 4. Amodal symbol system Perceptual states are transduced into a completely new representational system that describes these states amodally. The internal structure of these symbols is unrelated to the perceptual state that produced them.

Problems with amodal symbol system No account for the relation between cognition and perception. No empirical evidence that amodal systems exist. Transduction process that maps perceptual states into amodal symbols remains unclear. Converse of transduction problem : no account how perceptual states map to amodal symbols. Too powerful : amodal symbol systems are unconstrained, offer little insight into the phenomena.

Resurgence of perceptual symbol system Theorists develop perceptual views that are provocative, powerful. It provides a natural account of the relation between cognition and perception. An obvious account exists of how perceptual symbols are implemented in the brain. No need for a major leap in evolution Provide natural mechanisms for representing space and time Make clear a priori predictions that are falsifiable Growing empirical evidence for perceptual symbols

Basic assumption of perceptual views States arise in sensory-motor systems during contact with the physical world. traditionally viewed as conscious states but will be viewed here primarily as neural states these sensory-motor states are stored in memory to some extent (utilizing sensory-motor systems) stored perceptual states later support higher cognitive processes during memory, language, and thought may establish reference back into the physical world

Representation in Perceptual Symbol Systems Subsets of perceptual states in sensory-motor systems are extracted. The internal structure of these symbols is therefore modal and analogically related to the perceptual state that produced them.

Core assumptions about perceptual symbols Perceptual symbols are schematic. Perceptual symbols are multimodal Perceptual symbols enter into simulation competence. Perceptual symbols are productive. Perceptual symbols represent situation components. Perceptual symbols also represent abstract concepts

Experiments Spivey, M.J. et al. (2000) Eye movements during comprehension of spoken scene descriptions. Zwaan, R.A., Stanfield, R. Y. & Yaxley, R. H. (2002) Do language comprehenders routinely represent the shapes of objects. Psychological Science, 13, 168-171. Glenberg A. M. & Kaschak M. P. (2002) Grounding language in action. Psychonomic Bulletin & Review, 9, 558-565

Zwaan (2004) The immersed experiencer C = construal T = time S = spatial region (personal, action, vista) P = perspective F = focal entity R = relation B = background entity f = feature

Embodied Language Comprehension Taylor and Tversky (1992): “language is a surrogate for experience” Goal of language comprehension: creation of an embodied mental model In the brain words activate experiences with their referents (Pulvermüller) Perceptual simulation of a described object or situation; construction of a situation model 4) Language comprehension is based upon the same mechanisms as the comprehension or perception of the environment: it depends on the successive transformation of conceptualizations that are patterns of possible action. Language comprehension requires further suppression of the real environment. Tayler and Tversky: language is a surrogate for experience. For language to be a useful surrogate it must make use of the same sort of embodied representations that we use to understand the environment. Language does this: Understanding language is based on creating embodied conceptualizations of situations the language is describing. A mental model / situation model derived from language incorporates constraints and experiences like those that are derived from the real world. Constraints imposed upon a mental model of a situation are looser – an airplane described by a sentence can not grasp all the details of the plane that we perceive of it in the real world. Conceptualizations derived from language do not constrain action as efficiently as conceptualization derived from the environment. Nevertheless in understanding the situation described in a text passage we need to combine “mesh” the different patterns of action derived from each descriptive part / word to get a coherent and meaningful representation of the situation. in understanding language we create an embodied model of the situation in which the meshed patterns of possible actions constrain and modify further conceptualizations and make predictions about upcoming events possible. 5) Goal of language comprehension: creation of a conceptualizaton of meshed patterns of action. Interpretation of a word, phrase involves the meshing of actions derived from the specific word or phrase with patterns of actions that are derived from previous text  embodied mental model “In short, the basic assumption is reading or hearing a word activates experiential representations of words (lexical, grammatical, phonological, motoric, tactile) as well as associated experiential representations of their referents –motor, perceptual, and emotional representations, and often combinations of these (see also Sadoski & Paivio, 2001). These traces can be activated by verbal input and as such enable the reconstitution of experience. In this sense, then, comprehension is the vicarious experience of the described events through the integration and sequencing of traces from actual experience cued by the linguistic input “ Zwaan 2003 Immersed Experiencer

Visual field Fovea Para-fovea Plateau Periphery Temporal monocular

From the Retina to V1 nasal retinal fibres temporal retinal fibres LGN major relay station between the eye(s) and the visual cortex part of the thalamus; cells are arranged in layers; each layer receiving input from either the L eye or the R eye temporal retinal fibres

‘Vision for Perception’ Visual Pathways ‘Vision for Action’ Goodale & Milner (1992) streams differ on the basis what the vis. Info is USED for! Ventral path - vis. Perception of objects dorsal path - visually-guided actions directed at objects ‘Vision for Perception’ Goodale & Humphrey (1998)

Perceptual Experience and Perceptual States Perceptual experiences activate association areas in the brain which capture bottom-up patterns of activation in sensory-motor areas. top-down patterns, association areas partially reactivate sensory-motor areas to implement perceptual symbols. Perceptual States: Arise in sensory-motor systems with two components Unconscious neural representation of physical input Optional conscious experience Related perceptual symbols combine to form simulators that allow “the cognitive system to construct specific simulations of an entity or event in its absence

Perceptual symbols Are patterns that rise in hierarchical feature maps of sensory-motor systems during perception and action. May or may not be topographical; captured by association areas (Damasio, 1989) From local, to poly-sensory and to frontal Tuned for specific combinations of features Activating an associative pattern reinstates not necessarily complete nor veridical but partial records of the neural states that underlie perception Is dynamic, not discrete not necessarily representative of specific individuals potentially indeterminate

Perceptual symbols are NOT like physical pictures entire perceptual states If they were, the componential character of conceptual system would not be feasible mental images or conscious experience states in neural systems

Simulators Perceptual symbols of the same category are integrated together in a single system (simulators). A simulator An organized system of perceptual symbols that can produce simulations of a category in the absence of physical exemplars. Typically contains perceptual symbols extracted from many members of category on many modalities. Composed of frames, the simulations that the frame produces.

Simulators and Simulations Specific runs of a simulator that reenact the multimodal experience of a category Utilize a small subset of the information in a simulator Infinite many simulations is possible. Simulations are always partial and sketchy, never complete. A simulator goes beyond a simple empirical collection of sense impressions. A huge set of simulations that include the range of experience associated with a category i.e. the representation of a type, not a token

Simulations as vehicles. This figure is showing some partial frame for car, it illustrates how the frame has changed dynamically.

Categorization by construal When an entity is categorized, the best fitting simulator is found The simulator runs a simulation that provides a good fit to the entity Some examples Hearing a bark and simulating a dog Seeing a growling dog and simulating the experience of being attacked Smelling food and simulating what it is

Concepts and Offline Conceptualization Concepts and Simulators A concept is equivalent to a simulator. If we have an appropriate simulator of something, then it can be said that we understand the concept. Goal of human learning is to establish simulators. Offline conceptualizing during memory, language and thought Simulations provide inferences about likely properties of entities in their absence The simulations run in sensory-motor systems and they can be mapped later to perceived entities E.g. remembering one’s parking spot, finding the referent of a linguistic description.

Summary I: Perceptual Symbol Systems Perceptual states arise in the sensory-motor system. A subset of the state is extracted by selective attention and stored in long-term memory. This perceptual memory can function as a symbol entering into symbol manipulation. Collections of the perceptual symbols comprise our conceptual representations. The structure of a perceptual symbol corresponds (at least somewhat) to the perceptual state that produced it. Note: this does not claim that a perceptual symbol corresponds to the physical world.

Summary II: Perceptual Symbol Systems A very different approach to knowledge in the form of perceptual symbol systems. Perceptual states are not transduced into a completely new representational language, instead subsets of perceptual states are extracted to function symbolically and support the higher cognitive functions. Reference: Barsalou, L. W. (1999). “Perceptual Symbol Systems.” Behavioral and Brain Sciences 22: (507-569).