Presentation on theme: "1 Can We Simulate Negation? The Simulation Effects of Negation in English Intransitive Sentences Meylysa Tseng Jung-Hee Kim Benjamin Bergen University."— Presentation transcript:
1 Can We Simulate Negation? The Simulation Effects of Negation in English Intransitive Sentences Meylysa Tseng Jung-Hee Kim Benjamin Bergen University of Hawai ’ i at Manoa BLS 2006
2 Thanks to Amy Schafer William O ’ Grady Kathryn Wheeler Cognitive Linguistics Research Group at University of Hawaii
3 Mental imagery When you hear The dolphin soared, You may be mentally picturing (simulating) a visual scene Mental imagery comes from our perceptual or motor experience Mental imagery should resemble our perceptual or motor experience
4 Imagery in Language Imagery is crucial in language use (Zwaan 1999, Barsalou 1999, Glenberg & Robertson 2000, Bergen & Chang 2005) The idea: –Language is associated with concrete perceptual or motor experiences –These experiences are recreated through imagery during language understanding –Such perceptual or motor imagery is activated in language understanding The dolphin soared (visual imagery) I pushed the door (motor imagery)
5 Dimensions of Visual imagery used in language processing Orientation of objects (Stanfield & Zwaan 2001) Shape of objects (Zwaan et al. 2002) Spatial location (Richardson et al. 2003, Lindsay 2003) Upwards or downwards motion (Bergen 2005)
6 Bergen 2005: Up or Down visual imagery What parts of the sentences yield the simulation effects? –The mule climbed (Up-Verb) –The glass fell (Down-Verb) –The sky darkened (Up-Noun) –The ground shook (Down-Noun) It was concrete Subject Nouns and Verbs denoting up/down motion or location Understanding sentences with subject nouns or verbs that entail upness or downness visual imagery selectively interfered with the visual processing of objects in the same parts of the visual field
7 Today: Simulating Negation Negation presents an especially difficult case for a simulation-based approach You can ’ t imagine what ’ s not happening How can negation be captured or represented in a simulation-based account?
8 Questions remain Understanding negated sentences also engages mental simulation? Up/down visual imagery is activated in comprehending ‘ The mule didn ’ t climb ’ or ‘ The sky didn ’ t darken ’ ?
9 Simulation-based account of negation comprehension Negation can be understood as a process “ Negation is implicitly encoded in the simulation processes that are undertaken when comprehending a negative sentence ” (Kaup and L ü dtke 2005) Negation involves two simulations –Simulation of the counterfactual situation –Suppression –Simulation of the factual situation
10 This study: Negation & Up/Down Visual Imagery This study replicated Bergen 2005 We hypothesize : mental simulation is engaged in understanding negation We test: Up/Down visual imagery is activated by such up/down- associated verbs and nouns even when embedded in the negated sentences Stimuli : up/down-associated negative sentences –We negated Bergen ’ s literal sentences with up or down-associated nouns or verbs The mule didn ’ t climb (Up-Verb) The glass didn ’ t fall (Down-Verb) The sky didn ’ t darken (Up-Noun) The ground didn ’ t shake (Down-Noun)
11 Predictions (1)In processing negated sentences, if up/down associated noun and verbs induce up/down visual imagery especially in the counterfactual simulation stage, there should be an interaction effect between noun/verb location and object location in a visual object categorization task (2) Consistent with the proposed stages of negation simulation, we would see a shift of simulation from counterfactual to factual over time
12 Time course – 200ms, 500ms 200ms – the initial counterfactual simulation –Up/Down visual imagery was activated at 200ms time delay in processing up/down-associated ‘ positive ’ sentences (Bergen 2005) 500ms – the later factual simulation –‘ Negation seemed to effect some time between 500 and 1,000 msec from the time the negative sentence had been read ’ (Hasson and Gluksberg 2004:20) Prediction: –in processing up/down-associated ‘ negative ’ sentences, up/down visual imagery would be activated at 200ms and there would be a change in the simulation at 500ms
13 Method: replicate Bergen 2005 fixation cross (1s) Ss heard a sentence Either 200 ms or 500 ms delay Ss saw a circle or square on the screen: up, down, right, or left (for 200 msec) –Critical trials: up or down. –The same number of (non-critical) sentences followed by an object on the left or right Ss categorize visual object Subjects press circle (z) or square (x) A comprehension question was interleaved, to ensure listening for content
24 Result: Negated Noun Sentences Up: The rainbow didn ’ t fade Down: The ground didn ’ t shake “ The sky didn ’ t darken ” was eliminated due to having average means higher than 2.5 SD from the mean
28 Summary of Effect Verb 200msSignificant interaction 500msMarginal interaction Noun 200ms Significant interaction 500ms - Responses were faster when the sentence location coincided with the visual object location
29 Findings: Prediction 1 There was a significant interaction effect between up/down noun/verb location and object location Up/down nouns or verbs induced visual simulation even when embedded in negated sentences This thus supports our main hypothesis that negation induces mental simulation
30 Findings: Prediction 2 Same interaction effect at both 200ms and 500ms, which results from the counterfactual simulation We found an evidence for a simulation of the counterfactual situation shortly after processing of negative sentences We found no evidence for factual simulation We found no evidence of a shift in the simulation in negation processing
31 No evidence for factual simulation If the glass didn ’ t fall, … We didn ’ t give clear factual situation Without context, factual situation is relatively underspecified This lead to not clear factual simulation stage
32 Bergen 2005 vs This study interference Positive facilitation500 ms facilitation200 ms Negative These two studies showed that up/down visual imagery is activated in processing negated sentences as well as positive sentences with up/down nouns or verbs Interestingly, our interaction was in the opposite direction.
33 Bergen 2005 vs. This study (Interference vs. Facilitation) *** **
34 Discussion: Interference vs. Facilitation Interference and facilitation effects in imagery studies are both simulation effects. The two effects basically result from the use of the same neural structures to understand language and perform a perception Two factors (Kaschak et al. 2004) –Extent of Time overlap between the two processing involved (sentence processing and visual processing) –Extent of Integratability of the two tasks “the extent to which the perceptual stimulus (circle or square) can be integrated into the simulation constructed of the content of the sentence”
35 How the two factors work If sentence processing and visual processing are, –Sequential – Facilitation as a form of semantic priming –Simultaneous – Interference or Facilitation depending on Integratability Integratable – Facilitation Non-integratable – Interference Interference – perform two different (non-integratable) tasks at the same time Facilitation – perform two related (integratable) tasks at the same time
36 Interference in Bergen ’ s positive sentences Simultaneous processing (200ms time overlap) Less integratability in simulating positive sentences –The sky darkened –Positive sentence tells us exactly what happened. It is fully informative about the location to be simulated –Such detailed image construction is not readily integratable with a circle or a square visual processing –Such specifically simulated image blocks visual processing –Therefore, interference effect
37 Facilitation in our negative sentences Simultaneous processing (200ms overlap) Relatively more integratability involved in simulating negative sentence -The sky didn ’ t darken -A single negative sentence with no context is less informative (less specific or less detailed) in its conveyed meaning -Perceivers would simulate less well-defined location imagery and thus construct a less-detailed simulation -This less-detailed image construction and a circle or a square visual processing can be relatively integratable -Thus, this less-detailed simulated image doesn’t block visual processing -Therefore, facilitation effect
38 Summary of discussion Positive sentence processing –Fully informative –Well-defined and detailed image construction –Low integratability –Interference Negative sentence processing –Less informative –Less well-defined and less-detailed image construction –Makes the two tasks relatively integratable –Facilitation
39 Discussion: Verb effects significant only at 200ms * **
40 Weaker simulation effect for verbs at 500ms Scope of negation –The sky [didn’t darken] (Up-noun) –The glass [didn’t fall] (Down-verb) For nouns, the direction is not changed under negation: less subject variability For verbs, the direction is changed under negation: more subject variability This would make verbs exhibit relatively weaker simulation at 500ms
42 Summary Found an interaction effect of noun/verb direction with visual object recognition in comprehending negated intransitive sentences Negated sentences involve a less detailed simulation, causing facilitation Imagery is involved in the comprehension of negated sentences Found an evidence for a simulation of the counterfactual situation Found no evidence for factual simulation Found no evidence for a shift in the simulation of negation
43 Future Study How to capture a shift in the process of negation simulation? To capture the further clear counterfactual simulation, test sentences at an earlier time course (0ms) To capture the clear factual simulation, give a clear factual situation, since without context, the factual situation is relatively underspecified –The glass didn ’ t fall. It rose.
44 Conclusions Spatial imagery of varying degrees of detail is automatically or unconsciously engaged in language understanding Language about a visual scene induces visual simulation, even in negated sentences How we actually process negated sentences crucially depends upon the detailed semantics of sentential constituents, such as subject nouns and verbs Embodied view of meaning –“ [ … ] the core of our conceptual systems is directly grounded in perception, body movement, and experience of a physical and social character. ” (Lakoff 1987, p.xiv)
46 References BARSALOU, LAWRENCE W. 1999. Perceptual symbol systems. Behavioral and Brain Sciences, 22.577-660. BERGEN, BENJAMIN K., NARAYAN, SHWETA and FELDMAN, JEROME. 2003. Embodied verbal semantics: evidence from an image-verb matching task. Paper presented at Cognitive Science Conference.. GLENBERG, ARTHUR M. and KASCHAK, MICHAEL P. 2002. Grounding Language in Action. Psychonomic Bulletin & Review, 9.558-65. Hasson, Uri and Sam Glucksberg. 2004. Does Understanding Negation Entail Affirmation? An Examination of Negated Metaphors. manuscript. University of Chicago. Just, M. A., and P. A. Carpenter. 1976. Eye fixations and cognitive processes. Cognitive Psychology, 8: 441-480. Kaschak, Michael P, Rolf A. Zwaan, Mary Aveyard and Richard H. Yaxley. (in press). Perception of auditory motion affects language processing. Cognitive Science. Kaschak, Michael P., Carol J. Madden, David J. Therriault, Richard H. Yaxley, Mark Aveyard, Adrienne A. Blanchard & Rolf A. Zwaan. 2004. Perception of motion affects language processing. Cognition 94:B79-B89. Kaup, Barbara. 2001 Negation and its impact on the accessibility of text information. Memory & Cognition, 29, 860-967. KAUP, BARBARA and ZWANN, ROLF A. 2003. Effects of negation and situational presence on the accessibility of text information. Journal of Experimental Psychology: Learning, Memory, and Cognition, 29.439-46. Lindsay, Shane (2003) Visual priming of language comprehension. Masters Thesis. Department of Cognitive and Computing Sciences. University of Susses. MacDonald, Maryellen C and Just, Marcel A. 1989. Changes in activation levels with negation. Journal of Experimental Psychology: Learning, Memory and Cognition. Jul;15(4):633-42. NARAYAN, SHWETA, BERGEN, BENJAMIN K. and WEINBERG, ZACH. 2004. Embodied verbal semantics: Evidence from a lexical matching task. Paper presented at The Thirtieth Berkeley Linguistics Society Annual Meeting. RICHARDSON, DANIEL C., SPIVEY, MICHAEL J., BARSALOU, LAWRENCE W. and MCRAE, KEN. 2003. Spatial representations activated during real-time comprehension of verbs. Cognitive Science.767-80. ZWANN, ROLF A., STANFIELD, ROBERT A. and YAXLEY, RICHARD H. 2002. Language comprehenders mentally represent the shapes of objects. Psychological Science, 13.168-71. ZWANN, ROLF A. 1999. Embodied cognition, perceptual symbols, and situation models. Discourse Processes, 28.81-88.
48 Stimuli Down-Verb –The stone didn ’ t sink –The chair didn ’ t topple –The pipe didn ’ t drop –The glass didn ’ t fall –The cat didn ’ t descend Up-Verb –The mule didn ’ t climb –The lizard didn ’ t ascend –The cork didn ’ t rocket –The patient didn ’ t rise –The dolphin didn ’ t soar Down-Noun –The grass didn ’ t glisten –The submarine didn ’ t fire –The cellar didn ’ t flood –The ground didn ’ t shake –The shoe didn ’ t smell Up-Noun –The sky didn ’ t darken –The rainbow didn ’ t fade –The roof didn ’ t creak –The tree didn ’ t sway –The ceiling didn ’ t crack
49 NOUN RTs SameDifferent The cellar didn’t flood.NounDown631.00620.42 The grass didn’t glisten.NounDown659.68596.39 The ground didn’t shake.NounDown592.68535.95 The shoe didn’t smell.NounDown577.58687.50 The submarine didn’t fire.NounDown589.33657.51 The ceiling didn’t crack.NounUp615.32718.55 The rainbow didn’t fade.NounUp498.11704.59 The roof didn’t creak.NounUp475.05644.83 The tree didn’t sway.NounUp482.95750.37
50 VERB RTs SameDifferent The cat didn’t descend.LiteralDown602.91660.43 The chair didn’t topple.LiteralDown516.69512.38 The glass didn’t fall.LiteralDown477.27605.47 The pipe didn’t drop.LiteralDown467.45616.74 The stone didn’t sink.LiteralDown571.09477.73 The cork didn’t rocket.LiteralUp507.00451.55 The dolphin didn’t soar.LiteralUp452.67475.00 The lizard didn’t ascend.LiteralUp607.55590.51 The mule didn’t climb.LiteralUp444.95613.73 The patient didn’t rise.LiteralUp515.73502.09
51 Bergen ’ s findings interaction effect (interference) between noun/verb location and object location **