1. Details Regarding Stimuli
Integration of Gesture and Speech in Real-Time Reference Resolution
Ellen Campana†, Laura Silverman‡, Michael K. Tanenhaus†, and Loisa Bennetto‡
†Department of Brain and Cognitive Sciences
University of Rochester
‡ Department of Clinical and Social Sciences in Psychology
University of Rochester
Non-deictic manual gestures are ubiquitous in language production. People produce gesture even when their interlocuters cannot see them [1] and can experience difficulty speaking when their hands are restricted [5]. There is some consensus that the production system allows different aspects of a single message to be expressed simultaneously over, and distributed between, linguistic utterances and gestures [2]. However, researchers disagree about the role of gesture in comprehension; whether it is ignored [6], processed separately [4], used only when speakers are having difficulty [8], or immediately integrated with the content of the co-occurring speech [7]. The present experiment provides evidence in support of immediate integration between the two modalities.
Overview
In the example shown in the method section the actor in the center of the screen describes the contents of the top left-hand quadrant, saying “a mitten and a curved line with one loop” as she gestures the shape of the loop. In the speech-only version of this trial she says “a mitten and a curved line with one loop” without moving her hands
Details Regarding Stimuli
Note that in this example the point of disambiguation (POD) with respect to the speech alone occurs just after the start of the word “one.”
Conclusions & Future Work
The results from the present experiment demonstrate that listeners immediately integrate information from speech and co-occurring gesture to create a meaningful message as an utterance unfolds. In particular, the when descriptive gestures are present, listeners attend to relevant gestures and appear to process the speaker’s meaning even before it is fully articulated via speech.
Our future work involves extending the present methodology to examine gesture and speech comprehension in individuals with high-functioning autism and Asperger Syndrome. Individuals with autism have significant social and communicative impairments and specifically have difficulties interpreting the non-verbal behaviors of others, such as gaze and facial expressions. By examining speech and gesture integration, we can learn the extent to which difficulties with gesture comprehension may contribute to their overall social and communicative difficulties. Furthermore, studying the timing of speech and gesture integration may also illuminate whether individuals with autism have an underlying impairment in processing information across modalities, which may affect their functioning in other realms. The findings from this research have important treatment implications, and may also shed light on the role of multimodal integration in typical populations.
Method
Ten participants watched 14 videos of a woman describing simple shapes on a display in which the video was surrounded by four potential referents: the target, a speech competitor, a gesture competitor, and an unrelated foil. The task was to “click on the shape that the speaker was describing”
In half of the 14 videos the speaker used a natural combination of speech and gesture [3]. In the other half, the speaker’s hands remained in her lap. The stimuli were counterbalanced in two lists such that across participants each set of visual stimuli occurred with both speech+gesture videos and speech-only videos. The speech-only videos were directly modeled on the natural speech+gesture videos and contained an identical verbal description
[1] Alibali, M. W., Heath, D. C., and Myers, H. J. (2001). Effects of visibility between speaker and listener on gesture production: Some gestures are meant to be seen. Journal of Memory and Language, 44, 1-20.
[2] Alibali, M. W., Kita, S., and Young, A. J. (2000). Gesture and the process of speech production: We think, therefore we gesture. Language and Cognitive Processes, 15,
[3] Campana, E., Silverman, L., Tanenhaus, M. K., and Bennetto, L. (2004). Iconic gesture production in controlled referential domains. In Proceedings of the 26th Annual Meeting of the Cognitive Science Society. Chicago, IL.
[4] Goldin-Meadow, S., and Singer, M. A. (2003). From children’s hands to adults’ ears: Gesture’s role in the learning process. Developmental Psychology, 39,
[5] Graham, J. A., and Heywood, S. (1975). The effects of elimination of hand gestures and of codability on speech performance. European Journal of Social Psychology, 2,
[6] Krauss, R. M., Dushay, R. A., Chen, Y., and Rausher, F. (1995). The communicative value of conversational hand gestures. Journal of Experimental Social Psychology, 31,
[7] McNeill, D., Cassell, J., and McCullough, K.-E. (1994). Communicative effects of speech-mismatched gestures. Research on Language and Social Interaction, 27,
[8] Rauscher, F. H., Krauss, R. M. and Chen Y. (1996). Gesture, speech, and lexical access: The role of lexical movements on speech production. Psychological Science, 7,
References
Results
Speech+Gesture
0.1
0.2
0.3
0.4
0.5
0.6
S+G Target
S+G Speech
Competitor
S+G Foil 1 2 3
Participants are more likely to click on the correct target before reaching the POD in the gesture+speech condition, suggesting that gesture information speeds processing.
Participants are more likely to look at the speech competitor in the speech-only condition, suggesting that gesture information reduces competition.
Eye-movement graphs reveal that participants seem to experience longer-lasting competition from the speech competitor in the speech-only condition
Speech-only
0 ms
POD (2112 ms)
3000 ms
Speech+Gesture
We gratefully acknowledge the hard work of the undergraduate members our research group: Stephanie Packard, Hsi-Mei (Betty) Huang, Kathryn McNamara, Kelley Knock, and Sandra Vanegas.
0 ms
POD (2178 ms)
3000 ms