2. Intonational prominence: Production in dialogue, comprehension during visual search Shari R. Speer (with much input from Kiwako Ito, Co-I) Ohio State University Linguistics

3. Intonation & Pragmatic Status English Pitch Accent Distinctive tonal movement associated with stressed syllable.  perceivable prosodic prominence Local targets L and H (Pierrehumbert 1980) H*, (H+!H*), L*, L+H*, L*+H (a/c current ToBI guideline)

4. Pitch Accent Types & Pragmatics Pierrehumbert & Hirschburg, 1990 Compositional tune meaning PA typemeaning H*New items introduced to a discourse, added to mutual belief space. L*Already part of mutual beliefs. Salience-without-predication L*+HUncertainty, lack of speaker commitment L+H*Contrast with some alternative related item

5. Pierrehumbert & Hirschberg, 1990 Compositional tune meaning PA typesmeaning H*New items introduced to a discourse, added to mutual belief space. L*Already part of mutual beliefs. Salience-without-predication L*+HUncertainty, lack of speaker commitment L+H*Contrast with some alternative related item Pitch Accent Types & Pragmatics

6. Categorical distinction between H* and L+H* F0 minima preceding the accentual peak should be stable for L+H*, but not for H* Perception A clear categorical boundary should be observed in both identification and discrimination task along with H* height continuum. A clear categorical boundary should be observed in both identification and discrimination task along with H* height continuum. Tonal shape H* L+H* flat or subtle riseclear rise

7. Categorical distinction between H* and L+H* F0 minima between two successive H* are consistently aligned to the syllable onset (Ladd & Schepman, 2003). F0 minima between two successive H* are consistently aligned to the syllable onset (Ladd & Schepman, 2003). Listeners use the F0 minima alignment cue to distinguish ambiguous names, e.g. Norma Nelson vs. Norman Elson (Ladd & Schepman, 2003). Listeners use the F0 minima alignment cue to distinguish ambiguous names, e.g. Norma Nelson vs. Norman Elson (Ladd & Schepman, 2003). Although context identification tasks show categorical distinction between non-emphatic vs. emphatic contexts, listeners perform poorly in discrimination tasks along the accentual peak continuum. (Ladd & Morton, 1997) Although context identification tasks show categorical distinction between non-emphatic vs. emphatic contexts, listeners perform poorly in discrimination tasks along the accentual peak continuum. (Ladd & Morton, 1997)

8. Categorical distinction between H* and L+H* Testing tonal shape (Arvaniti & Garding, to appear) The height of peak and its alignment vary according to the degree of emphasis. The height of peak and its alignment vary according to the degree of emphasis. Dialectal differences: Dialectal differences: Contrast between H* & L+H* preserves in Southern Californian but not in Minnesotan. H* L+H* “gradient variation between two ends of a continuum” (Ladd & Schepman, 2003)

9. Analyzing Unscripted Speech Task: Give instructions for tree decoration Task: Give instructions for tree decoration Monitor display with a picture of ornament with its tag (e.g. blue ball) and a picture of a tree, indicating the location of the ornament. Monitor display with a picture of ornament with its tag (e.g. blue ball) and a picture of a tree, indicating the location of the ornament. Newness, Givenness & Contrastiveness manipulated by the order of mention. Newness, Givenness & Contrastiveness manipulated by the order of mention.

10. * * * * * 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 * * * * * * ** * orange candy

11. * * * * * 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 * * * * * * ** ** blue candy

12. * * * * * green candy 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 * * * * * * ** ***

13. * * * * * 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 * * * * * * ** * orange candy

14. * * * * * 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 * * * * * * ** ** blue candy

15. * * * * * green candy 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 * * * * * * ** ***

16. Example pitch traces

17. Accentuation Proportion NEW/GIVEN # OF TRIALS ADJECTIVENOUN NewNew31.94.84 NewGiven32.84.50 GivenNew32.91.81 GivenGiven96.89.67

18. Accentuation Proportion NEW/GIVEN # OF TRIALS ADJECTIVENOUN NewNew31.94.84 NewGiven32.84.50 GivenNew32.91.81 GivenGiven96.89.67

19. L+H* distribution NEW/GIVENPRAGMATICSTATUS ADJECTVENOUN # OF TRIALS L+H* L+H* New Contrastive26.4631.19 Non- contrastive 37.0333.06 Given Contrastive37.5133.18 91.0095.00

20. Is intonation informative? Effect of Intonation on visual search Listeners use pitch accent to single out the object referred to (Dahan et al. 2002) 1. “Put the candle/candy below the triangle.” 2. “NOW, put the CANDLE above the square.” “NOW, put the candle ABOVE THE SQUARE.” “NOW, put the candle ABOVE THE SQUARE.” fixation to candy fixation to candle candle  “CANDLE” candle  “candy”

21. Research Questions How does intonational prominence (L+H*) on a modifier adjective affect visual search? L+H*: Evokes a contrast set that contains alternative related item(s) (Pierrehumbert & Hirschberg, 1990) BLUE ball

22. Research Question How does intonational prominence (L+H*) on a modifier adjective affect visual search? L+H*: Evokes a contrast set that contains alternative related item(s) (Pierrehumbert & Hirschberg, 1990) BLUE ball green ball

23. Why track the eyes? Eyemovements can be traced: Eyemovements can be traced: during natural conversation during natural conversation for both speaker & listener for both speaker & listener to investigate the online language processing (Tanenhaus and Trueswell,2005) to investigate the online language processing (Tanenhaus and Trueswell,2005) Accentual prominence evokes reference to an alternative discourse entity (Dahan et al., 2002). Accentual prominence evokes reference to an alternative discourse entity (Dahan et al., 2002). Listeners fixate target objects 200ms from the onset of auditory stimuli during simple visual search (Allopenna, et al. 1998). As much as 150ms of this time is used to program and execute a ballistic saccade (Matin, Shao, & Boff, 1993). Listeners fixate target objects 200ms from the onset of auditory stimuli during simple visual search (Allopenna, et al. 1998). As much as 150ms of this time is used to program and execute a ballistic saccade (Matin, Shao, & Boff, 1993).

24. Experiment 1:Procedure Subjects followed recorded audio instruction to decorate Christmas trees. Subjects followed recorded audio instruction to decorate Christmas trees. (ASL e5000) at 60Hz. (ASL e5000) at 60Hz.

25. Experiment 1: Visual Stimuli Ornaments displayed on a grid with 11 cells (8 target + 3 filler) Ornaments displayed on a grid with 11 cells (8 target + 3 filler)

26. Experiment 1: Sound Stimuli Accentual patterns of stimuli were selected based on the analysis of spontaneous speech (Ito, Speer & Beckman, 2003).A trained female phonetician produced stimuli with the intended Pitch accent patterns. Accentual patterns of stimuli were selected based on the analysis of spontaneous speech (Ito, Speer & Beckman, 2003).A trained female phonetician produced stimuli with the intended Pitch accent patterns. Recorded 44.1KHz, 16bits Recorded 44.1KHz, 16bits F0 extracted with 10ms window F0 extracted with 10ms window All stimuli were ToBI transcribed by two native speakers of American English. All stimuli were ToBI transcribed by two native speakers of American English.

27. Experiment 1: Stimuli Adjective NounAccent Pattern New NewH* !H* New GivenH* !H* Given GivenH* !H* GivenNewH*H* NewCGivenL+H* no accent GivenNewCH*L+H*

28. Experiment 1: Stimuli Felicitous Patterns Felicitous Patterns green onion ORANGE onion L+H* no accent L+H* no accent brown ball brown ANGEL brown ball brown ANGEL H* L+H* H* L+H* Infelicitous Patterns Infelicitous Patterns gray stockingbrown STOCKING H* L+H* H* L+H* orange candy ORANGE onion L+H* no accent L+H* no accent

29. Experiment 1: Stimuli All stimuli were ToBI-transcribed by two native speakers of American English. Ave. 300 HzAve. 299 Hz

30. Experiment 1: Results

32. Experiment 1: Given/New Objects named by given nouns were fixated sooner and more often than objects named by new nouns. Objects named by given nouns were fixated sooner and more often than objects named by new nouns. No additional advantage came from having both a given adjective and a given noun (compared to difference for given noun alone). No additional advantage came from having both a given adjective and a given noun (compared to difference for given noun alone). Visual search facilitated by previous mention of object nouns, and previous successful search for the referent in object-organized array. Visual search facilitated by previous mention of object nouns, and previous successful search for the referent in object-organized array.

33. Experiment 1: Results green onion  ORANGE onion [L+H* noacc]  orange ONION [H* L+H*]

36. Experiment 1: Results brown drum  brown ANGEL  BROWN angel

37. Experiment 1: Felicitous vs. Infelicitous Felicitous L+H* on the color term in contrastive environments (blue egg, SILver egg) facilitated visual search compared to infelicitous L+H* on non-contrastive noun (grey candy, gold CANdy). Felicitous L+H* on the color term in contrastive environments (blue egg, SILver egg) facilitated visual search compared to infelicitous L+H* on non-contrastive noun (grey candy, gold CANdy). Felicitous L+H* on the object noun in contrastive environments (blue egg, blue CANdy) showed no advantage over infelictous L+H* on the adjective (blue egg, BLUE candy). Felicitous L+H* on the object noun in contrastive environments (blue egg, blue CANdy) showed no advantage over infelictous L+H* on the adjective (blue egg, BLUE candy). Listeners ‘tune’ their sensitivity to contrastive accent on the basis of the visual task? Listeners ‘tune’ their sensitivity to contrastive accent on the basis of the visual task?

38. Experiment 2: Questions Does L+H* on the contrastive adjective lead to anticipatory eye-movement compared to H*? Does L+H* on the contrastive adjective lead to anticipatory eye-movement compared to H*? e.g. brown drum --> RED/red drum Does infelicitous L+H* on the color adjective lead to an expectation strong enough to lead a garden- path in eye-movement? Does infelicitous L+H* on the color adjective lead to an expectation strong enough to lead a garden- path in eye-movement? e.g. red onion --> GREEN drum

39. Experiment 2 Procedure Same as Exp 1 Subjects followed audio instruction to decorate Christmas trees. Subjects followed audio instruction to decorate Christmas trees. Ornaments displayed on a grid with 8 target + 3 filler cells Ornaments displayed on a grid with 8 target + 3 filler cells Eye-movements monitoring at 60Hz. Eye-movements monitoring at 60Hz.Stimuli Prepared in the same way as Exp1 Prepared in the same way as Exp1 Same speaker and ToBI transcribers Same speaker and ToBI transcribers

40. Experiment 2: Results brown drum  RED drum [L+H* no acc]  red drum [H* !H*]

41. Experiment 2: Results red onion  green drum  GREEN drum

42. Experiment 2: Results red onion  GREEN drum [L+H* no acc] onion drum

43. The effect of infelicitous L+H*

44. Experiment 2: Results red onion  GREEN drum [L+H* no acc] onion drum

45. Experiment 2: Results red onion  green drum onion drum

46. Experiment 2: Summary For cases with repetition of the immediately preceding noun, L+H* showed a processing advantage as compared to H* on the adjective (brown drum --> RED/red drum) For cases with repetition of the immediately preceding noun, L+H* showed a processing advantage as compared to H* on the adjective (brown drum --> RED/red drum) For cases without repetition of the noun, infelicitous L+H* led to incorrect anticipatory fixations to the most-recently mentioned ornament type. For cases without repetition of the noun, infelicitous L+H* led to incorrect anticipatory fixations to the most-recently mentioned ornament type. significant delay in fixations to the real target

47. NO incorrect anticipatory fixations were observed when non-repeated targets were felicitously presented with [H* !H*] NO incorrect anticipatory fixations were observed when non-repeated targets were felicitously presented with [H* !H*] Experiment 2: Summary

48. Discussion Dahan et al results showed that L+H* accent led to the expectation of contrast. Contrastive accent led listeners to fixate the object type that had *not* just been mentioned. Dahan et al results showed that L+H* accent led to the expectation of contrast. Contrastive accent led listeners to fixate the object type that had *not* just been mentioned. Results here show contrast effects from L+H* accent on the modifier that preceded the noun. L+H* led listeners to restrict the set of expected referents to those of the object type that *had* been most recently mentioned. Results here show contrast effects from L+H* accent on the modifier that preceded the noun. L+H* led listeners to restrict the set of expected referents to those of the object type that *had* been most recently mentioned. This evidence suggests that L+H* evokes generation of a contrast set in the discourse representation of the listener. The set is based on the accented word interpreted within the structure of the utterance. Here, contrastive accent on the adjective restricted expectations about the coming head noun to the set of objects specified by the immediately preceding noun.

49. Conclusions so far Naïve speakers predictably used L+H* accents to convey contrast when directing a visual search during spontaneous dialogue. Naïve speakers predictably used L+H* accents to convey contrast when directing a visual search during spontaneous dialogue. Eyemovement latencies and fixation probabilities showed listeners’ immediate use of pitch accent information during discourse comprehension. Eyemovement latencies and fixation probabilities showed listeners’ immediate use of pitch accent information during discourse comprehension. Listeners were sensitive to the relevance of intonational cues for determining the intended referent during visual search. Listeners were sensitive to the relevance of intonational cues for determining the intended referent during visual search. L+H* accents on adjectives evoked an immediate interpretation of contrast, which led to anticipatory fixations to the most-recently mentioned object type. L+H* accents on adjectives evoked an immediate interpretation of contrast, which led to anticipatory fixations to the most-recently mentioned object type. These effects were not due to simple salience of the L+H* accented item, nor to word duration effects. These effects were not due to simple salience of the L+H* accented item, nor to word duration effects.

50. Additional studies “Christmas Tree” production and Eyemovement studies with Japanese speakers “Christmas Tree” production and Eyemovement studies with Japanese speakers Further investigation of discourse structure and the distribution of pitch accent type in English Further investigation of discourse structure and the distribution of pitch accent type in English Additional studies examining other aspects of intonational effects on discourse structure, e.g. Additional studies examining other aspects of intonational effects on discourse structure, e.g. Deaccentuation (reduction, deletion, pronouns) in production and comprehension Deaccentuation (reduction, deletion, pronouns) in production and comprehension

51. THANK YOU! Acknowledgements: Ping Bai Allison Blodgett Laurie Maynell Mary Beckman NSF BCS-018464 NIDCD R01-DC007090

52. Experiment 1: Pre-fixation data, Felicitous/Infelicitous

53. Experiment 2: Pre-fixation data, Felicitous/Infelicitous

54. Experiment 1: Additional findings Overall, shorter latencies for trials with immediately repeated object nouns Overall, shorter latencies for trials with immediately repeated object nouns Overall, shorter latencies for L+H* vs. H* trials Overall, shorter latencies for L+H* vs. H* trials Comparisons for ‘mismatch’ cases are between felicitous and infelicitous L+H* in different utterance positions Comparisons for ‘mismatch’ cases are between felicitous and infelicitous L+H* in different utterance positions Additional comparison needed: Additional comparison needed: For trials with repeated nouns For trials with repeated nouns Felicitous [L+H* no accent] vs. Infelicitous [H*!H*] For trials with non-repeated nouns For trials with non-repeated nouns Felicitous [H*!H*] vs. Infelicitous [L+H* no accent] For trials with L+H* in the same word position For trials with L+H* in the same word position Felicitous L+H* no accent vs. Infelicitous L+H* noaccent