2. Intonationally marked contrast in instructed visual search: Intersective color vs. subsective scalar adjectives Shari R. Speer Ohio State University

3. Acknowledgements NSF BCS-018464 NSF BCS-0617609 NIDCD R01- DC007090 Ping Bai Allison Blodgett Laurie Maynell Mary Beckman Co-I, Kiwako Ito

4. Contrast Similarities and differences  comparisons Bolinger (1961) “… cases where one or more individual items are singled out from a larger (but limited) set as being true as regards some relationship whereas others in the same set are untrue…” Zeevat (2004) Contrastors (alternatives)… “must be obtainable from the actual utterance by substituting something else for the intonationally prominent constituent.”

5. Contrast Discourse coherence prerequisite for anaphora resolution Must be in focus domain Often accompanied by structural parallelism Matter of degree?

6. Accentuation for expressing contrast B (vs. A) accent (Bolinger, 1961; Jackendoff, 1972) L+H* (vs. H*) (Pierrehumbert, 1980; ToBI) Pierrehumbert & Hirschberg (1990) “the accented item - and not some alternative related item - should be mutually believed (p 296.)” L+H*: +AGREED, theme accent (Steedman, 2003)

7. Intonationally marked contrast in sentence/discourse processing Faster comprehension of short contrastive discourse when prominent accents were appropriately located (vs. inappropriate) (Bock & Mazella, 1983) Faster comprehension and higher acceptance of appropriately accented Q-A pairs (English pitch accent: Birch & Clifton, 1995; Japanese, Basque pitch range expansion/peak alignment: Ito, 2002). Faster phoneme monitoring when the contrastive entity was negated with prominent accent (vs. no accent). Intonationally marked narrow focus in negation generates contrast set (Davidson, 2001).

8. Accentuation shows immediate effects on instructed visual search (Dahan, Tanenhaus & Chambers, 2002) Prominent accent interpreted non-anaphorically, lack of prominent accent interpreted anaphorically. 1. “Put the candle/candy below the triangle.” 2. “NOW, put the CANDLE above the square.” “NOW, put the candle ABOVE THE SQUARE.” fixation to candy fixation to candle candle  “CANDLE” candle  “candy”

9. L+H* on a modifier evokes a contrast set (Ito & Speer, 2008) Manipulated pitch accent type on a modifier adjective or modified noun L+H*: Evokes a contrast set that contains alternative related item(s) (Pierrehumbert & Hirschberg, 1990) GREENballGREENball blue ball

10. Holiday Tree task (Ito & Speer, 2008) Eyemovements monitored as naïve participants followed recorded audio instruction to decorate Christmas trees. (ASL e5000) at 60Hz.

11. Visual Stimuli (Ito & Speer, 2008) 44 ornaments displayed by type on a grid with 11 cells (8 target + 3 filler)

12. Sound Stimuli (Ito & Speer, 2008) Accentual patterns of stimuli were selected based on the analysis of spontaneous speech in a similar task (Ito, Speer & Beckman, 2003). A trained female phonetician produced stimuli with the intended pitch accent patterns. Adjective-noun pairs were presented in discourse context. Recorded 44.1KHz, 16bits All stimuli were ToBI-annotated by two native speakers of American English. Items on which they did not agree were re-recorded.

13. Experiment 1: Stimuli Felicitous L+H* Patterns green onion ORANGE onion L+H* no accent brown ball brown ANGEL H* L+H* Infelicitous L+H* Patterns gray stockingbrown STOCKING H* L+H* orange candy ORANGE onion L+H* no accent

14. Experiment 1 Results: Felicitous vs. Infelicitous L+H* (Ito & Speer, 2008)

15. Experiment 1: Summary Felicitous L+H* on the color term in contrastive environments (green drum, BLUE drum) facilitated visual search compared to infelicitous L+H* on non-contrastive noun (green drum, blue DRUM). Felicitous L+H* on the object noun in contrastive environments (blue egg, blue DRUM) showed a smaller advantage over infelicitous L+H* on the adjective (blue egg, BLUE drum). Listeners ‘tune’ their sensitivity to contrastive accent on the basis of the visual task?

16. Experiment 2: stimuli Does L+H* on the contrastive adjective lead to anticipatory eye-movement compared to H*? e.g. brown drum --> RED/red drum Does infelicitous L+H* on the color adjective lead to an expectation strong enough to lead a “garden- path” in eye-movement? Does infelicitous L+H* on the color adjective lead to an expectation strong enough to lead a “garden- path” in eye-movement? e.g. red onion --> GREEN drum

17. Experiment 2 Results: Felicitous/Infelicitous L+H* vs. H*

18. Experiment 2 results: “Garden-path” due to L+H*

19. Experiment 2 results: Refixation of previous cell w/out L+H*?

20. The effect of infelicitous L+H*

21. Ito & Speer 2008: Summary For cases with repetition of the immediately preceding noun, L+H* showed a processing advantage as compared to H* on the adjective (brown drum --> RED/red drum) For cases without repetition of the noun, infelicitous L+H* led to incorrect anticipatory fixations to the most-recently mentioned ornament type. significant delay in fixations to the real target NO incorrect anticipatory fixations were observed when non-repeated targets were felicitously presented with [H* !H*] NO incorrect anticipatory fixations were observed when non-repeated targets were felicitously presented with [H* !H*]

22. Discussion  Dahan et al results showed that L+H* accent led to the expectation of contrast. Contrastive accent on a (potentially) repeated word led listeners to fixate the object type that had *not* just been mentioned.  Results here show contrast effects from L+H* accent on the modifier that preceded the noun. L+H* led listeners to restrict the set of expected referents to those of the object type that *had* been most recently mentioned.

23. Discussion Evidence suggests that L+H* evokes generation of a contrast set in the listener’s discourse representation. The set is based on the accented word interpreted within the structure of the utterance. Here, contrastive accent on the adjective restricted expectations about the coming head noun to the set of objects specified by the immediately preceding noun. Facilitative anticipatory fixations and prosodic ‘garden- path’ effects due to prominence and have also been shown for German (Weber et al, 2006) and Japanese (Ito et al, under review).

24. Remaining questions In Ito&Speer 2008, rapid integration of contrastive L+H* involved refixation of the immediately preceding ornament cell. Were facilitation effects of L+H* dependent on the visual layout, where contrastive trials required re-fixation of a ‘just-visited’ ornament cell? Was predictive use of L+H* in the ‘garden-pathing’ trials dependent on the organization of the ornament array by objects (nouns)?  Are primary color adjectives, which describe a context- independent distinction between modified objects, more likely to require intonational contrast than other types of adjectives (e.g. scalars)?

25. Followup studies: We used adjective-sorted arrays, where contrastive sequences (e.g., red ball  green ball, small ball  medium ball) require fixations to a new cell (instead of returning fixations). We compared 2 adjective types: Color (context-independent, intersective) Size (scalar,subsective) Because relative size terms are inherently contrastive and context- dependent, prominent pitch accent may be redundant as a cue to contrast.

26. Exp3 Visual stimuli - Colors

27. Exp4 Visual stimuli - Sizes

28. Sound stimuli Articles rather than determiners, e.g. ‘Hang a green ball,’ ‘Hang a large ball,’ as there were multiple like ornaments. Same native English ToBI-trained speaker Same procedures for annotation, presentation and eyemovement monitoring Four conditions: Contrastive vs. non-contrastive H* !H* vs. L+H* no accent

29. Anticipatory effect w/out ‘refixation’ Exp3, color adjectives: “Hang a yellow/YELLOW star” Exp4, scalar adjectives: “Hang a large/LARGE tree”

30. Garden-path effects Color: fixations to green tree hearing ‘GREEN ball’ Scalar: fixations to large tree hearing “LARGE ball”

32. Summary: Fixation proportions L+H* facilitates eyemovements to the targets for both color and size adjectives in contrastive sequences. Visually more complex size-sorted boards led to slower eye movements to the target than color- sorted boards. L+H* led to more frequent fixations to the incorrect targets for both color and size adjectives in non- contrastive sequences. Non-prominent size adjective (with H*) did not lead to looks to contrastive cells, i.e., size adjectives are not automatically interpreted contrastively.

33.  Prominent pitch accent (L+H*) on a pre-nominal modifier evokes a contrast set and facilitates reference resolution with both intersective color adjectives and subsective scalar adjectives.  L+H* evokes the anticipation of contrast, and thus may lead to intonation-based “garden-pathing” regardless of the adjective type used.  The interpretation of pre-nominal adjectives depends on their efficacies within each referential environment rather than on their inherent semantics. Intonational prominence is used immediately and predictively for reference resolution regardless of the modifier type. Summary

34. Current directions L+H* vs. H* controversy:  Phonetic analyses of stimuli from 4 comprehension experiments underway  Correlation of first fixation latency and acoustic measures  Parallel analysis of spontaneous speech from production version of holiday tree task

35. Issue on categorical distinction: Is L+H* a kind of H*? Color adj with H*  weaker interpretation of contrast? Categorical interpretation but non-categorical perception (Ladd & Morton, 1997) Great overlap between H* and L+H* in pitch scale, shape & alignment (Tayler, 2000) Frequent uncertainty between H* and L+H* in expert-ToBI labeling (Brugos et al. 2008). More frequent use of H* than L+H* to mention contrastive discourse entities in story continuation (Metusalem & Ito, 2008). 1.How can we define categories of prosodic prominence? 2.What factors contribute to recognition & processing of contrastiveness?

36. Step-wise Multiple Linear Regressions: Predicting First fixation latency: L+H* Step Var tp 1 subject-2.91<.01 2 F0 diff-2.76<.01 Total R 2 :.051 F(2,256) = 7.96, p<.001 Step Var tp 1 F0 diff-5.66<.001 2 F0 peak4.44<.001 3 color-2.57<.05 4 size2.53<.05 Total R 2 :.245 F(2,296) = 25.38, p<.0001 EXP1: Color-sortedEXP2: Size-sorted

37. Step-wise Multiple Regressions: Predicting First fixation latency: H* Step Var tp 1 size-4.78<.0001 2 color4.03<.0001 3 subject-2.97<.01 Total R 2 :.108 F(3,257) = 11.59, p<.0001 Step Var tp 1 size-2.44<.05 2 F0 diff-3.21<.01 3 color-2.64<.01 Total R 2 :.088 F(3,293) = 10.57, p<.0001 EXP1: Color-sortedEXP2: Size-sorted

38. Further exploration needed: Different dependent variable? Additional phonetic variables:  Normalized F0 scaling  Peak, ‘low elbow’ alignment from vowel onset  Word/syllable intensity  Vowel quality (F1, F2, breathiness, etc.)  Following noun’s phonetic status F0 prominence intensity vowel quality, etc.