1. “Downstepped contours in the given/new distinction”
On the Role of Prosody in Structuring Discourse October 5, Berlin, Germany
“Downstepped contours in the given/new distinction”
Agustín Gravano
Spoken Language Processing Group
Columbia University, New York

2. Participants in this project
Columbia University (New York) Julia Hirschberg Stefan Benus Agustín Gravano
Northwestern University (Chicago) Gregory Ward Elisa Sneed
Agustín Gravano - Columbia University

3. Boston Directions Corpus
Introduction
ToBI
Discourse structure (Grosz & Sidner ’86)
Information status (Prince ’92)
Meaning of intonational contours
The downstepped contours
Boston Directions Corpus
Description of the corpus
Downstep and discourse structure
Downstep and information status
Games Project
Ongoing and future research
Agustín Gravano - Columbia University

4. Boston Directions Corpus
Introduction
ToBI
Discourse structure (Grosz & Sidner ’86)
Information status (Prince ’92)
Meaning of intonational contours
The downstepped contours
Boston Directions Corpus
Description of the corpus
Downstep and discourse structure
Downstep and information status
Games Project
Ongoing and future research
Theoretical Basis
Agustín Gravano - Columbia University

5. To(nes and)B(reak)I(ndices)
Prosody annotation convention.
Two tones: H and L, which may be combined (e.g. H+L)
Devised originally for Standard American English, but ToBI standards also proposed for Japanese, German, Italian, Spanish, British, Australian English,....
4 tiers:
orthographic tier: words
break-index tier: degrees of junction
tonal tier: pitch accents, phrase accents, boundary tones
miscellaneous tier: disfluencies, non-speech sounds, etc.
Agustín Gravano - Columbia University

6. Discourse Structure (G&S ’86)
Series of discourse segments, defined in terms of the speaker’s intentions: the discourse segment purpose (DSP).
Let a, b: DSP,
a satisfaction-precedes b iff a must first be achieved in order for b to succeed;
a dominates b iff fulfilling b partly fulfills a.
Barbara Grosz & Candace Sidner, “Attention, intentions, and the structure of discourse.” Computational Linguistics 12(3):
Agustín Gravano - Columbia University

7. Information Status (Prince ’92)
Discourse {
Given
New
Hearer
Inferrable
Ellen Prince, “The ZPG letter: Subjects, definiteness, and information-status.” In Discourse Description: Diverse Analyses of a Fund Raising Text, S. Thompson & W. Mann (eds.), , Philadelphia: John Benjamins B.V.
Agustín Gravano - Columbia University

8. Multiple “meanings” of intonational contours
“Declarative” contours (H* L- L%)
Statements
Wh-questions
Rise-fall-rise contours (L*+H L- H%)
Uncertainty
Incredulity
H* Downstepped contours (H* (!H*)+ L- (L%|H%)?)
Topic beginnings or endings?
“Given” information?
Intonational contours are ‘overloaded’ wrt meanings that they can convey and conditions under which they are appropriately used.
Agustín Gravano - Columbia University

9. Example: H* !H* !H* !H* L-H%
Agustín Gravano - Columbia University

10. Understanding the multiple uses of contours is useful and interesting
In most TTS systems
‘Standard’ declarative (H* L- L%) contour over-used
‘Given’ information deaccented too often
The H* (!H*)+ L- (L%|H%)? contours might be used instead, if they are appropriate
Agustín Gravano - Columbia University

11. H* (!H*)+ L- (L%|H%)? in Standard American English
Topic structure markers (Pierrehumbert & Hirschberg ’90)
Beginning and ending of topics
Professorial tone
Givenness (Hirschberg & Pierrehumbert ’86, Ladd ’96, Dahan et al ’02)
“This material should already be familiar to you.”
Alternates with deaccenting – when?
Agustín Gravano - Columbia University

12. Boston Directions Corpus
Introduction
ToBI
Discourse structure (Grosz & Sidner ’86)
Information status (Prince ’92)
Meaning of intonational contours
The downstepped contours
Boston Directions Corpus
Description of the corpus
Downstep and discourse structure
Downstep and information status
Games Project
Ongoing and future research
Agustín Gravano - Columbia University

13. Boston Directions Corpus
Introduction
ToBi
Discourse structure (Grosz & Sidner ’86)
Information status (Prince ’92)
Meaning of intonational contours
The downstepped contours
Boston Directions Corpus
Description of the corpus
Downstep and discourse structure
Downstep and information status
Games Project
Ongoing and future research
Agustín Gravano - Columbia University

14. Boston Directions Corpus
4 speakers
9 increasingly complex direction-giving tasks
Spontaneous speech transcribed and speakers returned and read
~67m spon; ~50m read

15. Boston Directions Corpus
first
enter the Harvard Square T stop
and buy a token
then
proceed to get on the
inbound um
Red Line
uh subway
and
take the subway
from Harvard Square
to Central Square
and then to Kendall Square
then get off the T
Agustín Gravano - Columbia University

16. BDC - Discourse Structure
first
enter the Harvard Square T stop
and buy a token
then
proceed to get on the
inbound um
Red Line
uh subway
and
take the subway
from Harvard Square
to Central Square
and then to Kendall Square
then get off the T
Agustín Gravano - Columbia University

17. BDC - Information Status
first
enter the Harvard Square T stop
and buy a token
then
proceed to get on the
inbound um
Red Line
uh subway
and
take the subway
from Harvard Square
to Central Square
and then to Kendall Square
then get off the T
Discourse Given
Agustín Gravano - Columbia University

18. BDC - Information Status
first
enter the Harvard Square T stop
and buy a token
then
proceed to get on the
inbound um
Red Line
uh subway
and
take the subway
from Harvard Square
to Central Square
and then to Kendall Square
then get off the T
Hearer Given
Hearer Inferrable
Agustín Gravano - Columbia University

19. BDC - DS Contours first enter the Harvard Square T stop
and buy a token
then
proceed to get on the
inbound um
Red Line
uh subway
and
take the subway
from Harvard Square
to Central Square
and then to Kendall Square
then get off the T
Agustín Gravano - Columbia University

20. Downstep and Discourse Structure
Distribution of use of DS contours for signaling discourse structure?
How frequently is discourse structure conveyed using DS contours?
Does this differ by speaking style (read vs. spontaneous speech)?
Is there notable speaker variation in either of these?
When speakers use DS contours, how often do they use them to begin or end a discourse (topic) segment?
When speakers begin or end a discourse (topic) segment, how often do they use a DS contour?
Agustín Gravano - Columbia University

21. Use of DS contours for discourse position
Spontaneous:
Contour
Seg Beg
Seg Final
Total
H* (!H*)+ L- (L%,H%)?
88 (18%)
196 (40%)
488
Read:
Contour
Seg Beg
Seg Final
Total
H* (!H*)+ L- (L%,H%)?
131 (29%)
195 (43%)
451
Agustín Gravano - Columbia University

22. Discourse position conveyed using DS contours
Spontaneous:
Contour
Seg Beg
Seg Final
H* (!H*)+ L- (L%,H%)?
88 (11%)
196 (28%)
Total
825 (100%)
693 (100%)
Read:
Contour
Seg Beg
Seg Final
H* (!H*)+ L- (L%,H%)?
131 (18%)
195 (31%)
Total
721 (100%)
635 (100%)
Agustín Gravano - Columbia University

23. Speaker variability
We found high variability (both in spontaneous and read speech) in:
Overall use of DS contours
Distribution of use of DS contours
Frequency with which discourse structure is conveyed using DS contours
Only exception:
Speakers employ ~40% or more of their DS contours over Segment Final phrases.
Agustín Gravano - Columbia University

24. Downstep and Information Status
Are DS contours used over given information, alternating with a deaccenting strategy?
If so, when do speakers choose one strategy over another?
Information status in the BDC data:
at the NP level (both discourse g/n and hearer g/i/n status),
at the word level (discourse g/n status for individual lexical items).
Smaller corpus: only spontaneous data labeled.
Agustín Gravano - Columbia University

25. Downstep and Information Status
Hearer Given
Hearer Inferrable
Hearer New
Discourse Given
Discourse New
All deacc
52 (5%)
6 (2%)
3 (2%)
46 (8%)
15 (2%)
Some accent DS
416 (39%)
200 (49%)
58 (45%)
261 (44%)
413 (44%)
Other DS
48 (5%)
25 (6%)
12 (9%)
32 (5%)
53 (6%)
Other
540 (51%)
175 (43%)
57 (44%)
257 (43%)
469 (49%)
Total
1056 (100%)
406 (100%)
130 (100%)
596 (100%)
950 (100%)
Spontaneous productions only.
Agustín Gravano - Columbia University

26. Downstep and Information Status
Hearer Given
Hearer Inferrable
Hearer New
Discourse Given
Discourse New
All deacc
45 (8%)
3 (4%)
0 (0%)
44 (8%)
4 (4%)
Some accent DS
260 (45%)
38 (54%)
3 (33%)
251 (45%)
50 (52%)
Other DS
28 (5%)
2 (3%)
2 (22%)
28 (5%)
Other
244 (42%)
27 (39%)
4 (44%)
237 (42%)
38 (40%)
Total
577 (100%)
70 (100%)
9 (100%)
560 (100%)
96 (100%)
Spon - Only NPs for which all lexical elements are Given.
Agustín Gravano - Columbia University

27. Downstep and Information Status
DS contours clearly dominate Hearer-Inferrables.
DS contours are commonly used over Given information.
Little evidence from this study that information status is a major predictor of the use of DS contours: equally likely to be used over New NPs.
Agustín Gravano - Columbia University

28. Boston Directions Corpus
Introduction
ToBI
Discourse structure (Grosz & Sidner ’86)
Information status (Prince ’92)
Meaning of intonational contours
The downstepped contours
Boston Directions Corpus
Description of the corpus
Downstep and discourse structure
Downstep and information status
Games Project
Ongoing and future research
Agustín Gravano - Columbia University

29. Boston Directions Corpus
Introduction
ToBI
Discourse structure (Grosz & Sidner ’86)
Information status (Prince ’92)
Meaning of intonational contours
The downstepped contours
Boston Directions Corpus
Description of the corpus
Downstep and discourse structure
Downstep and information status
Games Project
Ongoing and future research
Agustín Gravano - Columbia University

30. Games Project - Goal
Elicit a corpus of spontaneous dialogue containing:
given and new NPs
topic segmentation data
Agustín Gravano - Columbia University

31. Games Project - Design Session: 3 collaborative computer games.
2 players, each with an electronic game board.
Unrestricted speech.
No visual contact between subjects.
Subjects were paid a fixed amount of money, plus a bonus based on their performance.
Each subject participated in 2 sessions with different partners and on different days.
Agustín Gravano - Columbia University

32. PLAYER 1 “DESCRIBER”  PLAYER 2 “SEARCHER” 
Game # 1
PLAYER 1 “DESCRIBER” 
PLAYER 2 “SEARCHER” 
Agustín Gravano - Columbia University

33. PLAYER 1 “DESCRIBER”  PLAYER 2 “SEARCHER” 
Game # 2
PLAYER 1 “DESCRIBER” 
PLAYER 2 “SEARCHER” 
Agustín Gravano - Columbia University

34. Game # 3
PLAYER 1 “DESCRIBER” 
PLAYER 2 “SEARCHER” 

35. Games Project - Design
Study the relation between the choice of intonational contours and:
givenness status of NPs
syntactic position of NPs
complexity of NPs
proportion of given lexical elements in new NPs
discourse structure
Agustín Gravano - Columbia University

36. Games Project - Design How? Pretests Games 1 & 2: Game 3:
Cards have increasingly more features, increasing the complexity of NPs
Some features appear more frequently, becoming “given”.
Features appear in different sizes.
Game 3:
Subject  blinking/target image.
Objects  images surrounding the target image.
Pretests
Agustín Gravano - Columbia University

37. Games Project - Corpus Corpus:
Recorded in a sound-proof booth at Columbia’s Speech Lab in October 2004.
12 sessions.
~20 hours of spontaneous speech.
Fluent dialogues, each game with very different characteristics.
All dialogues have already been transcribed.
Currently doing ToBI labeling.
Agustín Gravano - Columbia University

38. Games Project - Studies
Ongoing studies
Discourse Markers (okay, mm-hm, yeah, etc.)
Turn-taking
Laughter
Future studies
Use of the downstepped contour with respect to discourse structure and info status.
Evolution of the description of lexical entities.
Disfluencies (false repairs, self-repairs, etc.) …
Agustín Gravano - Columbia University

39. Boston Directions Corpus
Introduction
ToBI
Discourse structure (Grosz & Sidner ’86)
Information status (Prince ’92)
Meaning of intonational contours
The downstepped contours
Boston Directions Corpus
Description of the corpus
Downstep and discourse structure
Downstep and information status
Games Project
Ongoing and future research
Agustín Gravano - Columbia University

40. Boston Directions Corpus
Introduction
ToBI
Discourse structure (Grosz & Sidner ’86)
Information status (Prince ’92)
Meaning of intonational contours
The downstepped contours
Boston Directions Corpus
Description of the corpus
Downstep and discourse structure
Downstep and information status
Games Project
Ongoing and future research
Agustín Gravano - Columbia University

41. “Downstepped contours in the given/new distinction”
On the Role of Prosody in Structuring Discourse October 5, Berlin, Germany
“Downstepped contours in the given/new distinction”
Agustín Gravano
Spoken Language Processing Group
Columbia University, New York