1. Agustín Gravano1,2 Julia Hirschberg1
Turn-Yielding Cues in Task-Oriented Dialogue
Agustín Gravano1,2
Julia Hirschberg1
Columbia University, New York, USA
(2) Universidad de Buenos Aires, Argentina

2. Interactive Voice Response Systems
Introduction
Interactive Voice Response Systems
Quickly spreading.
“Uncomfortable”, “awkward”.
ASR+TTS account for most IVR problems.
Other problems revealed.
Coordination of system-user exchanges.
Long pauses after user turns; interruptions.
Modeling turn-taking behavior should lead to improved system-user coordination.
Begin to show  revealed
An improved  improved
Agustín Gravano SIGdial 2009

3. Introduction
Goal
Learn when the speaker is likely to end her/his conversational turn.
Find turn-yielding cues.
Cues displayed by the speaker when approaching a potential turn boundary.
This should improve the coordination of IVRs:
Speech understanding: Detect the end of the user’s turn.
Speech generation: Display cues signalling the end of system’s turn.
Could improve  should improve
Agustín Gravano SIGdial 2009

4. Talk Outline Previous work Material Method Results Conclusions
Rest of this talk  Talk Outline or Outline of Talk
Agustín Gravano SIGdial 2009

5. Previous Work on Turn-Taking
Duncan 1972, 1973, 1974, inter alia.
Hypothesized 6 turn-yielding cues in face-to-face dialogue.
Conjectured a linear relation between the number of displayed cues and the likelihood of a turn-taking attempt.
Studies formalized and verified some of Duncan’s hypotheses. [For&Tho96; Wen&Sie03; Cut&Pea86; Wic&Cas01]
Implementations of turn-boundary detection.
Simulations [Fer&al.02,03; Edl&al.05; Sch06; Att&al.08; Bau08]
Actual systems: Let’s Go! [Rau&Esk08]
Exploiting turn-yielding cues improves performance.
Same comment on duncan as for interspeech talk
Agustín Gravano SIGdial 2009

6. Columbia Games Corpus 12 task-oriented spontaneous dialogues.
Material
Columbia Games Corpus
12 task-oriented spontaneous dialogues.
Standard American English.
13 subjects: 6 female, 7 male.
Series of collaborative computer games.
No eye contact. No speech restrictions.
9 hours of dialogue.
Manual orthographic transcription, alignment.
Manual prosodic annotations (ToBI).
Agustín Gravano SIGdial 2009

7. Columbia Games Corpus Material Player 1: Describer Player 2: Follower
In an Objects games, each player saw a board with 5-7 objects.
The boards were almost identical, with one object misplaced.
One of the players had to describe the position of the target object to the other player, who had to move it to the correct position.
Agustín Gravano SIGdial 2009

8. Turn-Yielding Cues
Cues displayed by the speaker when approaching a potential turn boundary.
Agustín Gravano SIGdial 2009

9. Method Turn-Yielding Cues
IPU (Inter Pausal Unit): Maximal sequence of words from the same speaker surrounded by silence ≥ 50ms.
Hold
Smooth switch
Speaker A:
Speaker B:
IPU1
IPU2
IPU3
Smooth switch: Speaker A finishes her utterance; speaker B takes the turn with no overlapping speech.
Trained annotators distinguished Smooth switches from Interruptions and Backchannels using a scheme based on Ferguson 1977, Beattie 1982.
Agustín Gravano SIGdial 2009

10. Method Turn-Yielding Cues To find turn-yielding cues, we compare:
Speaker A:
Speaker B:
Hold
Smooth switch
IPU1
IPU2
IPU3
To find turn-yielding cues, we compare:
IPUs preceding Holds,
IPUs preceding Smooth switches.
~200 features: acoustic, prosodic, lexical, syntactic.
Agustín Gravano SIGdial 2009

11. Individual Cues Final intonation: Faster speaking rate.
Turn-Yielding Cues
Individual Cues
Final intonation:
Falling (L-L%) or high-rising (H-H%).
Faster speaking rate.
Reduction of final lengthening.
Lower intensity level.
Lower pitch level.
Higher jitter, shimmer, NHR.
Related to perception of voice quality.
Longer IPU duration (seconds and #words).
Again, make it clear that you looked for many more….
Agustín Gravano SIGdial 2009

12. Before smooth switches:
Turn-Yielding Cues
Individual Cues
Textual completion (independent of intonation).
(1) Manually annotated a portion of the data.
Labelers read up to the end of a target IPU (no right context), judged whether it could constitute a ‘complete’ utterance. 400 tokens. K=0.81.
(2) Trained an SVM classifier. 19 lexical + syntactic features. Accuracy: 80%. Maj-class baseline: 55%. Human agreement: 91%.
(3) Labeled all IPUs in the corpus with the SVM model.
Again, make it clear that you looked for many more….
Incomplete
Complete
Before smooth switches:
Before holds:
18%
82%
47%
53%
(X2 test, p ~ 0)
Agustín Gravano SIGdial 2009

13. Individual Cues Final intonation: L-L% or H-H%. Faster speaking rate.
Turn-Yielding Cues
Individual Cues
Final intonation: L-L% or H-H%.
Faster speaking rate.
Lower intensity level.
Lower pitch level.
Higher jitter, shimmer, NHR.
Longer IPU duration.
Textual completion.
Again, make it clear that you looked for many more….
Agustín Gravano SIGdial 2009

14. Defining Presence of a Cue
Turn-Yielding Cues
Defining Presence of a Cue
2-3 representative features for each cue:
Final intonation
Abs. pitch slope over final 200ms, 300ms.
Speaking rate
Syllables/sec, phonemes/sec over IPU.
Intensity level
Mean intensity over final 500ms, 1000ms.
Pitch level
Mean pitch over final 500ms, 1000ms.
Voice quality
Jitter, shimmer, NHR over final 500ms.
IPU duration
Duration in ms, and in number of words.
Textual completion
Complete vs. incomplete (binary).
Define presence/absence based on whether the value is closer to the mean before S or H.
Agustín Gravano SIGdial 2009

15. Top Frequencies of Complex Cues
digit == cue present
dot == cue absent
Turn-yielding cues:
1: Final intonation
2: Speaking rate
3: Intensity level
4: Pitch level
5: IPU duration
6: Voice quality
7: Completion
Agustín Gravano SIGdial 2009

16. Number of cues conjointly displayed
Turn-Yielding Cues
Combined Cues
r 2 = 0.969
Percentage of turn-taking attempts
Number of cues conjointly displayed
Agustín Gravano SIGdial 2009

17. IVR Systems After each IPU from the user:
Turn-Yielding Cues
IVR Systems
After each IPU from the user:
if estimated likelihood > threshold
then take the turn
To signal the end of a system’s turn:
Include as many cues as possible in the system’s final IPU.
Agustín Gravano SIGdial 2009

18. Summary Study of turn-yielding cues.
Objective, automatically computable.
Combined cues.
Improve turn-taking decisions of IVR systems.
Results drawn from task-oriented dialogues.
Not necessarily generalizable.
Suitable for most IVR domains.
Interspeech 2009: Study of backchannel-inviting cues.
Agustín Gravano SIGdial 2009

19. Special thanks to… Julia Hirschberg Thesis Committee Members
Maxine Eskenazi, Kathy McKeown, Becky Passonneau, Amanda Stent.
Speech Lab at Columbia University
Stefan Benus, Fadi Biadsy, Sasha Caskey, Bob Coyne, Frank Enos, Martin Jansche, Jackson Liscombe, Sameer Maskey, Andrew Rosenberg.
Collaborators
Gregory Ward and Elisa Sneed German (Northwestern U); Ani Nenkova (UPenn); Héctor Chávez, David Elson, Michel Galley, Enrique Henestroza, Hanae Koiso, Shira Mitchell, Michael Mulley, Kristen Parton, Ilia Vovsha, Lauren Wilcox.
Agustín Gravano SIGdial 2009