1. Turn-taking and Disfluencies
Julia Hirschberg
CS 4706
11/23/2018

2. Today Turn-taking behaviors Disfluencies Conversational Analysis
Importance in real systems
Disfluencies
How to model? Detect?
Role in human-human interaction
Importance in real systems?
11/23/2018

3. Turn-taking Expected patterns of behavior Deviation is significant
How do we find the patterns?
Ordinary conversation
Telephone talk
Meetings ?
Who looks for these?
11/23/2018

4. Terminology Examples: Adjacency pairs Preference Pre-sequence Repair
Telephone openings, closings
Broadcasts
11/23/2018

5. Could this be useful when we build SDS?
What do we expect to hear?
What should we produce?
11/23/2018

6. Auditory Cues to Turn-Taking
M. Schegloff “Reflections on studying prosody in talk-in-interaction, ” Language and Speech 41, (Michael Mu.)
H. Koiso et al ‘99 “An analysis of turn-taking and backchannels based on prosodic and syntactic features…,” Language and Speech 41, (Sarah)
11/23/2018

7. Disfluencies and Self-Repairs
Are these just ‘noise’?
For people
S. Brennan & M. Williams, “The Feeling of Another’s Knowing,” J Memory and Language 34, (Judd)
S. Brennan & Schober, “How listeners compensate for disfluencies in spontaneous speech,” J Memory and Language 44, (Aron)
For parsers
For speech recognizers
11/23/2018

8. Hindle ’83: Finding the Edit Signal
If we have it, can we ‘repair’ the self-repair automatically?
Builds a correcting parser, Fidditch, for spontaneous speech
Given a string with an edit signal marked, produces a ‘repaired’ version
I was * I am really annoyed
If X1 * X2 are similar linguistic elements separated by an edit signal, replace X1 w/X2
11/23/2018

9. What does it mean to be the same
Same surface string
Well if they’d * if they’d…
Same category
I was just that * the kind of guy…
Same constituent
I think that you get * it’s more strict in Catholic schools
Restarts are completely different…
I just think * Do you want something to eat?
11/23/2018

10. Bear et al ’92: Detecting and Correcting Self-Repairs
Use multiple knowledge sources but not edit signal
Lexical pattern matching
Parsing failure + pattern matching + re-parsing
Acoustic information: pause, peak F0,
Cue words: well, no
Fragments
11/23/2018

11. But…is there an edit signal?
11/23/2018

12. 11/23/2018

13. RIM Model of Self-Repairs (Nakatani & Hirschberg ’94)
ATIS corpus
6414 turns with 346 (5.4%) repairs, 122 speakers
Hand-labeled for repairs and prosodic features
Findings:
Reparanda: 73% end in fragments, 30% in glottalization, co-articulatory gestures
DI: pausal duration differs significantly from fluent boundaries,small increase in f0 and amplitude
I.e. 346 repairs
Reparanda:
73% end in fragments
30% end in glottalization (short, no decrease in energy or f0)
co-articulatory gestures
DI:
signif diffs in duration of pause but can’t be used (alone) to predict disfluencies as too many false positive
small increase in f0 and amplitude across to repair
Repairs:
offsets occur at phrase boundaries
differences in phrasing compared to fluent speech
CART prediction of repairs: 86% precision, 91% recall (192 of 223 interruption sites, 19 false positives). Features imp: dur of interva, fragment, pause filler, pos, lexical matching across DI.
11/23/2018

14. Does it identify self-repairs reliably?
CART prediction: 86% precision, 91% recall
Duration of interval, presence of fragment, pause filler, p.o.s., lexical matching across DI
Are there edit signals?
11/23/2018

15. Next Week Spoken Dialogue Systems Andy, David and Vera reporting
11/23/2018