2. Classification of Discourse Functions of Affirmative Words in Spoken Dialogue Julia Agustín Gravano, Stefan Benus, Julia Hirschberg Shira Mitchell, Ilia Vovsha INTERSPEECH, Antwerp, August 2007 Spoken Language Processing Group Columbia University

3. Agustín Gravano INTERSPEECH 20072 Cue Words Ambiguous linguistic expressions used for Making a semantic contribution, or Conveying a pragmatic function. Examples: now, well, so, alright, and, okay, first, by the way, on the other hand. Single affirmative cue words Examples: alright, okay, mm-hm, right, uh-huh, yes. May be used to convey acknowledgment or agreement, to change topic, to backchannel, etc.

4. Agustín Gravano INTERSPEECH 20073 Research Goals Learn which features best characterize the different functions of single affirmative cue words. Determine how these can be identified automatically. Important in Spoken Dialogue Systems: Understand user input. Produce output appropriately.

5. Agustín Gravano INTERSPEECH 20074 Previous Work Classification of cue words into discourse vs. sentential use. Hirschberg & Litman ’87, ’93; Litman ’94; Heeman, Byron & Allen ’98; Zufferey & Popescu-Belis ’04. In our corpus: right: 15% discourse, 85% sentential. All other affirmative cue words: 99% disc., 1% sent. Discourse vs. sentential distinction insufficient. Need to define new classification tasks.

6. Agustín Gravano INTERSPEECH 20075 Talk Overview Columbia Games Corpus Classification tasks Experimental features Results

7. Agustín Gravano INTERSPEECH 20076 The Columbia Games Corpus 12 spontaneous task-oriented dyadic conversations in Standard American English. 2 subjects playing computer games; no eye contact.

8. Agustín Gravano INTERSPEECH 20077 The Columbia Games Corpus Function of Affirmative Cue Words Cue Words alright gotcha huh mm-hm okay right uh-huh yeah yep yes yup Functions Acknowledgment / Agreement Backchannel Cue beginning discourse segment Cue ending discourse segment Check with the interlocutor Stall / Filler Back from a task Literal modifier Pivot beginning: Ack/Agree + Cue begin Pivot ending: Ack/Agree + Cue end 7.9% of the words in our corpus

9. Agustín Gravano INTERSPEECH 20078 Literal Modifier that’s pretty much okay Backchannel Speaker 1:between the yellow mermaid and the whale Speaker 2:okay Speaker 1:and it is Cue beginning discourse segment okay we gonna be placing the blue moon The Columbia Games Corpus Function of Affirmative Cue Words

10. Agustín Gravano INTERSPEECH 20079 The Columbia Games Corpus Function of Affirmative Cue Words 3 trained labelers Inter-labeler agreement: Fleiss’ Kappa = 0.69 (Fleiss ’71) In this study we use the majority label for each affirmative cue word. Majority label: label chosen by at least two of the three labelers.

11. Agustín Gravano INTERSPEECH 200710 Identification of a discourse segment boundary function Segment beginning vs. Segment end vs. No discourse segment boundary function Identification of an acknowledgment function Acknowledgment vs. No acknowledgment Method Two new classification tasks

12. Agustín Gravano INTERSPEECH 200711 ML Algorithm JRip: Weka’s implementation of the propositional rule learner Ripper (Cohen ’95). We also tried J4.8, Weka’s implementation of the decision tree learner C4.5 (Quinlan ’93, ’96), with similar results. 10-fold cross validation in all experiments. Method Machine Learning Experiments

13. Agustín Gravano INTERSPEECH 200712 IPU (Inter-pausal unit) Maximal sequence of words delimited by pause > 50ms. Conversational Turn Maximal sequence of IPUs by the same speaker, with no contribution from the other speaker. Method Experimental features

14. Agustín Gravano INTERSPEECH 200713 Text-based features Extracted from the text transcriptions. Lexical id; POS tags; position of word in IPU / turn; etc. Timing features Extracted from the time alignment of the transcriptions. Word / IPU / turn duration; amount of overlap; etc. Acoustic features {min, mean, max, stdev} x {pitch, intensity} Slope of pitch, stylized pitch, and intensity, over the whole word, and over its last 100, 200, 300ms. Acoustic features from the end of the other speaker’s previous turn. Method Experimental features

15. Agustín Gravano INTERSPEECH 200714 Results Discourse segment boundary function Feature SetError Rate F-Measure BeginEnd Text-based11.6 %.77.30 Timing11.3 %.73.52 Acoustic14.2 %.66.19 Text-based + Timing 9.8 %.81.53 Full set 9.6 %.81.57 Baseline (1) 19.0 %.00 Human labelers (2) 5.7 %.94.71 (1) Majority class baseline: NO BOUNDARY. (2) Calculated wrt each labeler’s agreement with the majority labels.

16. Agustín Gravano INTERSPEECH 200715 Results Acknowledgment function Feature SetError RateF-Measure Text-based 8.3 %.94 Timing11.0 %.92 Acoustic17.2 %.87 Text-based + Timing6.2 %.95 Full set 6.5 %.95 Baseline (1) 16.7 %.88 Human labelers (2) 5.5 %.98 (1) Baseline based on lexical identity:{huh, right }  no ACK all other words  ACK (2) Calculated wrt each labeler’s agreement with the majority labels.

17. Agustín Gravano INTERSPEECH 200716 Best-performing features Discourse Segment Boundary Function Acknowledgment Function Lexical identity POS tag of the following word Number and proportion of succeeding words in the turn Context-normalized mean intensity Lexical identity POS tag of preceding word Number and proportion of preceding words in the turn IPU and turn length

18. Agustín Gravano INTERSPEECH 200717 Results Classification of individual words Classification of each individual word into its most common functions. alright  Ack/Agree, Cue Begin, Other mm-hm  Ack/Agree, Backchannel okay  Ack/Agree, Backchannel, Cue Begin, Ack+CueBegin, Ack+CueEnd, Other right  Ack/Agree, Check, Literal Modifier yeah  Ack/Agree, Backchannel

19. Agustín Gravano INTERSPEECH 200718 Results Classification of the word ‘okay’ Feature Set Error Rate F-Measure Ack / Agree Back- channel Cue Begin Ack/Agree + Cue Begin Ack/Agree + Cue End Text-based31.7.76.16.77.09.33 Acoustic40.2.69.24.64.03.25 Text-based + Timing25.6.79.31.82.18.67 Full set25.5.80.46.83.21.66 Baseline (1) 48.3.68.00 Human labelers (2) 14.0.89.78.94.56.73 (1) Majority class baseline: ACK/AGREE. (2) Calculated wrt each labeler’s agreement with the majority labels.

20. Agustín Gravano INTERSPEECH 200719 Summary Discourse/sentential distinction is insufficient for affirmative cue words in spoken dialogue. Two new classification tasks: Detection of an acknowledgment function. Detection of a discourse boundary function. Best performing ML models: Based on textual and timing features. Slight improvement when using acoustic features.

21. Agustín Gravano INTERSPEECH 200720 Further Work Gravano et al, 2007 On the role of context and prosody in the interpretation of ‘okay’. ACL 2007, Prague, Czech Republic, June 2007. Benus et al, 2007 The prosody of backchannels in American English. ICPhS 2007, Saarbrücken, Germany, August 2007.

22. Classification of Discourse Functions of Affirmative Words in Spoken Dialogue Julia Agustín Gravano, Stefan Benus, Julia Hirschberg Shira Mitchell, Ilia Vovsha INTERSPEECH, Antwerp, August 2007 Spoken Language Processing Group Columbia University

23. Agustín Gravano INTERSPEECH 200722 alrightmm-hmokayrightuh-huhyeahOtherTotal Ack / Agree99611137114188081332370 Backchannel640212114143725763 Cue Begin8905482020641 Cue End8010000018 Pivot Begin5068000073 Pivot End1312232202217298 Back from Task9133000043 Check0065301868 Stall1015102019 Literal Modifier902910790011118 ?56272351036511407 Total295503243412751649721755818