1. Adding Sentence Boundaries to Conversational Speech Transcriptions using Noisily Labelled Examples
Tetsuya Nasukawa, IBM Tokyo Research Lab
Diwakar Punjani, IBM India Research Lab
Shourya Roy , IBM India Research Lab
L V Subramaniam , IBM India Research Lab
Hironori Takeuchi, IBM Tokyo Research Lab
Presented by : Shourya Roy
December 9, 2018
IBM Research

2. What are We Trying to do?
Automatically identifying sentence boundaries in noisy transcriptions of conversational data.
Transcriptions can be manual or automatic (ASR)
It can work without any manual supervision
The accuracy improves with manual supervision
Detects only periods – not comma, semicolon
December 9, 2018
IBM Research

3. Importance – One Motivating Example from Real Life
Huge amount of telephonic conversational data produced in various domains such as CRM, BPO
Important to analyze to improve customer satisfaction, agent productivity, market reputation
NLP techniques on transcriptions is an obvious approach
Transcriptions are noisy and does not contain any punctuation marks
POS taggers and syntactic parsers perform poorly in absence of sentence boundaries
Importance of analysis of transcriptions
Importance of sentence boundary detection for transcriptions analysis
December 9, 2018
IBM Research

4. Why Non Trivial Noise in the dataset
Spontaneous nature of conversation
Variation in style of speaking
Boundary density varies from call to call
Removing the calls with very low boundary density improves the scores by approx. 10%
December 9, 2018
IBM Research

5. Existing Solutions SBD on conversational data – not many work
Based on Pause (Silence) Information
December 9, 2018
IBM Research

6. Example: Manual Transcription
A: i've i've barely been out of the country. i wouldn't {breath}
B: {lipsmack} {breath}
A: i think my most memorable trip was when i was in high school.
B: {breath} uh-huh.
A: i went to %uh ^London and ^Paris.
B: %oh that's cool.
A: and that's about as exotic as it ever got.
B: {breath} was it fun?
A: %uh other than that, i haven't been west of ^Texas
B: %hm.
B: {breath} it looks like you are a east *coaster born and raised.
A: yeah. how about yourself? where are you?
B: {breath} i'm in ^Philly
A: you're in ^Philly, i guess? i wonder if everybody here is in ^Philly? probably. {breath}
B: yeah.
B: yeah, i think so because it's a ~U ^Penn thing. they probably just did it locally. plus
B: %uh are you using an ^Omnipoint phone?
A: uh-huh
Timing
Meta Info
Names of Places
Speaker
December 9, 2018
IBM Research

7. Example : Automatic Transcription
then go to properties ok now once when you go to properties up if you scroll down there that he's having internet protocol ok you have to no i'm sorry just any scroll down that you're having a net firewall so that's no we have to check if there's a check next to it ok if it's not checked you have to get a check that ok and if if you do not so if you are calling you having a check all you have to do is i can check the net firewalls so this ok and you have to go ahead and reboot the system
December 9, 2018
IBM Research

8. Example
then go to properties ok now once when you go to properties up if you scroll down there that he's having internet protocol ok you have to no i'm sorry just any scroll down that you're having a net firewall so that's no we have to check if there's a check next to it ok if it's not checked you have to get a check that ok and if if you do not so if you are calling you having a check all you have to do is i can check the net firewalls so this ok and you have to go ahead and reboot the system
December 9, 2018
IBM Research

9. Summary of Proposed Technique
From (possibly imprecisely) marked sentence boundaries in conversational data identify n-grams which are more likely to occur at sentence boundaries than inside the sentence
Mark sentence boundaries before (or after) head or (tail) n-grams in test data
December 9, 2018
IBM Research

10. Technique Preprocessing of data
Pause filling words, repetitions, unclear words are removed
Identify frequent head and tail n-grams from training data which occur in beginning and ending of sentences
Filter n-grams which also occur significant number of times in middle of the sentences
Threshold on head/tail:middle of sentence ratio
Handle interruption and continuation across turns separately
Words indicating incomplete turn e.g. get, and
December 9, 2018
IBM Research

11. Technique (Contd.)
In the test set mark a boundary before every head n-gram and after every tail n-gram
In the case of boundaries marked based on silence information on ASR data, add new sentence boundaries
If the turn does not end with a word from the set of words indicating incomplete turn mark a boundary at the end of the turn
December 9, 2018
IBM Research

12. Nature of Data Manual Transcriptions Automatic Transcriptions
Switchboard corpus and the Call-home corpus of transcribed phone conversations from LDC
Automatic Transcriptions
Manually put punctuations
Automatically put punctuations based on silence
ASR transcribed calls from IBM helpdesk
Data Statistics
December 9, 2018
IBM Research

13. Results Result of punctuation insertion for helpdesk data Method
Precision
Recall F1
Word Error Rate (WER)
Only Silence
0.54
0.28
0.37
0.96
Only Head/Tail
0.78
0.55
0.65
0.60
Increasing
Decreasing
Head/Tail + Silence
0.66
0.72
0.68
Head/Tail + Silence – FalseBoundaries
0.69
0.70
0.58
Result of punctuation insertion for helpdesk data
December 9, 2018
IBM Research

14. Improvement in PoS Tagging
PoS Tagging Accuracy on Helpdesk Data
An example PoS tagging improving with sentence boundary detection
Ideally ‘i’ should be pronoun and ‘yeah’ and ‘oh’ should be interjection
December 9, 2018
IBM Research

15. Improvement in PoS Tagging (Contd.)
Extracted top 10 Noun Phrases from Switchboard Data Set
December 9, 2018
IBM Research

16. Summary
Fundamental operation to be performed to apply state-of-the-art NLP techniques on (automatic) transcriptions of conversations
We proposed a technique to train a sentence boundary detector with minimal manual supervision
It would be interesting to see how much improvement is happening in actual extraction task!
December 9, 2018
IBM Research

17. Questions?
December 9, 2018
IBM Research