1. Using Conversational Word Bursts in Spoken Term Detection Justin Chiu Language Technologies Institute Presented at University of Cambridge September 6 2013

2. Introduction Spoken Term Detection (STD): detecting word or phrase targets in conversational speech Queries: Text Target: Audio The BABEL program: rapid development of key word search for novel languages, with limited resources This research: using (ideally universal) properties of conversation to enhance performance

3. Current Approach 1ASR produces candidates (as a confusion network) 2Search on the ASR result identifies targets – Single-word query: Posterior probability for the query word in confusion network – Multi-word query: Product of the posterior probability for each word in confusion network 3Deciding detection result with search result probability/threshold (YES/NO decision)

4. Challenge for Current Approach Sensitive to ASR performance High-performance ASR requires extensive training resources; limited resources produce low-accuracy In the BABEL LimitedLP condition, only 10 hour training data are provided

5. Intuition Can we use other sources of information to enhance STD performance even with poor ASR results? We specifically focus on conversational structure, since we believe it is language- independent

6. Word Burst Word burst refers to a phenomenon in conversational speech in which particular content words tend to occur in close proximity to each other, as a byproduct of some topic under discussion Content Word: Word that do not occur with high frequency

7. Occurrence pattern for the Tagalog word “magkano”

10. Word Burst rescoring on the Confusion Network Give Bonus to the posterior probability for the content word that is in a Word Burst Penalize the posterior probability for the content word that is not in a Word Burst Goal: Improve the quality for the confusion network for better search performance

11. Word Burst Rescoring Parameters Window Size: time region for deciding Word Burst Stop word %: top X% of words in training data; assumed not to be content words Penalty for non-burst word: penalty for words that do not appear in a Word Burst Trained from language pack (development) data

12. Word Burst Rescoring Formula

18. Datasets 4 Language, 2 different size of training data (80 hour, 10 hour) An additional 10 hours of data in each language for 5 fold cross-validation LanguageSetupLexicon Size Cantonese FullLP18769 LimitedLP5112 Pasto FullLP17904 LimitedLP6219 Tagalog FullLP21098 LimitedLP5565 Turkish FullLP38849 LimitedLP10173

19. ATWV Actual Term Weighted Value In BABEL, the C/V is 0.1, and the is 10 -4 Having a Correct Detection is much more valuable than reducing false alarm

20. Comparison between 10/80 hours of training data In the condition with limited data, the introduction of additional information can compensate for the lack of training data LanguageSetupBaselineRescoreChange Cantonese FullLP0.3220.320-0.7% LimitedLP0.1030.109+6% Pashto FullLP0.2140.215+0.5% LimitedLP0.0950.114+19% Tagalog FullLP0.358 -0.3% LimitedLP0.1300.144+11% Turkish FullLP0.385 +0.1% LimitedLP0.2620.265+1%

21. Tuned LimitedLP(10h) performance Rescoring significantly reduces the False Alarm from detection result, indicating that penalizing an isolated content word is beneficial to STD performance LanguageBaselineRescore∆ ATWVFA Cantonese0.1140.1184%-21% Pashto0.0730.09429%-32% Tagalog0.1430.15911%-21% Turkish0.2410.2452%+4%

22. Best Parameters for LimitedLP(10h) Apart from Turkish, parameter values are similar across languages. Words do not repeat in the same form in Turkish, an agglutinative language. LanguageWindow (sec)Stop (%)Penalty Cantonese1240.1 Pashto1810.05 Tagalog1120.15 Turkish981

23. Analysis/Discussion 1.The impact of Word Burst on WER? 2.How often does a Word Burst occur? 3.Issues in Turkish/Vietnamese? 4.Difference between Word Burst and Cache- base/Trigger-based language model?

24. WER Word Burst does not provide significant improvement on WER The rescoring helps the candidate’s probability to cross the detection threshold

25. How often do Word Bursts occur? Query/Content word burst percentage (10sec) More than 35% of Content/Query words occur as part of a Word Burst, indicating it’s generality Query/Content word burst percentages are similar, except for Turkish LanguageCantonesePashtoTagalogTurkish Content43.635.740.735.4 Query41.235.336.827.7

26. Issues in Turkish/Vietnamese Turkish word will re-occur in different form, can not count on Word Burst for the same word Using tool for morphological normalization might be able to find Word Burst for the same word Vietnamese performance is in the same range as Cantonese Some of the evaluation queries are syllables, not words, which Burst does not sounds reasonable

27. How Word Burst differs from Cache/Trigger Language Models 1.It not only considers the top 1 hypothesis, it considers the entire confusion set 2.It examines both past and future; Cache/Trigger based language model focus on the past 3.It focuses on time-based information, instead of token based information

28. Conclusion Our experiments indicate that the Word Burst is a property of conversational speech and is language-independent phenomenon Information from language or conversational structure (such as Word Burst) can be used to improve performance in the STD task, particularly when language resources are otherwise limited. It is potentially useful in other applications

29. Special Thanks Alex RudnickyAlan Black Florian Metze Yajie MiaoYuchen Zhang

30. Questions?