Presentation is loading. Please wait.

Presentation is loading. Please wait.

Word-subword based keyword spotting with implications in OOV detection Jan “Honza” Černocký, Igor Szöke, Mirko Hannemann, Stefan Kombrink Brno University.

Published byBrian Hunter Modified over 8 years ago

Similar presentations

Presentation on theme: "Word-subword based keyword spotting with implications in OOV detection Jan “Honza” Černocký, Igor Szöke, Mirko Hannemann, Stefan Kombrink Brno University."— Presentation transcript:

1 Word-subword based keyword spotting with implications in OOV detection Jan “Honza” Černocký, Igor Szöke, Mirko Hannemann, Stefan Kombrink Brno University of Techbnology BUT Speech@FIT 44 th Asilomar Conference on Signals, Systems and Computers, 8.11.2010

2 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.20102/34 Agenda Word-based STD, OOV problem, subwords Experiments Sub-word units Hybrid word-subword system What can we do with OOVs Conclusion

3 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.20103/34 Goal of STD and glossary of terms Goal: detect keywords or key-phrases in input speech, for each detection, output: Identity Position Score Glossary Large Vocabulary Continuous Speech Recognizer – LVCSR – system converting spoken speech into text. Out-of-vocabulary – OOV – word which is not in the LVCSR vocabulary. Term – textual entry consisting of one or more words in sequence. Spoken Term Detection – STD – a way to search for a term in spoken data. Subword(s) – unit(s) that are parts of words (phones, syllables, automatically found, etc.).

4 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.20104/34 Word-based STD Due to the presence of language model, Word-based STD systems are reaching better accuracies than acoustic ones.

5 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.20105/34 Implementation Term is searched in recognition lattice Allows to estimate posterior probability of a term.

6 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.20106/34 The OOV problem REF: THIS IS AN EXAMPLE OF RECOGNIZER OUTPUT REC: THIS IS AMEX APPLE OF RECOGNIZER OUTPUT One OOV causes several errors: OOV can not be found (in the output of LVCSR). OOV impairs recognition of neighboring words. OOV usually carries lot of information (named entity).  We need to handle OOVs ! Word accuracy. Spoken term detection accuracy. Practical (memory, CPU, index size, etc.).

7 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.20107/34 Answer to OOV problem – sub-word STD Subword recognizer is built (output is subword lattice). Term is converted from words to sequence of subwords. This sequence is searched in the subword lattice.

8 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.20108/34 Agenda Word-based STD, OOV problem, subwords Experiments Sub-word units Hybrid word-subword system What can we do with OOVs Conclusion

9 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.20109/34 Evaluation - TWV Defined by NIST for NIST STD 2006 evaluation: one number higher is better depending on normalization Requires full STD system

10 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.201010/34 Normalization-independent evaluation - UBTVW UBTWV - Upper Bound Term Weighted Value Finds optimum threshold for each term one number higher is better Independent on normalization

11 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.201011/34 Data NIST STD 2006 evaluations. 3h of English telephone conversations. 373 1-4 words long terms occurring 4737/196 times.

12 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.201012/34 Recognizer I. LVCSR developed in AMI/AMIDA project State-of the art system including VTLN, MPE, posterior features, SAT, 3 passes. Acoustic models trained on 278h of speech. Language model trained on 977M word tokens (50k vocabulary). Dictionary pruned to generate OOVs -> WRDRED. Word accuracy – 69.04%.

13 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.201013/34 Recognizer II.

14 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.201014/34 Results Words Words converted to phones Phone recognizer Phones too small => need longer units

15 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.201015/34 Agenda Word-based STD, OOV problem, subwords Experiments Sub-word units Hybrid word-subword system What can we do with OOVs Conclusion

16 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.201016/34 Better subwords – phone multigrams Statistics of phone n-grams are collected (up to 6) from training data (phone transcriptions of speech). Probabilities of all units are estimated. Training data are segmented by the most probable sequence of multigrams. Statistics are recomputed and low occurring units are deleted. Several iterations. N-gram language model is estimated on top of the multigram segmentation of the training data.

17 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.201017/34 Constrained multigrams nosil – sil is not part of multigram unit. noxwrd – add information of word boundary to multigram unit. Term (word representation): PRIME MINISTER Term pronunciation: p r ay m m ih n ih s t axr Term (subword representation): *p-r-ay m* *m-ih-n ih-s t-axr*

18 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.201018/34 Results Subword search can process OOV terms. Subword search is not so accurate as word search of in-vocabulary terms. Subword search consumes more index space. => Need for combination of word and subword searches.

19 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.201019/34 Agenda Word-based STD, OOV problem, subwords Experiments Sub-word units Hybrid word-subword system What can we do with OOVs Conclusion

20 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.201020/34 Parallel word-subword … works, but needs to maintain and run 2 systems.

21 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.201021/34 Hybrid word-subword

22 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.201022/34 Implementation by composition of networks

23 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.201023/34 Multigram dictionary for hybrid system For hybrid system, phone multigrams must not be trained on utterances. Phone multigrams are trained on dictionary. Experimented with LVCSR vs. big vs. OOV dictionary.

24 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.201024/34 Results – different configurations Pruning factors play role in the memory consumption, size of index, RT factor … “Reasonable system” ~2.5x slower than word ~2.5x bigger index than word Matches the accuracy of word system for IV OOVs found.

25 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.201025/34 Agenda Word-based STD, OOV problem, subwords Experiments Sub-word units Hybrid word-subword system What can we do with OOVs Conclusion

26 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.201026/34 OOV detection by the hybrid system Comparison of the subword confidence measure to a threshold => detection of OOVs

27 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.201027/34 OOV recovery Use of phoneme to grapheme (P2G) to derive word-form of detected OOV

28 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.201028/34 Alignment error model Some detected OOVs could be even converted back to in-vocabulary words ! But the phone pronunciation in 1-best output is not ideal… … alignment error model Parameters (probabilities of deletion, insertion, substitution) trained from data. Can process dictionary and look up detected OOVs.

29 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.201029/34 Going more complex … Can construct an wFST accounting for Sequences of in-vocabulary words In-vocabulary words + common pre- and suffixes OOVs And combinations … m ey sh en -> INFORMATION ae l k ax hh aa l ih z em (ALCOHOLISM) -> ALCOHOL / ISM aa f ax s m ae k s (’Office Max’) -> OFFICE OOV1572

30 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.201030/34 OOV clustering Alignment model allows for the evaluation of similarity Clustering possible

31 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.201031/34 Agenda Word-based STD, OOV problem, subwords Experiments Sub-word units Hybrid word-subword system What can we do with OOVs Conclusion

32 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.201032/34 Conclusion Subword system with constrained multigrams - very good STD performace and OOV tolerant system. Improved hybrid word-subword system tested from STD accuracy and real application point of view. Hybrid system brings better accuracy/size ratio and is faster than the standalone system. It works well in a real indexing & search engine. With a hybrid system, we can Recover OOVs (simple P2G or more elaborate model) Measure similarity of OOVs Cluster them, find re-occurring ones, update vocabulary.

33 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.201033/34 Reading and playing with Igor Szöke: Hybrid word-subword spoken term detection, Ph.D. thesis, Brno University of Technology, Oct 2010 Stefan Kombrink, Mirko Hannemann, Lukáš Burget, and Hynek Heřmanský: Recovery of Rare Words in Lecture Speech, in Proc. Text, Speech and Dialogue (TSD) 2010, Brno, 2010 Mirko Hannemann, Stefan Kombrink, Martin Karafiát, and Lukáš Burget: Similarity Scoring for Recognizing Repeated Out-of-VocabularyWords, in Proc. Interspeech 2010, Makuhari, Japan, 2010. … ‘Publications’ section of http://speech.fit.vutbr.cz/http://speech.fit.vutbr.cz/ http://www.superlectures.com/odyssey/

34 ASILOMAR SS & C Černocký, Szöke, Hanneman, Kombrink8.11.201034/34 Thank you for your attention

Download ppt "Word-subword based keyword spotting with implications in OOV detection Jan “Honza” Černocký, Igor Szöke, Mirko Hannemann, Stefan Kombrink Brno University."

Similar presentations

Robust Speech recognition V. Barreaud LORIA. Mismatch Between Training and Testing n mismatch influences scores n causes of mismatch u Speech Variation.

Robust Speech recognition V. Barreaud LORIA. Mismatch Between Training and Testing n mismatch influences scores n causes of mismatch u Speech Variation.
Yasuhiro Fujiwara (NTT Cyber Space Labs)

Yasuhiro Fujiwara (NTT Cyber Space Labs)
Speech Recognition Part 3 Back end processing. Speech recognition simplified block diagram Speech Capture Speech Capture Feature Extraction Feature Extraction.

Speech Recognition Part 3 Back end processing. Speech recognition simplified block diagram Speech Capture Speech Capture Feature Extraction Feature Extraction.
Nonparametric-Bayesian approach for automatic generation of subword units- Initial study Amir Harati Institute for Signal and Information Processing Temple.

Nonparametric-Bayesian approach for automatic generation of subword units- Initial study Amir Harati Institute for Signal and Information Processing Temple.
Development of Automatic Speech Recognition and Synthesis Technologies to Support Chinese Learners of English: The CUHK Experience Helen Meng, Wai-Kit.

Development of Automatic Speech Recognition and Synthesis Technologies to Support Chinese Learners of English: The CUHK Experience Helen Meng, Wai-Kit.
Confidence Measures for Speech Recognition Reza Sadraei.

Confidence Measures for Speech Recognition Reza Sadraei.
Speech Recognition Training Continuous Density HMMs Lecture Based on:

Speech Recognition Training Continuous Density HMMs Lecture Based on:
Acoustical and Lexical Based Confidence Measures for a Very Large Vocabulary Telephone Speech Hypothesis-Verification System Javier Macías-Guarasa, Javier.

Acoustical and Lexical Based Confidence Measures for a Very Large Vocabulary Telephone Speech Hypothesis-Verification System Javier Macías-Guarasa, Javier.
DIVINES – Speech Rec. and Intrinsic Variation W.S.May 20, 2006 Richard Rose DIVINES SRIV Workshop The Influence of Word Detection Variability on IR Performance.

DIVINES – Speech Rec. and Intrinsic Variation W.S.May 20, 2006 Richard Rose DIVINES SRIV Workshop The Influence of Word Detection Variability on IR Performance.
IBM Haifa Research Lab © 2008 IBM Corporation Retrieving Spoken Information by Combining Multiple Speech Transcription Methods Jonathan Mamou Joint work.

IBM Haifa Research Lab © 2008 IBM Corporation Retrieving Spoken Information by Combining Multiple Speech Transcription Methods Jonathan Mamou Joint work.
Automatic Continuous Speech Recognition Database speech text Scoring.

Automatic Continuous Speech Recognition Database speech text Scoring.
Introduction to Automatic Speech Recognition

Introduction to Automatic Speech Recognition
Rapid and Accurate Spoken Term Detection Owen Kimball BBN Technologies 15 December 2006.

Rapid and Accurate Spoken Term Detection Owen Kimball BBN Technologies 15 December 2006.
1 7-Speech Recognition (Cont’d) HMM Calculating Approaches Neural Components Three Basic HMM Problems Viterbi Algorithm State Duration Modeling Training.

1 7-Speech Recognition (Cont’d) HMM Calculating Approaches Neural Components Three Basic HMM Problems Viterbi Algorithm State Duration Modeling Training.
A Phonotactic-Semantic Paradigm for Automatic Spoken Document Classification Bin MA and Haizhou LI Institute for Infocomm Research Singapore.

A Phonotactic-Semantic Paradigm for Automatic Spoken Document Classification Bin MA and Haizhou LI Institute for Infocomm Research Singapore.
English Pronunciation Learning System for Japanese Students Based on Diagnosis of Critical Pronunciation Errors Yasushi Tsubota, Tatsuya Kawahara, Masatake.

English Pronunciation Learning System for Japanese Students Based on Diagnosis of Critical Pronunciation Errors Yasushi Tsubota, Tatsuya Kawahara, Masatake.
Zero Resource Spoken Term Detection on STD 06 dataset Justin Chiu Carnegie Mellon University 07/24/2012, JHU.

Zero Resource Spoken Term Detection on STD 06 dataset Justin Chiu Carnegie Mellon University 07/24/2012, JHU.
1 CSE 552/652 Hidden Markov Models for Speech Recognition Spring, 2006 Oregon Health & Science University OGI School of Science & Engineering John-Paul.

1 CSE 552/652 Hidden Markov Models for Speech Recognition Spring, 2006 Oregon Health & Science University OGI School of Science & Engineering John-Paul.
Graphical models for part of speech tagging

Graphical models for part of speech tagging
Learning of Word Boundaries in Continuous Speech using Time Delay Neural Networks Colin Tan School of Computing, National University of Singapore.

Learning of Word Boundaries in Continuous Speech using Time Delay Neural Networks Colin Tan School of Computing, National University of Singapore.

Similar presentations

Ads by Google