1. Efficient Computer Interfaces Using Continuous Gestures, Language Models, and Speech Keith Vertanen Inference Group August 4th, 2004

2. The problem Speech recognizers make mistakes Correcting mistakes is inefficient  140 WPM Uncorrected dictation  14 WPMCorrected dictation, mouse/keyboard  32 WPMCorrected typing, mouse/keyboard Voice-only correction is even slower and more frustrating

3. Research overview Make correction of dictation:  More efficient  More fun  More accessible Approach:  Build a word lattice from a recognizer’s n-best list  Expand lattice to cover likely recognition errors  Make a language model from expanded lattice  Use model in a continuous gesture interface to perform confirmation and correction

4. Building lattice Example n-best list: 1: jack studied very hard 2: jack studied hard 3: jill studied hard 4: jill studied very hard 5: jill studied little

5. Insertion errors

6. Acoustic confusions Given a word, find words that sound similar Look pronunciation up in dictionary: studieds t ah d iy d Use observed phone confusions to generate alternative pronunciations:s t ah d iy d s ao d iy s t ah d iy … Map pronunciation back to words: s t ah d iy d studied s ao d iysaudi s t ah d iystudy

7. Acoustic confusions: “Jack studied hard”

8. Morphology confusions Given a word, find words that share the same “root”. Using the Porter stemmer: jacking jacks jack jacked study studying studied studies studi jack

9. Morphology confusions: “Jack studied hard”

10. Language model confusions: “Jack studied hard” Look at words before or after a node, add likely alternate words based on n-gram LM

11. Expansion results (on WSJ1)

12. Probability model Our confirmation and correction interface requires probability of a letter given prior letters:

13. Probability model Keep track of possible paths in lattice Prediction based on next letter on paths Interpolate with default language model Example, user has entered “the_cat”: 1.00

14. Handling word errors Use default language model during entry of erroneous word Rebuild paths allowing for an additional deletion or substitution error Example, user has entered “the_cattle_”: 0.25 0.0625

15. Using expanded lattice Paths using arcs added during lattice expansion are penalized. Example, user has entered “jack_”: 0.04 1.00

16. Evaluating expansion Assume a good model requires as little information from the user as possible

17. Results on test set Model evaluated on held out test set (Hub1) Default language model  2.4 bits/letter  User decides between 5.3 letters Best speech-based model  0.61 bits/letter  User decides between 1.5 letters

18. “To the mouse snow means freedom from want and fear”

19. “The hibernating skunk curled up in his deep den uncurls himself and ventures forth to prowl the world”

20. Conclusions One-third of recognition errors covered by expanding lattice. Only insertion error expansion improves efficiency. Speech-based model significantly improves efficiency (2.4 bits -> 0.61 bits). A good correction interface is possible using Dasher and an off-the-shelf recognizer.

21. Future work Update Speech Dasher to use lattice-based probability model. Incorporate hypothesis probabilities into lattice (or even better get at recognizer’s lattice). Improve efficiency on sentences with few or no errors. User trials to validate numeric results.

22. Questions?