2. BravoBrava Mississippi State University Can Advances in Speech Recognition make Spoken Language as Convenient and as Accessible as Online Text? Joseph Picone, PhD Professor, Electrical Engineering Mississippi State University Patti Price, PhD VP Business Development BravoBrava

3. Mississippi State University Outline Introduction and state of the art (Price) Research issues (Picone) –Evaluation metrics –Acoustic modeling –Language modeling –Practical issues –Technology demands Conclusion and future directions (Price)

4. BravoBrava Mississippi State University Introduction What is Speech Recognition? Speech Recognition Words “How are you?” Speech Signal Goal: Automatically extract the string of words spoken from the speech signal Speech recognition does NOT determine –Who is talker (speaker recognition, Heck and Reynolds) –Speech output (speech synthesis, Fruchterman and Ostendorf) –What the words mean (speech understanding)

5. BravoBrava Mississippi State University Introduction Speech in the Information Age Speech & text were revolutionary because of information access New media and connectivity yield information overload Can speech technology help? Time Source of Information SpeechText Film, video, multimedia, voice mail, radio, television, conferences, web, on-line resources Access to Information Listen, remember Read books Computer typing Careful spoken, written input Conversational language

6. BravoBrava Mississippi State University State of the Art Initial and Current Applications Database query –Resource management –Flight information –Stock quote 1997 Command and control –Manufacturing –Consumer products http://www.speech.be.philips.com/ Dictation –http://www.dragonsys.comhttp://www.dragonsys.com –http://www-4.ibm.com/software/speechhttp://www-4.ibm.com/software/speech Nuance, American Airlines: 1-800-433-7300, touch 1

7. BravoBrava Mississippi State University State of the Art How Do You Measure? What benchmarks? Have other systems used same one? What was training set? What was test set? Were training and test independent? How large was the vocabulary and the sample size? What speakers? What style speech? What kind of noise?

8. BravoBrava Mississippi State University all speakers of the language including foreign application independent or adaptive all styles including human-human (unaware) wherever speech occurs 2005 State of the Art Factors that Affect Performance vehicle noise radio cell phones regional accents native speakers competent foreign speakers some application– specific data and one engineer year natural human- machine dialog (user can adapt) 2000 expert years to create app– specific language model speaker independent and adaptive normal office various microphones telephone planned speech 1995 NOISE ENVIRONMENT SPEECH STYLE USER POPULATION COMPLEXITY 1985 quiet room fixed high – quality mic careful reading speaker- dep. application – specific speech and language

9. BravoBrava Mississippi State University Spontaneous telephone speech is still a “grand challenge”. Telephone-quality speech is still central to the problem. Vision for speech technology continues to evolve. Broadcast news is a very dynamic domain. 0% 10% 30% 40% 20% Word Error Rate Level Of Difficulty Digits Continuous Digits Command and Control Letters and Numbers Broadcast News Read Speech Conversational Speech Evaluation Metrics Evolution

10. BravoBrava Mississippi State University 0% 5% 15% 20% 10% 10 dB16 dB22 dB Quiet Wall Street Journal (Additive Noise) Machines Human Listeners (Committee) Word Error Rate Speech-To-Noise Ratio Human performance exceeds machine performance by a factor ranging from 4x to 10x depending on the task. On some tasks, such as credit card number recognition, machine performance exceeds humans due to human memory retrieval capacity. The nature of the noise is as important as the SNR (e.g., cellular phones). A primary failure mode for humans is inattention. A second major failure mode is the lack of familiarity with the domain (i.e., business terms and corporation names). Evaluation Metrics Human Performance

11. BravoBrava Mississippi State University 100% 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 1% 10% Read Speech 1k 5k 20k Noisy Varied Microphones Spontaneous Speech Conversational Speech Broadcast Speech (Foreign) 10 X Common evaluations fuel technology development. Tasks become progressively more ambitious and challenging. A Word Error Rate (WER) below 10% is considered acceptable. Performance in the field is typically 2x to 4x worse than performance on an evaluation. Evaluation Metrics Machine Performance

12. BravoBrava Mississippi State University Information extraction is the analysis of natural language to collect information about specified types of entities. As the focus shifts to providing enhanced annotations, WER may not be the most appropriate measure of performance (content-based scoring). F-Measure 0%10%20% 30% 70% 90% 80% 100% Word Error Rate (Hub-4 Eval’98) Evaluation Metrics Beyond WER: Named Entity Recall = # slots correctly filled # slots filled in key Precision = # slots correctly filled # slots filled by system F-Measure = 2 x recall x precision (recall + precision) An example of named entity annotation: Mr. Sears bought a new suit at Sears in Washington yesterday Evaluation Metrics:

13. BravoBrava Mississippi State University Our measurements of the signal are ambiguous. Region of overlap represents classification errors. Reduce overlap by introducing acoustic and linguistic context (e.g., context-dependent phones). Feature No. 1 Feature No. 2 Ph_1 Ph_3 Ph_2 Comparison of “aa” in “IOck” vs. “iy” in bEAt for conversational speech (SWB) Recognition Architectures Why Is Speech Recognition So Difficult?

14. BravoBrava Mississippi State University Message Source Linguistic Channel Articulatory Channel Acoustic Channel Observable: MessageWordsSounds Features Bayesian formulation for speech recognition: P(W|A) = P(A|W) P(W) / P(A) Recognition Architectures A Communication Theoretic Approach Objective: minimize the word error rate Approach: maximize P(W|A) during training Components: P(A|W) : acoustic model (hidden Markov models, mixtures) P(W) : language model (statistical, finite state networks, etc.) The language model typically predicts a small set of next words based on knowledge of a finite number of previous words (N-grams).

15. BravoBrava Mississippi State University Input Speech Recognition Architectures Incorporating Multiple Knowledge Sources Acoustic Front-end Acoustic Front-end The signal is converted to a sequence of feature vectors based on spectral and temporal measurements. Acoustic Models P(A/W) Acoustic Models P(A/W) Acoustic models represent sub-word units, such as phonemes, as a finite- state machine in which states model spectral structure and transitions model temporal structure. Recognized Utterance Search Search is crucial to the system, since many combinations of words must be investigated to find the most probable word sequence. The language model predicts the next set of words, and controls which models are hypothesized. Language Model P(W)

16. BravoBrava Mississippi State University Fourier Transform Fourier Transform Cepstral Analysis Cepstral Analysis Perceptual Weighting Perceptual Weighting Time Derivative Time Derivative Time Derivative Time Derivative Energy + Mel-Spaced Cepstrum Delta Energy + Delta Cepstrum Delta-Delta Energy + Delta-Delta Cepstrum Input Speech Incorporate knowledge of the nature of speech sounds in measurement of the features. Utilize rudimentary models of human perception. Acoustic Modeling Feature Extraction Measure features 100 times per sec. Use a 25 msec window for frequency domain analysis. Include absolute energy and 12 spectral measurements. Time derivatives to model spectral change.

17. BravoBrava Mississippi State University Acoustic models encode the temporal evolution of the features (spectrum). Gaussian mixture distributions are used to account for variations in speaker, accent, and pronunciation. Phonetic model topologies are simple left-to-right structures. Skip states (time-warping) and multiple paths (alternate pronunciations) are also common features of models. Sharing model parameters is a common strategy to reduce complexity. Acoustic Modeling Hidden Markov Models

18. BravoBrava Mississippi State University Closed-loop data-driven modeling supervised only from a word-level transcription The expectation/maximization (EM) algorithm is used to improve our parameter estimates. Computationally efficient training algorithms (Forward-Backward) have been crucial. Batch mode parameter updates are typically preferred. Decision trees are used to optimize parameter-sharing, system complexity, and the use of additional linguistic knowledge. Acoustic Modeling Parameter Estimation Initialization Single Gaussian Estimation 2-Way Split Mixture Distribution Reestimation 4-Way Split Reestimation

19. BravoBrava Mississippi State University Language Modeling Is A Lot Like Wheel of Fortune

20. BravoBrava Mississippi State University Language Modeling N-Grams: The Good, The Bad, and The Ugly Bigrams (SWB): Most Common: “you know”, “yeah SENT!”, “!SENT um-hum”, “I think” Rank-100: “do it”, “that we”, “don’t think” Least Common:“raw fish”, “moisture content”, “Reagan Bush” Trigrams (SWB): Most Common: “!SENT um-hum SENT!”, “a lot of”, “I don’t know” Rank-100: “it was a”, “you know that” Least Common:“you have parents”, “you seen Brooklyn” Unigrams (SWB): Most Common: “I”, “and”, “the”, “you”, “a” Rank-100: “she”, “an”, “going” Least Common: “Abraham”, “Alastair”, “Acura”

21. BravoBrava Mississippi State University Language Modeling Integration of Natural Language Natural language constraints can be easily incorporated. Lack of punctuation and search space size pose problems. Speech recognition typically produces a word-level time-aligned annotation. Time alignments for other levels of information also available.

22. BravoBrava Mississippi State University Typical LVCSR systems have about 10M free parameters, which makes training a challenge. Large speech databases are required (several hundred hours of speech). Tying, smoothing, and interpolation are required. Implementation Issues Search Is Resource Intensive

23. BravoBrava Mississippi State University Dynamic programming is used to find the most probable path through the network. Beam search is used to control resources. Implementation Issues Dynamic Programming-Based Search Search is time synchronous and left-to-right. Arbitrary amounts of silence must be permitted between each word. Words are hypothesized many times with different start/stop times, which significantly increases search complexity.

24. BravoBrava Mississippi State University Cross-word Decoding: since word boundaries don’t occur in spontaneous speech, we must allow for sequences of sounds that span word boundaries. Cross-word decoding significantly increases memory requirements. Implementation Issues Cross-Word Decoding Is Expensive

25. BravoBrava Mississippi State University Implementation Issues Decoding Example

26. BravoBrava Mississippi State University Implementation Issues Internet-Based Speech Recognition

27. BravoBrava Mississippi State University Technology Conversational Speech Conversational speech collected over the telephone contains background noise, music, fluctuations in the speech rate, laughter, partial words, hesitations, mouth noises, etc. WER has decreased from 100% to 30% in six years. Laughter Singing Unintelligible Spoonerism Background Speech No pauses Restarts Vocalized Noise Coinage

28. BravoBrava Mississippi State University Technology Audio Indexing of Broadcast News Broadcast news offers some unique challenges: Lexicon: important information in infrequently occurring words Acoustic Modeling: variations in channel, particularly within the same segment (“ in the studio” vs. “on location”) Language Model: must adapt (“ Bush,” “Clinton,” “Bush,” “McCain,” “???”) Language: multilingual systems? language-independent acoustic modeling?

29. BravoBrava Mississippi State University From President Clinton’s State of the Union address (January 27, 2000): “These kinds of innovations are also propelling our remarkable prosperity... Soon researchers will bring us devices that can translate foreign languages as fast as you can talk... molecular computers the size of a tear drop with the power of today’s fastest supercomputers.” Technology Real-Time Translation Imagine a world where: You book a travel reservation from your cellular phone while driving in your car without ever talking to a human (database query) You converse with someone in a foreign country and neither speaker speaks a common language (universal translator) You place a call to your bank to inquire about your bank account and never have to remember a password (transparent telephony) You can ask questions by voice and your Internet browser returns answers to your questions (intelligent query) Human Language Engineering: a sophisticated integration of many speech and language related technologies... a science for the next millennium.

30. BravoBrava Mississippi State University What have we learned? supervised training is a good machine learning technique large databases are essential for the development of robust statistics What are the challenges? discrimination vs. representation generalization vs. memorization pronunciation modeling human-centered language modeling What are the algorithmic issues for the next decade: Better features by extracting articulatory information? Bayesian statistics? Bayesian networks? Decision Trees? Information-theoretic measures? Nonlinear dynamics? Chaos? Technology Future Directions 1970 Hidden Markov Models Analog Filter Banks Dynamic Time-Warping 1980 1990 2000 1960

31. BravoBrava Mississippi State University To Probe Further References Journals and Conferences: [1] N. Deshmukh, et. al., “Hierarchical Search for LargeVocabulary Conversational Speech Recognition,” IEEE Signal Processing Magazine, vol. 1, no. 5, pp. 84- 107, September 1999. [2] N. Deshmukh, et. al., “Benchmarking Human Performance for Continuous Speech Recognition,” Proceedings of the Fourth International Conference on Spoken Language Processing, pp. SuP1P1.10, Philadelphia, Pennsylvania, USA, October 1996. [3] R. Grishman, “Information Extraction and Speech Recognition,” presented at the DARPA Broadcast News Transcription and Understanding Workshop, Lansdowne, Virginia, USA, February 1998. [4] R. P. Lippmann, “Speech Recognition By Machines and Humans,” Speech Communication, vol. 22, pp. 1-15, July 1997. [5] M. Maybury (editor), “News on Demand,” Communications of the ACM, vol. 43, no. 2, February 2000. [6]D. Miller, et. al., “Named Entity Extraction from Broadcast News,” presented at the DARPA Broadcast News Workshop, Herndon, Virginia, USA, February 1999. [7] D. Pallett, et. al., “Broadcast News Benchmark Test Results,” presented at the DARPA Broadcast News Workshop, Herndon, Virginia, USA, February 1999. [8] J. Picone, “Signal Modeling Techniques in Speech Recognition,” IEEE Proceedings, vol. 81, no. 9, pp. 1215- 1247, September 1993. [9] P. Robinson, et. al., “Overview: Information Extraction from Broadcast News,” presented at the DARPA Broadcast News Workshop, Herndon, Virginia, USA, February 1999. [10] F. Jelinek, Statistical Methods for Speech Recognition, MIT Press, 1998. URLs and Resources: [11] “Speech Corpora,” The Linguistic Data Consortium, http://www.ldc.upenn.edu. [12] “Technology Benchmarks,” Spoken Natural Language Processing Group, The National Institute for Standards, http://www.itl.nist.gov/iaui/894.01/index.html. [13] “Signal Processing Resources,” Institute for Signal and Information Technology, Mississippi State University, http://www.isip.msstate.edu. [14] “Internet- Accessible Speech Recognition Technology,” http://www.isip.msstate.edu/projects/speech/index.html. [15] “A Public Domain Speech Recognition System,” http://www.isip.msstate.edu/projects/speech/software/index.html. [16] “Remote Job Submission,” http://www.isip.msstate.edu/projects/speech/experiments/index.html. [17] “The Switchboard Corpus,” http://www.isip.msstate.edu/projects/switchboard/index.html.

32. BravoBrava Mississippi State University Conclusion and Future Directions Trends We need new technology to help with information overload Speech information sources are everywhere –Voice mail messages –Professional talk –Lectures, broadcasts Speech sources of information will increase –As devices shrink –As mobility increases –New uses: annotation, documentation Speech as AccessSpeech as SourceInformation as Partner What are the words?What does it mean?Here’s what you need.

33. BravoBrava Mississippi State University Conclusion and Future Directions Limitations on Applications Recognition performance, especially in error recovery Natural language understanding (speech differs from text) –Speech unfolds linearly in time –Speech is more indeterminate than text –Speech has different syntax and semantics –Prosody differs from punctuation Cost to develop applications (too few experts) Cost to integrate/interoperate with other technologies New capabilities –"When did he say Y and was he angry?” –Scanning, refocusing quickly (browsing) –Proactive information: Match past pattern, find novel aspects –Gist, summarize, translate for different purposes

34. BravoBrava Mississippi State University Conclusion and Future Directions Applications on the Horizon Beginnings of speech as source of information ISLIP http://www.mediasite.net/info/frames.htmhttp://www.mediasite.net/info/frames.htm Virage http://www.virage.comhttp://www.virage.com Why doesn’t belong in the classroom Beulah Arnott: also true of indoor plumbing BravoBrava: Co-evolving technology and people can – Dramatically reduce the cost of delivery of content – Increase its timeliness, quality and appropriateness – Target needs of individual and/or group – Reading Pal demo Speech technology in education and training Cliff Stoll, High Tech Heretic –Good schools need no computers –Bad schools won’t be improved by them

35. BravoBrava Mississippi State University Reading Pal Child reads Errors in red Looks up word “massive” Clicks ‘You’ to play it back Clicks ‘Listen’ To play back from “massive”

36. BravoBrava Mississippi State University Summary Goal: Speech Better Than Text Healthy loop between research and applications Research leads to applications, which lead to new research opportunities We need collaboration Too much for one person, one site, one country Humans will probably continue to be better than machines at many things Can we learn to use technology and training to augment human-human and human-machine collaboration? We need to “micronize” education and training It’s not a solved problem Further technology development is needed to enable the vision