1. CS 188: Artificial Intelligence Spring 2007 Speech Recognition 03/20/2007 Srini Narayanan – ICSI and UC Berkeley

2. Announcements  Midterm graded  Median 78  Mean 75  5 100+s  25 % above 90  HW 5 up today  BN inference  Due 4/9 (2 weeks + Spring break) < 508 50 - 6012 60 – 7017 70 - 8011 80 - 9017 90 - 10018 > 1004

3. Hidden Markov Models  Hidden Markov models (HMMs)  Underlying Markov chain over states X  You observe outputs (effects) E at each time step  As a Bayes’ net:  Several questions you can answer for HMMs:  Last time: filtering to track belief about current X given evidence  Last time: Vitterbi estimation to compute most likely sequence X5X5 X2X2 E1E1 X1X1 X3X3 X4X4 E2E2 E3E3 E4E4 E5E5

4. Real HMM Examples  Speech recognition HMMs:  Observations are acoustic signals (continuous valued)  States are specific positions in specific words (so, tens of thousands)  Machine translation HMMs:  Observations are words (tens of thousands)  States are translation positions (dozens)  Robot tracking:  Observations are range readings (continuous)  States are positions on a map (continuous)

5. The Speech Recognition Problem  We want to predict a sentence given an acoustic sequence:  The noisy channel approach:  Build a generative model of production (encoding)  To decode, we use Bayes’ rule to write  Now, we have to find a sentence maximizing this product

6. The noisy channel model  Ignoring the denominator leaves us with two factors: P(Source) and P(Signal|Source)

7. Acoustic Modeling Describes the sounds that make up speech Lexicon Describes which sequences of speech sounds make up valid words Language Model Describes the likelihood of various sequences of words being spoken Speech Recognition Speech Recognition Knowledge Sources

8. Digitizing Speech

9. Speech in an Hour  Speech input is an acoustic wave form s p ee ch l a b Graphs from Simon Arnfield’s web tutorial on speech, Sheffield: http://www.psyc.leeds.ac.uk/research/cogn/speech/tutorial/ “l” to “a” transition:

10. She just had a baby  What can we learn from a wavefile?  Vowels are voiced (vocal cord vibrates), long, loud  Length in time = length in space in waveform picture  Voicing: regular peaks in amplitude  When stops closed: no peaks: silence.  Peaks = voicing:.46 to.58 (vowel [i], from second.65 to.74 (vowel [u]) and so on  Silence of stop closure (1.06 to 1.08 for first [b], or 1.26 to 1.28 for second [b])  Fricatives (f tongue hits upper teeth) like [  ] intense irregular pattern; see.33 to.46

11.  Frequency gives pitch; amplitude gives volume  sampling at ~8 kHz phone, ~16 kHz mic (kHz=1000 cycles/sec)  Fourier transform of wave displayed as a spectrogram  darkness indicates energy at each frequency s p ee ch l a b frequency amplitude Spectral Analysis

12. Adding 100 Hz + 1000 Hz Waves

13. Spectrum 100 1000 Frequency in Hz Amplitude Frequency components (100 and 1000 Hz) on x-axis

14. Part of [ae] from “had”  Note complex wave repeating nine times in figure  Plus smaller waves which repeats 4 times for every large pattern  Large wave has frequency of 250 Hz (9 times in.036 seconds)  Small wave roughly 4 times this, or roughly 1000 Hz  Two little tiny waves on top of peak of 1000 Hz waves

15. Back to Spectra  Spectrum represents these freq components  Computed by Fourier transform, algorithm which separates out each frequency component of wave.  x-axis shows frequency, y-axis shows magnitude (in decibels, a log measure of amplitude)  Peaks at 930 Hz, 1860 Hz, and 3020 Hz.

16. Vowel Formants

17. Resonances of the vocal tract  The human vocal tract as an open tube  Air in a tube of a given length will tend to vibrate at resonance frequency of tube.  Constraint: Pressure differential should be maximal at (closed) glottal end and minimal at (open) lip end. Closed end Open end Length 17.5 cm. Figure from W. Barry Speech Science slides

18. From Mark Liberman’s website

19. Articulation Process  Articulatory facts:  Vocal cord vibrations create harmonics  The mouth is a selective amplifier  Depending on shape of mouth, some harmonics are amplified more than others

20. Figures from Ratree Wayland slides from his website Vowel [i] sung at successively higher pitch. 1 2 3 4 5 6 7

21. How to read spectrograms  bab: closure of lips lowers all formants: so rapid increase in all formants at beginning of "bab ”  dad: first formant increases, but F2 and F3 slight fall  gag: F2 and F3 come together: this is a characteristic of velars. Formant transitions take longer in velars than in alveolars or labials From Ladefoged “A Course in Phonetics”

22. Final Feature Vector  39 (real) features per 10 ms frame:  12 MFCC features  12 Delta MFCC features  12 Delta-Delta MFCC features  1 (log) frame energy  1 Delta (log) frame energy  1 Delta-Delta (log frame energy)  So each frame is represented by a 39D vector  (You don’t have to know these details!)

23. Acoustic Feature Sequence  Time slices are translated into acoustic feature vectors (~39 real numbers per slice)  These are the observations, now we need the hidden states X frequency …………………………………………….. e 12 e 13 e 14 e 15 e 16 ………..

24. State Space  P(E|X) encodes which acoustic vectors are appropriate for each phoneme (each kind of sound)  P(X|X’) encodes how sounds can be strung together  We will have one state for each sound in each word  From some state x, can only:  Stay in the same state (e.g. speaking slowly)  Move to the next position in the word  At the end of the word, move to the start of the next word  We build a little state graph for each word and chain them together to form our state space X

25. HMMs for Speech

26. Phones are not homogeneous!

27. Each phone has 3 subphones

28. Resulting HMM word model for “six”

29. ASR Lexicon: Markov Models

31. Speech Architecture meets Noisy Channel

32. Search space with bigrams

33. Markov Process with Bigrams Figure from Huang et al page 618

34. Decoding  While there are some practical issues, finding the words given the acoustics is an HMM inference problem  Here the state sequence is the sequence of phones  Observations are the acoustic vectors  We want to know which state sequence x 1:T is most likely given the evidence e 1:T :

35. Viterbi Algorithm  Question: what is the most likely state sequence given the observations?  Slow answer: enumerate all possibilities  Better answer: cached incremental version

36. Viterbi with 2 Words + Unif. LM Figure from Huang et al page 612

37. It's not easy to wreck a nice beach. It's not easy to recognize speech.

38. © Copyright 2002 Michael G. Christel and Alexander G. Hauptmann 38 Carnegie Mellon Continual Progress in Speech Recognition Increasingly Difficult Tasks, Steadily Declining Error Rates CONVERSATIONAL SPEECH Non-English English BROADCAST NEWS 20,000 Word Varied microphones Standard microphone Noisy environment Unlimited Vocabulary 5000 word All results are Speaker -Independent READ SPEECH 1000 Word vocabulary 100 50 10 1 Word Error Rate (%) 1988 198919901991 1992 1993 19941995199619971998 NSA/Wayne/Doddington

39. Current error rates 20.764,000+Conversational Telephone 9.964,000+Broadcast news <6.620KWSJ read speech 3.05KWSJ read speech 0.5511Digits Error Rate%VocabularyTask

40. Example

41. Dynamic Bayes Nets DBN = Multiple Hidden State Variables. Each State is a BN

42. Structured Probabilistic Inference

43. Next Class  Next part of the course: machine learning  We’ll start talking about how to learn model parameters (like probabilities) from data  One of the most heavily used technologies in all of AI

44. Extra Slides

45. Examples from Ladefoged bad pad spat

46. Simple Periodic Sound Waves  Y axis: Amplitude = amount of air pressure at that point in time  Zero is normal air pressure, negative is rarefaction  X axis: time. Frequency = number of cycles per second.  Frequency = 1/Period  20 cycles in.02 seconds = 1000 cycles/second = 1000 Hz

47. Deriving Schwa  Reminder of basic facts about sound waves  f = c/  c = speed of sound (approx 35,000 cm/sec)  A sound with =10 meters: f = 35 Hz (35,000/1000)  A sound with =2 centimeters: f = 17,500 Hz (35,000/2)

48. From Sundberg

49. Computing the 3 Formants of Schwa  Let the length of the tube be L  F 1 = c/ 1 = c/(4L) = 35,000/4*17.5 = 500Hz  F 2 = c/ 2 = c/(4/3L) = 3c/4L = 3*35,000/4*17.5 = 1500Hz  F 1 = c/ 2 = c/(4/5L) = 5c/4L = 5*35,000/4*17.5 = 2500Hz  So we expect a neutral vowel to have 3 resonances at 500, 1500, and 2500 Hz  These vowel resonances are called formants

50. HMMs for Continuous Observations?  Before: discrete, finite set of observations  Now: spectral feature vectors are real-valued!  Solution 1: discretization  Solution 2: continuous emissions models  Gaussians  Multivariate Gaussians  Mixtures of Multivariate Gaussians  A state is progressively:  Context independent subphone (~3 per phone)  Context dependent phone (=triphones)  State-tying of CD phone

51. Viterbi Decoding