1. Automatic Speech Recognition
Lecture 7
Automatic Speech Recognition
CS 4705

2. What is speech recognition?
Transcribing words?
Understanding meaning?
Today:
Overview ASR issues
Building an ASR system
Using an ASR system
Future research

3. “It’s hard to ... recognize speech/wreck a nice beach”
Speaker variability: within and across
Recording environment varies wrt noise
Transcription task must handle all of this and produce a transcript of what was said, from limited, noisy information in the speech signal
Success: low word error rate (WER)
WER = (S+I+D)/N * 100
Thesis test vs. This is a test. 75% WER
Understanding task must do more: from words to meaning

4. Measure concept accuracy (CA) of string in terms of accuracy of recognition of domain concepts mentioned in string and their values
I want to go from Boston to Baltimore on September 29
Domain concepts Values
source city Boston
target city Baltimore
travel date September 29
Score recognized string “Go from Boston to Washington on December 29” (1/3 = 33% CA)
“Go to Boston from Baltimore on September 29”

5. Again, the Noisy Channel Model
Source
Noisy Channel
Decoder
Input to channel: spoken sentence s
Output from channel: an observation O
Decoding task: find s = P(s|O)
Using Bayes Rule
And since P(O) doesn’t change for any hypothetical s’
s’ = P(O|s) P(s)
P(O|s) is the observation likelihood, or Acoustic Model, and P(s) is the prior, or Language Model

6. What do we need to build use an ASR system?
Corpora for training and testing of components
Feature extraction component
Pronunciation Model
Acoustic Model
Language Model
Algorithms to search hypothesis space efficiently

7. Training and Test Corpora
Collect corpora appropriate for recognition task at hand
Small speech + phonetic transcription to associate sounds with symbols (Acoustic Model)
Large (>= 60 hrs) speech + orthographic transcription to associate words with sounds (Acoustic Model)
Very large text corpus to identify unigram and bigram probabilities (Language Model)

8. Representing the Signal
What parameters (features) of the speech input
Can be extracted automatically
Will preserve phonetic identity and distinguish it from other phones
Will be independent of speaker variability and channel conditions
Will not take up too much space
Speech representations (for [ae] in had):
Waveform: change in sound pressure over time
LPC Spectrum: component frequencies of a waveform
Spectrogram: overall view of how frequencies change from phone to phone

9. Signal divided into frames
Speech captured by microphone and sampled (digitized) -- may not capture all vital information
Signal divided into frames
Power spectrum computed to represent energy in different bands of the signal
LPC spectrum, Cepstra, PLP
Each frame’s spectral features represented by small set of numbers
Frames clustered into ‘phone-like’ groups (phones in context) -- Gaussian or other models

10. Why this works?
Different phonemes have different spectral characteristics
Why it doesn’t work?
Phonemes can have different properties in different acoustic contexts, spoken by different people …
Nice white rice

11. Pronunciation Model
Models likelihood of word given network of candidate phone hypotheses (weighted phone lattice)
Allophones: butter vs. but
Multiple pronunciations for each word
Lexicon may be weighted automaton or simple dictionary
Words come from all corpora; pronunciations from pronouncing dictionary or TTS system

12. Acoustic Models
Model likelihood of phones or subphones given spectral features and prior context
Use pronunciation models
Usually represented as HMM
Set of states representing phones or other subword units
Transition probabilities on states: how likely is it to see one phone after seeing another?
Observation/output likelihoods: how likely is spectral feature vector to be observed from phone state i, given phone state i-1?

13. Initial estimates for
Transition probabilities between phone states
Observation probabilities associating phone states with acoustic examples
Re-estimate both probabilities by feeding the HMM the transcribed speech training corpus (forced alignment)
I.e., we tell the HMM the ‘right’ answers -- which words to associate with which sequences of sounds
Iteratively retrain the transition and observation probabilities by running the training data through the model and scoring output until no improvement

14. Language Model
Models likelihood of word given prior word and of entire sentence
Ngram models:
Build the LM by calculating bigram or trigram probabilities from text training corpus
Smoothing issues very important for real systems
Grammars
Finite state grammar or Context Free Grammar (CFG) or semantic grammar
Out of Vocabulary (OOV) problem

15. Entropy H(X): the amount of information in a LM, grammar
How many bits will it take on average to encode a choice or a piece of information?
More likely things will take fewer bits to encode
Perplexity 2H: a measure of the weighted mean number of choice points in e.g. a language model

16. Search/Decoding Find the best hypothesis P(O|s) P(s) given
Lattice of subword units (Acoustic Model)
Segmentation of all paths into possible words (Pronunciation Model)
Probabilities of word sequences (Language Model)
Produces a huge search space: How to reduce?
Lattice minimization and determinization
Forward algorithm: sum of all paths leading to a state
Viterbi algorithm: max of all paths leading to a state

17. Forward-backward (Baum-Welch, Expectation-Maximization) algorithm: computes probability of sequence at any state in search space
Beam search: prune the lattice

18. Varieties of Speech Recognition
Telephone, microphones
Input device
> 30dB (high)  < 10dB (low)
SNR
<10  > 100
Perplexity
Finite state, ngrams, CFGs, CSGs
Language Model
<20  5K --> 60K -->~1M
Vocabulary size
Speaker-dependent or independent
Enrollment
Read, prepared, spontaneous
Style
Isolated words  continuous
Mode

19. Challenges for Transcription
Robustness to channel characteristics and noise
Portability to new applications
Adapatation: to speakers, to environments
LMs: simple ngrams need help
Confidence measures
OOV words
New speaking styles/genres
New applications

20. Challenges for Understanding
Recognizing communicative ‘problems’
ASR errors
User corrections
Disfluencies and self-repairs
Possibilities:
Recognizing speaker emotion
Identifying speech acts: okay
Locating topic boundaries for topic tracking, audio browsing, speech data mining

21. An Unsuccessful Dialogue
S: Hi, this is AT&T Amtrak schedule system. This is TOOT. How may I help you?
U: I'd like to find a train from Ch- to Chicago from Baltimore.
S: Sorry, I can't understand you. Please repeat your utterance.
U: I'd like to find a train from Baltimore to Chicago.
S: I heard you say “leave today around 5 thirty in the morning .“ Which city do you want to go to?
U: I would like to leave Saturday at eight in the morning.
Note that the previous speaker was not a native speaker of English -- non-native speakers are often harder for recognizers to understand. Here however is a native speaker who has a terrible time with TOOT -- which never seems to understand ‘Baltimore’. Note the different ways she produces that city name during the dialogue. You’ll here some clear examples of hyperarticulation.
$TOOT/non_adapt/MixedImplicit/award/ -- TOOT can never understand Baltimore

22. S: You can say the arrival city name, such as “New York City."
U: Baltimore to Chicago.
S: You can say the arrival city name, such as “New York City.”
U: Baltimore to Chicago. (I hate it.)
U: Train from Baltimore ...S: ...I heard you s...U: to Chicago.... Hello?
S: You can say the name of your departure city, such as "New York City."
U: Baltimore.
Note how she changes her pronunciation of “Baltimore”, hoping TOOT will finally here it clearly enough!

23. U: Bal-ti-more....
S: You can say...
U: Baltimore.
S: Do you want to continue and find a new set of trains?
U: Yes.
S: You can say "yes" or "I am done here" to find a new set of trains.
S: Thank you for using AT&T Amtrak train time table system.
See you next time.
U: I nev-

24. Summary
ASR technology relies upon a large number of phenomena and techniques we’ve already seen to convert sound into words
Phonetic/phonological, morphological, and lexical events
FSA’s, Ngrams, Dynamic programming algorithms
Better modeling of linguistic phenomena will be needed to improve performance on transcription and especially on understanding
For next class: we’ll start talking about larger structures in language above the word (Ch 8)

25. Disfluencies and Self-Repairs
Disfluencies abound in spontaneous speech
every 4.6s in radio call-in (Blackmer & Mitton ‘91)
hesitation: Ch- change strategy.
filled pause: Um Baltimore.
self-repair: Ba- uh Chicago.
Hard to recognize
Ch- change strategy. --> to D C D C today ten fifteen.
Um Baltimore. --> From Baltimore ten.
Ba- uh Chicago. --> For Boston Chicago.
Kasl & Mahl: 41% more filled pauses in audio only vs ftf; Oviatt: 8.83 to 5.50% disfluencies in phone conversations vs. non
/n/u118/exp98/adapt/MixedImplicit/mccoy/task1 line 198, rehesitation (mor- morning --> not really)
/n/u118/exp98/adapt/UserNoConfirm/sgoel/task1 line 1315, filled pause (um baltimore --> from baltimoreten)
/n/u118/exp98/adapt/UserNoConfirm/selina/task1, line 1132, repair (Ba- uh Chicago --> for Boston Chicago)