0. Speech recognition, machine learning
Lirong Xia
Friday, April 4, 2014

1. Particle Filtering Sometimes |X| is too big to use exact inference
|X| may be too big to even store B(X)
E.g. X is continuous
|X|2 may be too big to do updates
Solution: approximate inference
Track samples of X, not all values
Samples are called particles
Time per step is linear in the number of samples
But: number needed may be large
In memory: list of particles
This is how robot localization works in practice

2. Forward algorithm vs. particle filtering
Elapse of time
B’(Xt)=Σxt-1p(Xt|xt-1)B(xt-1)
Observe
B(Xt) ∝p(et|Xt)B’(Xt)
Renormalize
B(xt) sum up to 1
Elapse of time
x--->x’
Observe
w(x’)=p(et|x)
Resample
resample N particles

3. Dynamic Bayes Nets (DBNs)
We want to track multiple variables over time, using multiple sources of evidence
Idea: repeat a fixed Bayes net structure at each time
Variables from time t can condition on those from t-1
DBNs with evidence at leaves are HMMs

4. HMMs: MLE Queries HMMs defined by: Query: most likely explanation:
States X
Observations E
Initial distribution: p(X1)
Transitions: p(X|X-1)
Emissions: p(E|X)
Query: most likely explanation:

5. Viterbi Algorithm

6. Today Speech recognition Start to learn machine learning
A massive HMM!
Details of this section not required
CSCI 4962: Natural language processing by Prof. Heng Ji
Start to learn machine learning
CSCI 4100: machine learning by Prof. Malik Magdon-Ismail

7. Speech and Language Speech technologies
Automatic speech recognition (ASR)
Text-to-speech synthesis (TTS)
Dialog systems
Language processing technologies
Machine translation
Information extraction
Web search, question answering
Text classification, spam filtering, etc…

8. Digitizing Speech

9. The Input Speech input is an acoustic wave form
Graphs from Simon Arnfield’s web tutorial on speech, sheffield:

11. The Input Frequency gives pitch; amplitude gives volume
Sampling at ~8 kHz phone, ~16 kHz mic
Fourier transform of wave displayed as a spectrogram
Darkness indicates energy at each frequency

12. Acoustic Feature Sequence
Time slices are translated into acoustic feature vectors (~39 real numbers per slice)
These are the observations, now we need the hidden states X

13. State Space
p(E|X) encodes which acoustic vectors are appropriate for each phoneme (each kind of sound)
p(X|X’) encodes how sounds can be strung together
We will have one state for each sound in each word
From some state x, can only:
Stay in the same state (e.g. speaking slowly)
Move to the next position in the word
At the end of the word, move to the start of the next word
We build a little state graph for each word and chain them together to form our state space X

14. HMMs for Speech

15. Transitions with Bigrams

16. Decoding
While there are some practical issues, finding the words given the acoustics is an HMM inference problem
We want to know which state sequence x1:T is most likely given the evidence e1:T:
From the sequence x, we can simply read off the words

17. Machine Learning
Up until now: how to reason in a model and how make optimal decisions
Machine learning: how to acquire a model on the basis of data / experience
Learning parameters (e.g. probabilities)
Learning structure (e.g. BN graphs)
Learning hidden concepts (e.g. clustering)

18. Parameter Estimation Estimating the distribution of a random variable
Elicitation: ask a human (why is this hard?)
Empirically: use training data (learning!)
E.g.: for each outcome x, look at the empirical rate of that value:
This is the estimate that maximizes the likelihood of the data

19. Estimation: Smoothing
Relative frequencies are the maximum likelihood estimates (MLEs)
In Bayesian statistics, we think of the parameters as just another random variable, with its own distribution
????

20. Estimation: Laplace Smoothing
Laplace’s estimate:
Pretend you saw every outcome once more than you actually did
Can derive this as a MAP estimate with Dirichlet priors

21. Estimation: Laplace Smoothing
Laplace’s estimate (extended):
Pretend you saw every outcome k extra times
What’s Laplace with k=0?
k is the strength of the prior
Laplace for conditionals:
Smooth each condition independently:

22. Example: Spam Filter Input: email Output: spam/ham Setup:
Get a large collection of example s, each labeled “spam” or “ham”
Note: someone has to hand label all this data!
Want to learn to predict labels of new, future s
Features: the attributes used to make the ham / spam decision
Words: FREE!
Text patterns: $dd, CAPS
Non-text: senderInContacts ……

23. Example: Digit Recognition
Input: images / pixel grids
Output: a digit 0-9
Setup:
Get a large collection of example images, each labeled with a digit
Note: someone has to hand label all this data!
Want to learn to predict labels of new, future digit images
Features: the attributes used to make the digit decision
Pixels: (6,8) = ON
Shape patterns: NumComponents, AspectRation, NumLoops ……

24. A Digit Recognizer
Input: pixel grids
Output: a digit 0-9

25. Naive Bayes for Digits Simple version: Naive Bayes model:
One feature Fij for each grid position <i,j>
Possible feature values are on / off, based on whether intensity is more or less than 0.5 in underlying image
Each input maps to a feature vector, e.g.
Here: lots of features, each is binary valued
Naive Bayes model:
What do we need to learn?

26. General Naive Bayes A general naive Bayes model:
We only specify how each feature depends on the class
Total number of parameters is linear in n

27. Inference for Naive Bayes
Goal: compute posterior over causes
Step 1: get joint probability of causes and evidence
Step 2: get probability of evidence
Step 3: renormalize

28. General Naive Bayes What do we need in order to use naive Bayes?
Inference (you know this part)
Start with a bunch of conditionals, p(Y) and the p(Fi|Y) tables
Use standard inference to compute p(Y|F1…Fn)
Nothing new here
Estimates of local conditional probability tables
p(Y), the prior over labels
p(Fi|Y) for each feature (evidence variable)
These probabilities are collectively called the parameters of the model and denoted by θ
Up until now, we assumed these appeared by magic, but…
… they typically come from training data: we’ll look at this now

29. Examples: CPTs 1
0.1 2 3 4 5 6 7 8 9 1
0.01 2
0.05 3 4
0.30 5
0.80 6
0.90 7 8
0.60 9
0.50 1
0.05 2
0.01 3
0.90 4
0.80 5 6 7
0.25 8
0.85 9
0.60

30. Important Concepts
Data: labeled instances, e.g. s marked spam/ham
Training set
Held out set
Test set
Features: attribute-value pairs which characterize each x
Experimentation cycle
Learn parameters (e.g. model probabilities) on training set
(Tune hyperparameters on held-out set)
Compute accuracy of test set
Very important: never “peek” at the test set!
Evaluation
Accuracy: fraction of instances predicted correctly
Overfitting and generalization
Want a classifier which does well on test data
Overfitting: fitting the training data very closely, but not generalizing well
We’ll investigate overfitting and generalization formally in a few lectures