Speech recognition, machine learning Lirong Xia Friday, April 4, 2014

Particle Filtering Sometimes |X| is too big to use exact inference |X| may be too big to even store B(X) E.g. X is continuous |X|2 may be too big to do updates Solution: approximate inference Track samples of X, not all values Samples are called particles Time per step is linear in the number of samples But: number needed may be large In memory: list of particles This is how robot localization works in practice

Forward algorithm vs. particle filtering Elapse of time B’(Xt)=Σxt-1p(Xt|xt-1)B(xt-1) Observe B(Xt) ∝p(et|Xt)B’(Xt) Renormalize B(xt) sum up to 1 Elapse of time x--->x’ Observe w(x’)=p(et|x) Resample resample N particles

Dynamic Bayes Nets (DBNs) We want to track multiple variables over time, using multiple sources of evidence Idea: repeat a fixed Bayes net structure at each time Variables from time t can condition on those from t-1 DBNs with evidence at leaves are HMMs

HMMs: MLE Queries HMMs defined by: Query: most likely explanation: States X Observations E Initial distribution: p(X1) Transitions: p(X|X-1) Emissions: p(E|X) Query: most likely explanation:

Viterbi Algorithm

Today Speech recognition Start to learn machine learning A massive HMM! Details of this section not required CSCI 4962: Natural language processing by Prof. Heng Ji Start to learn machine learning CSCI 4100: machine learning by Prof. Malik Magdon-Ismail

Speech and Language Speech technologies Automatic speech recognition (ASR) Text-to-speech synthesis (TTS) Dialog systems Language processing technologies Machine translation Information extraction Web search, question answering Text classification, spam filtering, etc…

Digitizing Speech

The Input Speech input is an acoustic wave form Graphs from Simon Arnfield’s web tutorial on speech, sheffield: http://www.psyc.leeds.ac.uk/research/cogn/speech/tutorial/

The Input Frequency gives pitch; amplitude gives volume Sampling at ~8 kHz phone, ~16 kHz mic Fourier transform of wave displayed as a spectrogram Darkness indicates energy at each frequency

Acoustic Feature Sequence Time slices are translated into acoustic feature vectors (~39 real numbers per slice) These are the observations, now we need the hidden states X

State Space p(E|X) encodes which acoustic vectors are appropriate for each phoneme (each kind of sound) p(X|X’) encodes how sounds can be strung together We will have one state for each sound in each word From some state x, can only: Stay in the same state (e.g. speaking slowly) Move to the next position in the word At the end of the word, move to the start of the next word We build a little state graph for each word and chain them together to form our state space X

HMMs for Speech

Transitions with Bigrams

Decoding While there are some practical issues, finding the words given the acoustics is an HMM inference problem We want to know which state sequence x1:T is most likely given the evidence e1:T: From the sequence x, we can simply read off the words

Machine Learning Up until now: how to reason in a model and how make optimal decisions Machine learning: how to acquire a model on the basis of data / experience Learning parameters (e.g. probabilities) Learning structure (e.g. BN graphs) Learning hidden concepts (e.g. clustering)

Parameter Estimation Estimating the distribution of a random variable Elicitation: ask a human (why is this hard?) Empirically: use training data (learning!) E.g.: for each outcome x, look at the empirical rate of that value: This is the estimate that maximizes the likelihood of the data

Estimation: Smoothing Relative frequencies are the maximum likelihood estimates (MLEs) In Bayesian statistics, we think of the parameters as just another random variable, with its own distribution ????

Estimation: Laplace Smoothing Laplace’s estimate: Pretend you saw every outcome once more than you actually did Can derive this as a MAP estimate with Dirichlet priors

Estimation: Laplace Smoothing Laplace’s estimate (extended): Pretend you saw every outcome k extra times What’s Laplace with k=0? k is the strength of the prior Laplace for conditionals: Smooth each condition independently:

Example: Spam Filter Input: email Output: spam/ham Setup: Get a large collection of example emails, each labeled “spam” or “ham” Note: someone has to hand label all this data! Want to learn to predict labels of new, future emails Features: the attributes used to make the ham / spam decision Words: FREE! Text patterns: $dd, CAPS Non-text: senderInContacts ……

Example: Digit Recognition Input: images / pixel grids Output: a digit 0-9 Setup: Get a large collection of example images, each labeled with a digit Note: someone has to hand label all this data! Want to learn to predict labels of new, future digit images Features: the attributes used to make the digit decision Pixels: (6,8) = ON Shape patterns: NumComponents, AspectRation, NumLoops ……

A Digit Recognizer Input: pixel grids Output: a digit 0-9

Naive Bayes for Digits Simple version: Naive Bayes model: One feature Fij for each grid position <i,j> Possible feature values are on / off, based on whether intensity is more or less than 0.5 in underlying image Each input maps to a feature vector, e.g. Here: lots of features, each is binary valued Naive Bayes model: What do we need to learn?

General Naive Bayes A general naive Bayes model: We only specify how each feature depends on the class Total number of parameters is linear in n

Inference for Naive Bayes Goal: compute posterior over causes Step 1: get joint probability of causes and evidence Step 2: get probability of evidence Step 3: renormalize

General Naive Bayes What do we need in order to use naive Bayes? Inference (you know this part) Start with a bunch of conditionals, p(Y) and the p(Fi|Y) tables Use standard inference to compute p(Y|F1…Fn) Nothing new here Estimates of local conditional probability tables p(Y), the prior over labels p(Fi|Y) for each feature (evidence variable) These probabilities are collectively called the parameters of the model and denoted by θ Up until now, we assumed these appeared by magic, but… … they typically come from training data: we’ll look at this now

Examples: CPTs 1 0.1 2 3 4 5 6 7 8 9 1 0.01 2 0.05 3 4 0.30 5 0.80 6 0.90 7 8 0.60 9 0.50 1 0.05 2 0.01 3 0.90 4 0.80 5 6 7 0.25 8 0.85 9 0.60

Important Concepts Data: labeled instances, e.g. emails marked spam/ham Training set Held out set Test set Features: attribute-value pairs which characterize each x Experimentation cycle Learn parameters (e.g. model probabilities) on training set (Tune hyperparameters on held-out set) Compute accuracy of test set Very important: never “peek” at the test set! Evaluation Accuracy: fraction of instances predicted correctly Overfitting and generalization Want a classifier which does well on test data Overfitting: fitting the training data very closely, but not generalizing well We’ll investigate overfitting and generalization formally in a few lectures