Hidden Markov Models IP notice: slides from Dan Jurafsky

Outline Markov Chains Markov Chains Hidden Markov Models Hidden Markov Models Three Algorithms for HMMs Three Algorithms for HMMs  The Forward Algorithm  The Viterbi Algorithm  The Baum-Welch (EM Algorithm) Applications: Applications:  The Ice Cream Task  Part of Speech Tagging

Chomsky Grammars Distinguish grammatical English from ungrammatical English: Distinguish grammatical English from ungrammatical English:  John thinks Sara hit the boy  * The hit thinks Sara John boy  John thinks the boy was hit by Sara  Who does John think Sara hit?  John thinks Sara hit the boy and the girl  * Who does John think Sara hit the boy and?  John thinks Sara hit the boy with the bat  What does John think Sara hit the boy with?  Colorless green ideas sleep furiously.  * Green sleep furiously ideas colorless.

Acceptors and Transformers Chomsky’s grammars are about which utterances are acceptable Chomsky’s grammars are about which utterances are acceptable Other research programs are aimed at transforming utterances Other research programs are aimed at transforming utterances  Translate a English sentence into Japanese…  Transform a speech waveform into transcribed words…  Compress a sentence, summarize a text…  Transform a syntactic analysis into a semantic analysis…  Generate a text from a semantic representation…

Strings and Trees Early on, trees were realized to be a useful tool in describing what is grammatical Early on, trees were realized to be a useful tool in describing what is grammatical  A sentence is a noun phrase (NP) followed by a verb phrase (VP)  A noun phrase is a determiner (DT) followed by a noun (NN)  A noun phrase is a noun phrase (NP) followed by a prepositional phrase (PP)  A PP is a preposition (IN) followed by an NP A string is acceptable if it has an acceptable tree … A string is acceptable if it has an acceptable tree … Transformations may take place at the tree level … Transformations may take place at the tree level … S NPVP NPPP INNP

Natural Language Processing 1980s: Many tree-based grammatical formalisms 1980s: Many tree-based grammatical formalisms 1990s: Regression to string-based formalisms 1990s: Regression to string-based formalisms  Hidden Markov Models (HMM), Finite-State Acceptors (FSAs) and Transducers (FSTs)  N-gram models for accepting sentences [eg, Jelinek 90]  Taggers and other statistical transformations [eg, Church 88]  Machine translation [eg, Brown et al 93]  Software toolkits implementing generic weighted FST operations [eg, Mohri, Pereira, Riley 00]

Natural Language Processing 2000s: Emerging interest in tree-based probabilistic models 2000s: Emerging interest in tree-based probabilistic models  Machine translation [Wu 97, Yamada & Knight 02, Melamed 03, Chiang 05, …]  Summarization [Knight & Marcu 00, …]  Paraphrasing [Pang et al 03, …]  Question answering [Echihabi & Marcu 03, …]  Natural language generation [Bangalore & Rambow 00, …]

FSA-s and FST-s

Finite-State Transducer (FST) k n i g h t Original input:Transformation: q i : AY q q2q2 q final q3q3 q4q4 k : ε n : N h : ε g : ε t : T

Finite-State (String) Transducer q 2 Original input:Transformation: k n i g h t q q2q2 q final q3q3 q4q4 k : ε n : N h : ε g : ε t : T i : AY

Finite-State (String) Transducer N Original input:Transformation: k n i g h t q q2q2 q final q3q3 q4q4 k : ε n : N h : ε g : ε t : T i : AY

Finite-State (String) Transducer q AY N Original input:Transformation: t q q2q2 q final q3q3 q4q4 k : ε n : N h : ε g : ε t : T i : AY k n i g h t

Finite-State (String) Transducer q3 AY N Original input:Transformation: k n i g h t q q2q2 q final q3q3 q4q4 k : ε n : N h : ε g : ε t : T i : AY

Finite-State (String) Transducer q4q4 AY N Original input:Transformation: k n i g h t q q2q2 q final q3q3 q4q4 k : ε n : N h : ε g : ε t : T i : AY

Finite-State (String) Transducer T q final AY N k n i g h t Original input:Transformation: q q2q2 q final q3q3 q4q4 k : ε n : N h : ε g : ε t : T i : AY

Definitions A weighted finite-state automaton (WFSA) A weighted finite-state automaton (WFSA)  An FSA with probabilities on the arcs  The sum of the probabilities leaving any arc must sum to one A Markov chain (or observable Markov Model) A Markov chain (or observable Markov Model)  a special case of a WFST in which the input sequence uniquely determines which states the automaton will go through Markov chains can’t represent inherently ambiguous problems Markov chains can’t represent inherently ambiguous problems  Useful for assigning probabilities to unambiguous sequences

Weighted Finite State Transducer FST: FSA whose state transitions are labeled with both input and output symbols. FST: FSA whose state transitions are labeled with both input and output symbols. A weighted transducer puts weights on transitions in addition to the input and output symbols A weighted transducer puts weights on transitions in addition to the input and output symbols Weights may encode probabilities, durations, penalties,... Weights may encode probabilities, durations, penalties,... Used in speech recognition Used in speech recognition Tutorial at Tutorial at

Markov chain for weather

Markov chain for words

Markov chain = “First-order observable Markov Model” A set of states Q A set of states Q  q 1, q 2 …q N sequence of states: state at time t is q t Transition probabilities: Transition probabilities:  a set of probabilities A = a 01 a 02 …a n1 …a nn.  Each a ij represents the probability of transitioning from state i to state j Distinguished start and end states Distinguished start and end states

Markov chain = “First-order observable Markov Model” Markov Assumption Markov Assumption Current state only depends on previous state Current state only depends on previous state P(q i | q 1 … q i-1 ) = P(q i | q i-1 )

Another representation for start state Instead of start state Instead of start state Special initial probability vector  Special initial probability vector   An initial distribution over probability of start states Constraints: Constraints:

The weather model using 

The weather model: specific example

Markov chain for weather What is the probability of 4 consecutive warm days? What is the probability of 4 consecutive warm days? Sequence is warm-warm-warm-warm Sequence is warm-warm-warm-warm i.e., state sequence is i.e., state sequence is P(3, 3, 3, 3) =  3 a 33 a 33 a 33 a 33 = 0.2 (0.6) 3 =

How about? Hot hot hot hot Hot hot hot hot Cold hot cold hot Cold hot cold hot What does the difference in these probabilities tell you about the real world weather info encoded in the figure? What does the difference in these probabilities tell you about the real world weather info encoded in the figure?

HMM for Ice Cream You are a climatologist in the year 2799 You are a climatologist in the year 2799 Studying global warming Studying global warming You can’t find any records of the weather in Baltimore, MD for summer of 2008 You can’t find any records of the weather in Baltimore, MD for summer of 2008 But you find Jason Eisner’s diary But you find Jason Eisner’s diary Which lists how many ice-creams Jason ate every date that summer Which lists how many ice-creams Jason ate every date that summer Our job: figure out how hot it was Our job: figure out how hot it was

Hidden Markov Model For Markov chains, the output symbols are the same as the states. For Markov chains, the output symbols are the same as the states.  See hot weather: we’re in state hot But in named-entity or part-of-speech tagging (and speech recognition and other things) But in named-entity or part-of-speech tagging (and speech recognition and other things)  The output symbols are words  But the hidden states are something else Part-of-speech tags Named entity tags So we need an extension! So we need an extension! A Hidden Markov Model is an extension of a Markov chain in which the input symbols are not the same as the states. A Hidden Markov Model is an extension of a Markov chain in which the input symbols are not the same as the states. This means we don’t know which state we are in. This means we don’t know which state we are in.

Hidden Markov Models

Assumptions Markov assumption: Markov assumption: P(q i | q 1 … q i-1 ) = P(q i | q i-1 ) Output-independence assumption Output-independence assumption

Eisner task Given Given  Ice Cream Observation Sequence: 1,2,3,2,2,2,3… Produce: Produce:  Weather Sequence: H,C,H,H,H,C…

HMM for ice cream

Different types of HMM structure Bakis = left-to-right Ergodic = fully-connected

The Three Basic Problems for HMMs Problem 1 (Evaluation): Given the observation sequence O=(o 1 o 2 …o T ), and an HMM model  = (A,B), how do we efficiently compute P(O|  ), the probability of the observation sequence, given the model Problem 1 (Evaluation): Given the observation sequence O=(o 1 o 2 …o T ), and an HMM model  = (A,B), how do we efficiently compute P(O|  ), the probability of the observation sequence, given the model Problem 2 (Decoding): Given the observation sequence O=(o 1 o 2 …o T ), and an HMM model  = (A,B), how do we choose a corresponding state sequence Q=(q 1 q 2 …q T ) that is optimal in some sense (i.e., best explains the observations) Problem 2 (Decoding): Given the observation sequence O=(o 1 o 2 …o T ), and an HMM model  = (A,B), how do we choose a corresponding state sequence Q=(q 1 q 2 …q T ) that is optimal in some sense (i.e., best explains the observations) Problem 3 (Learning): How do we adjust the model parameters  = (A,B) to maximize P(O|  ) ? Problem 3 (Learning): How do we adjust the model parameters  = (A,B) to maximize P(O|  ) ? Jack Ferguson at IDA in the 1960s

Problem 1: computing the observation likelihood Given the following HMM: Given the following HMM: How likely is the sequence 3 1 3? How likely is the sequence 3 1 3?

How to compute likelihood For a Markov chain, we just follow the states and multiply the probabilities For a Markov chain, we just follow the states and multiply the probabilities But for an HMM, we don’t know what the states are! But for an HMM, we don’t know what the states are! So let’s start with a simpler situation. So let’s start with a simpler situation. Computing the observation likelihood for a given hidden state sequence Computing the observation likelihood for a given hidden state sequence  Suppose we knew the weather and wanted to predict how much ice cream Jason would eat.  i.e. P( | H H C)

Computing likelihood of given hidden state sequence

Computing joint probability of observation and state sequence

Computing total likelihood of We would need to sum over We would need to sum over  Hot hot cold  Hot hot hot  Hot cold hot  …. How many possible hidden state sequences are there for this sequence? How many possible hidden state sequences are there for this sequence? How about in general for an HMM with N hidden states and a sequence of T observations? How about in general for an HMM with N hidden states and a sequence of T observations? NTNT So we can’t just do separate computation for each hidden state sequence. So we can’t just do separate computation for each hidden state sequence.

Instead: the Forward algorithm A kind of dynamic programming algorithm A kind of dynamic programming algorithm  Just like Minimum Edit Distance  Uses a table to store intermediate values Idea: Idea:  Compute the likelihood of the observation sequence  By summing over all possible hidden state sequences  But doing this efficiently By folding all the sequences into a single trellis

The forward algorithm The goal of the forward algorithm is to compute The goal of the forward algorithm is to compute P(o 1, o 2 … o T, q T = q F | ) We’ll do this by recursion We’ll do this by recursion

The forward algorithm Each cell of the forward algorithm trellis  t (j) Each cell of the forward algorithm trellis  t (j)  Represents the probability of being in state j  After seeing the first t observations  Given the automaton Each cell thus expresses the following probability Each cell thus expresses the following probability   t (j) = P(o 1, o 2 … o t, q t = j | )

The Forward Recursion

The Forward Trellis

We update each cell

The Forward Algorithm

Decoding Given an observation sequence Given an observation sequence  And an HMM And an HMM The task of the decoder The task of the decoder  To find the best hidden state sequence Given the observation sequence O=(o 1 o 2 …o T ), and an HMM model  = (A,B), how do we choose a corresponding state sequence Q=(q 1 q 2 …q T ) that is optimal in some sense (i.e., best explains the observations) Given the observation sequence O=(o 1 o 2 …o T ), and an HMM model  = (A,B), how do we choose a corresponding state sequence Q=(q 1 q 2 …q T ) that is optimal in some sense (i.e., best explains the observations)

Decoding One possibility: One possibility:  For each hidden state sequence Q HHH, HHC, HCH,HHH, HHC, HCH,  Compute P(O|Q)  Pick the highest one Why not? Why not? NTNTNTNT Instead: Instead:  The Viterbi algorithm  Is again a dynamic programming algorithm  Uses a similar trellis to the Forward algorithm

Viterbi intuition We want to compute the joint probability of the observation sequence together with the best state sequence We want to compute the joint probability of the observation sequence together with the best state sequence

Viterbi Recursion

The Viterbi trellis

Viterbi intuition Process observation sequence left to right Process observation sequence left to right Filling out the trellis Filling out the trellis Each cell: Each cell:

Viterbi Algorithm

Viterbi backtrace

Training a HMM Forward-backward or Baum-Welch algorithm (Expectation Maximization) Forward-backward or Baum-Welch algorithm (Expectation Maximization) Backward probability Backward probability  t(i) = P(o t+1, o t+2 …o T | q t = i, )  T (i) = a i,F 1  i  N

function FORWARD-BACKWARD(observations of len T, output vocabulary V, hidden state set Q) returns HMM=(A,B) initialize A and B iterate until convergence E-stepM-step return A, B

Hidden Markov Models for Part of Speech Tagging

Part of speech tagging 8 (ish) traditional English parts of speech 8 (ish) traditional English parts of speech  Noun, verb, adjective, preposition, adverb, article, interjection, pronoun, conjunction, etc.  This idea has been around for over 2000 years (Dionysius Thrax of Alexandria, c. 100 B.C.)  Called: parts-of-speech, lexical category, word classes, morphological classes, lexical tags, POS  We’ll use POS most frequently  Assuming that you know what these are

POS examples Nnoun chair, bandwidth, pacing Nnoun chair, bandwidth, pacing Vverb study, debate, munch Vverb study, debate, munch ADJadjpurple, tall, ridiculous ADJadjpurple, tall, ridiculous ADVadverbunfortunately, slowly, ADVadverbunfortunately, slowly, Pprepositionof, by, to Pprepositionof, by, to PROpronounI, me, mine PROpronounI, me, mine DETdeterminerthe, a, that, those DETdeterminerthe, a, that, those

POS Tagging example WORDtag theDET koalaN put V the DET keysN onP theDET tableN

POS Tagging Words often have more than one POS: back Words often have more than one POS: back  The back door = JJ  On my back = NN  Win the voters back = RB  Promised to back the bill = VB The POS tagging problem is to determine the POS tag for a particular instance of a word. The POS tagging problem is to determine the POS tag for a particular instance of a word. These examples from Dekang Lin

POS tagging as a sequence classification task We are given a sentence (an “observation” or “sequence of observations”) We are given a sentence (an “observation” or “sequence of observations”)  Secretariat is expected to race tomorrow  She promised to back the bill What is the best sequence of tags which corresponds to this sequence of observations? What is the best sequence of tags which corresponds to this sequence of observations? Probabilistic view: Probabilistic view:  Consider all possible sequences of tags  Out of this universe of sequences, choose the tag sequence which is most probable given the observation sequence of n words w 1 …w n.

Getting to HMM We want, out of all sequences of n tags t 1 …t n the single tag sequence such that P(t 1 …t n |w 1 …w n ) is highest. We want, out of all sequences of n tags t 1 …t n the single tag sequence such that P(t 1 …t n |w 1 …w n ) is highest. Hat ^ means “our estimate of the best one” Hat ^ means “our estimate of the best one” Argmax x f(x) means “the x such that f(x) is maximized” Argmax x f(x) means “the x such that f(x) is maximized”

Getting to HMM This equation is guaranteed to give us the best tag sequence This equation is guaranteed to give us the best tag sequence But how to make it operational? How to compute this value? But how to make it operational? How to compute this value? Intuition of Bayesian classification: Intuition of Bayesian classification:  Use Bayes rule to transform into a set of other probabilities that are easier to compute

Using Bayes Rule

Likelihood and prior n

Two kinds of probabilities (1) Tag transition probabilities P(t i |t i-1 ) Tag transition probabilities P(t i |t i-1 )  Determiners likely to precede adjs and nouns That/DT flight/NN The/DT yellow/JJ hat/NN So we expect P(NN|DT) and P(JJ|DT) to be high But P(DT|JJ) to be low  Compute P(NN|DT) by counting in a labeled corpus:

Two kinds of probabilities (2) Word likelihood probabilities P(w i |t i ) Word likelihood probabilities P(w i |t i )  VBZ (3sg Pres verb) likely to be “is”  Compute P(is|VBZ) by counting in a labeled corpus:

POS tagging: likelihood and prior n

An Example: the verb “race” Secretariat/NNP is/VBZ expected/VBN to/TO race/VB tomorrow/NR Secretariat/NNP is/VBZ expected/VBN to/TO race/VB tomorrow/NR People/NNS continue/VB to/TO inquire/VB the/DT reason/NN for/IN the/DT race/NN for/IN outer/JJ space/NN People/NNS continue/VB to/TO inquire/VB the/DT reason/NN for/IN the/DT race/NN for/IN outer/JJ space/NN How do we pick the right tag? How do we pick the right tag?

Disambiguating “race”

P(NN|TO) = P(NN|TO) = P(VB|TO) =.83 P(VB|TO) =.83 P(race|NN) = P(race|NN) = P(race|VB) = P(race|VB) = P(NR|VB) =.0027 P(NR|VB) =.0027 P(NR|NN) =.0012 P(NR|NN) =.0012 P(VB|TO)P(race|VB)P(NR|VB) = P(VB|TO)P(race|VB)P(NR|VB) = P(NN|TO)P(race|NN)P(NR|NN) = P(NN|TO)P(race|NN)P(NR|NN) = So we (correctly) choose the verb reading So we (correctly) choose the verb reading

Transitions between the hidden states of HMM, showing A probs

B observation likelihoods for POS HMM

The A matrix for the POS HMM

The B matrix for the POS HMM

Viterbi intuition: we are looking for the best ‘path’ S1S1 S2S2 S4S4 S3S3 S5S5 Slide from Dekang Lin

Viterbi example

Outline Markov Chains Markov Chains Hidden Markov Models Hidden Markov Models Three Algorithms for HMMs Three Algorithms for HMMs  The Forward Algorithm  The Viterbi Algorithm  The Baum-Welch (EM Algorithm) Applications: Applications:  The Ice Cream Task  Part of Speech Tagging  Next time: Named Entity Tagging