1. Statistical NLP: Hidden Markov Models Updated 8/12/2005

2. Markov Models Markov models are statistical tools that are useful for NLP because they can be used for part-of-speech-tagging applications Their first use was in modeling the letter sequences in works of Russian literature They were later developed as a general statistical tool More specifically, they model a sequence (perhaps through time) of random variables that are not necessarily independent They rely on two assumptions: Limited Horizon and Time Invariant

3. Markov Assumptions Let X=(X 1,.., X t ) be a sequence of random variables taking values in some finite set S={s 1, …, s n }, the state space, the Markov properties are: Limited Horizon: P(X t+1 =s k |X 1,.., X t )=P(X t+1 = s k |X t ) i.e., a word ’ s tag only depends on the previous tag. Time Invariant: P(X t+1 =s k |X t )=P(X 2 =s k |X 1 ) i.e., the dependency does not change over time. If X possesses these properties, then X is said to be a Markov Chain

4. Markov Model parameters A Markov chain is described by: Stochastic transition matrix a ij = p(X t+1 = s j | X t = s i ) Probabilities of initial states of chain  i = p(X 1 = s i ) there are obvious normalization constraints on a and on . Finally a Markov Model is formally specified by a three-tuple (S, , A) where S is the set of states, and , A are the probabilities for the initial state and state transitions.

5. Probability of sequence of states By Markov assumption P(X 1, X 2, … X T ) = p(X 1 ) p(X 2 |X 1 ) … p(X T |X T-1 ) =  X1  t=1 T a Xt, Xt+1

6. Example of a Markov Chain 1.4 1.3.4.6 1.4 te h a p i Start Probability of sequence of states: P(t, i, p) = p(X1 = t) p(X 2 =i| X 1 =t) p(X 3 =p| X 2 =i) = 1.0 * 0.3 * 0.6 = 0.18

7. Are n-gram models Markov Models? Bi-gram models are obviously Markov Models. What about tri-gram, four-gram models etc.?

8. Hidden Markov Models (HMM) Here states are hidden. We have only ‘ clues ’ about states by the symbols each state outputs: P(O t = k | X t = s i, X t+1 = s j ) = b i,j,k

9. The crazy soft drink machine lemice_tcola 0.30.10.6CP 0.20.70.1IP Cola Preffered Ice tea Preffered 0.7 0.3 0.5 start Output probability Comment: for this machine the output really depends only on s i namely b ijk = b ij

10. The crazy soft drink machine – cont. What is the probability of observing {lem, ice_t}? Need to sum over all 4 possible paths that might be taken through the HMM: 0.7*0.3*0.7*0.1 + 0.7*0.3*0.3*0.1 + 0.3*0.3*0.5*0.7 + 0.3*0.3*0.5*0.7 = 0.084

11. Why Use Hidden Markov Models? HMMs are useful when one can think of underlying events probabilistically generating surface events. Example: Part-of-Speech-Tagging or speech. HMMs can efficiently be trained using the EM Algorithm. Another example where HMMs are useful is in generating parameters for linear interpolation of n-gram models.

12. wawbwawb wbw1wbw1 wbw2wbw2 wbwMwbwM 1 ab 2 ab 3 ab w 1 : P 1 (w 1 )  : 1  : 2  : 3 w 2 :P 1 (w 2 ) w M :P 1 (w M ) w M :P 3 (w M | w a w b ) interpoloating parameters for n-gram models

13. interpoloating parameters for n- gram models – comments the HMM calculation of observing the sequence (w n-2, w n-1, w n ) is equivalent to P lin (w n |w n-2,w n-1 )= 1 P 1 (w n )+ 2 P 2 (w n |w n-1 )+ 3 P 3 (w n |w n-1,w n-2 ) Transitions are special transitions that produce no output symbol In the above model each word pair (a,b) has a different HMM. This is relaxed by using tied states.

14. General Form of an HMM An HMM is specified by a five-tuple (S, K, , A, B) where S and K are the set of states and the output alphabet, and , A, B are the probabilities for the initial state, state transitions, and symbol emissions, respectively. Given a specification of an HMM, we can simulate the running of a Markov process and produce an output sequence using the algorithm shown on the next page. More interesting than a simulation, however, is assuming that some set of data was generated by a HMM, and then being able to calculate probabilities and probable underlying state sequences.

15. A Program for a Markov Process modeled by HMM t:= 1; Start in state s i with probability  i (i.e., X 1 =i) Forever do Move from state s i to state s j with probability a ij (i.e., X t+1 = j) Emit observation symbol o t = k with probability b ijk t:= t+1 End

16. The Three Fundamental Questions for HMMs Given a model  =(A, B,  ), how do we efficiently compute how likely a certain observation is, that is, P(O|  ) Given the observation sequence O and a model , how do we choose a state sequence (X 1, …, X T+1 ) that best explains the observations? Given an observation sequence O, and a space of possible models found by varying the model parameters  = (A, B,  ), how do we find the model that best explains the observed data?

17. Finding the probability of an observation I Given the observation sequence O=(o 1, …, o T ) and a model  = (A, B,  ), we wish to know how to efficiently compute P(O|  ). This process is called decoding. For any state sequence X=(X 1, …, X T+1 ), we find: P(O|  )=  X1 … XT+1  X1  t=1 T a XtXt+1 b XtXt+1ot This is simply the sum of the probability of the observation occurring according to each possible state sequence. Direct evaluation of this expression, however, is extremely inefficient.

18. Finding the probability of an observation II In order to avoid this complexity, we can use dynamic programming or memorization techniques. In particular, we use trellis algorithms. We make a square array of states versus time and compute the probabilities of being at each state at each time in terms of the probabilities for being in each state at the preceding time. A trellis can record the probability of all initial subpaths of the HMM that end in a certain state at a certain time. The probability of longer subpaths can then be worked out in terms of the shorter subpaths.

19. Finding the probability of an observation III: The forward procedure A forward variable,  i (t)= P(o 1 o 2 … o t-1, X t =i|  ) is stored at (s i, t)in the trellis and expresses the total probability of ending up in state si at time t. Forward variables are calculated as follows: Initialization:  i (1)=  i, 1  i  N Induction:  j (t+1)=  i=1 N  i (t)a ij b ijot, 1  t  T, 1  j  N Total: P(O|  )=  i=1 N  i (T+1) This algorithm requires 2N 2 T multiplications (much less than the direct method which takes (2T+1).N T+1

20. Finding the probability of an observation IV: The backward procedure The backward procedure computes backward variables which are the total probability of seeing the rest of the observation sequence given that we were in state s i at time t. Backward variables are useful for the problem of parameter estimation.

21. Finding the probability of an observation V: The backward procedure Let  i (t) = P(o t … o T | X t = i,  ) be the backward variables. Backward variables can be calculated working backward through the trellis as follows: Initialization :  i (T+1) = 1, 1  i  N Induction:  i (t) =  j=1 N a ij b ijot  j (t+1), 1  t  T, 1  i  N Total: P(O|  )=  i=1 N  i  i (1)

22. Finding the probability of an observation VI: Combining Forward and backward P(O, X t = i |  ) = P(o 1 … o T,X t = i |  ) = P(o 1 … o t-1,X t = i, o t, … o T |  ) = P(o 1 … o t-1,X t = i |  ) P( o t, … o T |o 1 … o t-1, X t = i,  ) = P(o 1 … o t-1,X t = i |  ) P( o t, … o T |X t = i,  ) =  i (t)  i (t), Total: P(O|  ) =  i=1 N  i (t)  i (t), 1  t  T+1

23. Finding the Best State Sequence I One method consists of finding the states individually: For each t, 1  t  T+1, we would like to find X t that maximizes P(X t |O,  ). Let  i (t) = P(X t = i |O,  ) = P(X t = i, O|  )/P(O|  ) = (  i (t)  i (t)/  j=1 N  j (t)  j (t)) The individually most likely state is X t =argmax 1  i  N  i (t), 1  t  T+1 This quantity maximizes the expected number of states that will be guessed correctly. However, it may yield a quite unlikely state sequence. ^

24. Finding the Best State Sequence II: The Viterbi Algorithm The Viterbi algorithm efficiently computes the most likely state sequence. Commonly, we want to find the most likely complete path, that is: argmax X P(X|O,  ) To do this, it is sufficient to maximize for a fixed O: argmax X P(X,O|  ) We define  j (t) = max X1..Xt-1 P(X 1 … X t-1, o 1..o t-1, X t =j|  )  j (t) records the node of the incoming arc that led to this most probable path.

25. Finding the Best State Sequence II: The Viterbi Algorithm The Viterbi Algorithm works as follows: Initialization:  j (1) =  j, 1  j  N Induction:  j (t+1) = max 1  i  N  i (t)a ij b ijot, 1  j  N Store backtrace:  j (t+1) = argmax 1  i  N  j (t)a ij b ijot, 1  j  N Termination and path readout: X T+1 = argmax 1  i  N  j (T+1) Xt =  Xt+1 (t+1) P(X) = max 1  i  N  j (T+1) ^ ^ ^

26. Parameter Estimation III Given a certain observation sequence, we want to find the values of the model parameters  =(A, B,  ) which best explain what we observed. Using Maximum Likelihood Estimation, we can want find the values that maximize P(O|  ), i.e. argmax  P(O training |  ) There is no known analytic method to choose  to maximize P(O|  ). However, we can locally maximize it by an iterative hill-climbing algorithm known as Baum-Welch or Forward-Backward algorithm. (special case of the EM Algorithm)

27. Parameter Estimation III (1) We don ’ t know what the model is, but we can work out the probability of the observation sequence using some (perhaps randomly chosen) model. Looking at that calculation, we can see which state transitions and symbol emissions were probably used the most. By increasing the probability of those, we can choose a revised model which gives a higher probability to the observation sequence.

28. Parameter Estimation III (2) Define P t (i,j) as the probability of traversing a certain arc at time t given observation sequence:

29. Parameter Estimation III (3) The probability of traversing a certain arc at time t given observation sequence can be expressed as: The probability of leaving state i at time t i

30. Parameter Estimation III (4)