1. Hidden Markov Models HMM Hassanin M. Al-Barhamtoshy hassanin@kau.edu.sa

2. Agenda Markov Models Description: What is HMM HMM Applications Demonstration

3. Set of states or number of states : Q = {q 1 ; q 2 ; : : : ; q T } Process moves from one state to another generating a sequence of states : Markov chain property: probability of each subsequent state depends only on what was the previous state: To define Markov model, the following probabilities have to be specified: transition probabilities and initial probabilities Markov Models

4. Rain Dry 0.70.3 0.20.8 Two states : ‘Rain’ and ‘Dry’. Transition probabilities: P( ‘Rain’ | ‘Rain’ ) = 0.3, P( ‘Dry’ | ‘Rain’ ) = 0.7, P( ‘Rain’ | ‘Dry’ ) = 0.2, P( ‘Dry’ | ‘Dry’ ) = 0.8 Initial probabilities: say P( ‘Rain’ ) = 0.4, P( ‘Dry’ ) = 0.6. Example 1 of Markov Model

5. By Markov chain property, probability of state sequence can be found by the formula: Suppose we want to calculate a probability of a sequence of states in our example, {‘Dry’, ’Dry’, ’Rain’, ‘Rain’}. P( {‘Dry’, ’Dry’, ’Rain’, ‘Rain’} ) = P( ‘Rain’ | ’Rain’ ) P( ‘Rain’ | ’Dry’ ) P( ‘Dry’ | ’Dry’ ) P( ‘Dry’ ) = 0.3 x 0.2 x 0.8 x 0.6 Calculation of Sequence probability Rain Dry 0.70.3 0.20.8

6. What is an HMM? Graphical Model Circles indicate states Arrows indicate probabilistic dependencies between states

7. What is an HMM? Green circles are hidden states Dependent only on the previous state

8. What is an HMM? Purple nodes are observed states Dependent only on their corresponding hidden state

9. HMM Formalism { S, K,  S : {s 1 …s N } are the values for the hidden states K : {k 1 …k M } are the values for the observations SSS KKK S K S K

10. HMM Formalism {S, K,     are the initial state probabilities A = {a ij } are the state transition probabilities B = {b ik } are the observation state probabilities A B AAA BB SSS KKK S K S K

11. Inference in an HMM Compute the probability of a given observation sequence Given an observation sequence, compute the most likely hidden state sequence Given an observation sequence and set of possible models, which model most closely fits the data?

12. oToT o1o1 otot o t-1 o t+1 Given an observation sequence and a model, compute the probability of the observation sequence Decoding

13. oToT o1o1 otot o t-1 o t+1 x1x1 x t+1 xTxT xtxt x t-1

14. Decoding oToT o1o1 otot o t-1 o t+1 x1x1 x t+1 xTxT xtxt x t-1

15. Decoding oToT o1o1 otot o t-1 o t+1 x1x1 x t+1 xTxT xtxt x t-1

16. Decoding oToT o1o1 otot o t-1 o t+1 x1x1 x t+1 xTxT xtxt x t-1

17. Decoding oToT o1o1 otot o t-1 o t+1 x1x1 x t+1 xTxT xtxt x t-1

18. Forward Procedure oToT o1o1 otot o t-1 o t+1 x1x1 x t+1 xTxT xtxt x t-1 Special structure gives us an efficient solution using dynamic programming. Intuition: Probability of the first t observations is the same for all possible t+1 length state sequences. Define:

19. oToT o1o1 otot o t-1 o t+1 x1x1 x t+1 xTxT xtxt x t-1 Forward Procedure

20. oToT o1o1 otot o t-1 o t+1 x1x1 x t+1 xTxT xtxt x t-1 Forward Procedure

21. oToT o1o1 otot o t-1 o t+1 x1x1 x t+1 xTxT xtxt x t-1 Forward Procedure

22. oToT o1o1 otot o t-1 o t+1 x1x1 x t+1 xTxT xtxt x t-1 Forward Procedure

23. oToT o1o1 otot o t-1 o t+1 x1x1 x t+1 xTxT xtxt x t-1 Forward Procedure

24. oToT o1o1 otot o t-1 o t+1 x1x1 x t+1 xTxT xtxt x t-1 Forward Procedure

25. oToT o1o1 otot o t-1 o t+1 x1x1 x t+1 xTxT xtxt x t-1 Forward Procedure

26. oToT o1o1 otot o t-1 o t+1 x1x1 x t+1 xTxT xtxt x t-1 Forward Procedure

27. oToT o1o1 otot o t-1 o t+1 x1x1 x t+1 xTxT xtxt x t-1 Backward Procedure Probability of the rest of the states given the first state

28. oToT o1o1 otot o t-1 o t+1 x1x1 x t+1 xTxT xtxt x t-1 Decoding Solution Forward Procedure Backward Procedure Combination

29. oToT o1o1 otot o t-1 o t+1 Best State Sequence Find the state sequence that best explains the observations Viterbi algorithm

30. oToT o1o1 otot o t-1 o t+1 Viterbi Algorithm The state sequence which maximizes the probability of seeing the observations to time t-1, landing in state j, and seeing the observation at time t x1x1 x t-1 j

31. oToT o1o1 otot o t-1 o t+1 Viterbi Algorithm Recursive Computation x1x1 x t-1 xtxt x t+1

32. oToT o1o1 otot o t-1 o t+1 Viterbi Algorithm Compute the most likely state sequence by working backwards x1x1 x t-1 xtxt x t+1 xTxT

33. oToT o1o1 otot o t-1 o t+1 Parameter Estimation Given an observation sequence, find the model that is most likely to produce that sequence. No analytic method Given a model and observation sequence, update the model parameters to better fit the observations. A B AAA BBBB x1x1 x t-1 xTxT

34. oToT o1o1 otot o t-1 o t+1 Parameter Estimation A B AAA BBBB Probability of traversing an arc Probability of being in state i

35. oToT o1o1 otot o t-1 o t+1 Parameter Estimation A B AAA BBBB Now we can compute the new estimates of the model parameters.

36. HMM Applications Generating parameters for n-gram models Tagging speech Speech recognition

37. oToT o1o1 otot o t-1 o t+1 The Most Important Thing A B AAA BBBB We can use the special structure of this model to do a lot of neat math and solve problems that are otherwise not solvable.

38. Remember: Hidden Markov Models. Set of states: Process moves from one state to another generating a sequence of states : Markov chain property: probability of each subsequent state depends only on what was the previous state: States are not visible, but each state randomly generates one of M observations (or visible states) To define hidden Markov model, the following probabilities have to be specified: matrix of transition probabilities A=(a ij ), a ij = P(s i | s j ), matrix of observation probabilities B=(b i (v m )), b i (v m ) = P(v m | s i ) and a vector of initial probabilities  =(  i ),  i = P(s i ). Model is represented by M=(A, B,  ).

39. Low High 0.70.30.20.8 Dry Rain 0.6 0.4 Example of Hidden Markov Model

40. Two states : ‘Low’ and ‘High’ atmospheric pressure. Two observations : ‘Rain’ and ‘Dry’. Transition probabilities: P( ‘Low’|‘Low’ ) =0.3, P( ‘High’|‘Low’ ) =0.7, P( ‘Low’|‘High’ ) =0.2, P( ‘High’|‘High’ ) =0.8 Observation probabilities: P( ‘Rain’|‘Low’ ) = 0.6, P( ‘Dry’|‘Low’ ) = 0.4, P( ‘Rain’|‘High’ ) = 0.4, P( ‘Dry’|‘High’ ) = 0.3. Initial probabilities: say P( ‘Low’ ) =0.4, P( ‘High’ ) =0.6. Example of Hidden Markov Model Low High 0.70.3 0.20.8 Dry Rain 0.6 0.4

41. Suppose we want to calculate a probability of a sequence of observations in our example, {‘Dry’, ’Rain’}. Consider all possible hidden state sequences: P( {‘Dry’,’Rain’} ) = P( {‘Dry’,’Rain’}, {‘Low’,’Low’} ) + P( {‘Dry’,’Rain’}, {‘Low’,’High’} ) + P( {‘Dry’,’Rain’}, {‘High’,’Low’} ) + P( {‘Dry’,’Rain’}, {‘High’,’High’} ) where first term is : P( {‘Dry’,’Rain’}, {‘Low’,’Low’} ) = P( {‘Dry’,’Rain’} | {‘Low’,’Low’} ) P( {‘Low’,’Low’} ) = P( ‘Dry’|’Low’ )P( ‘Rain’|’Low’ ) P( ‘Low’ )P( ‘Low’|’Low ) = 0.4*0.6*0.4*0.3 Calculation of observation sequence probability Low High 0.70.3 0.20.8 Dry Rain 0.6 0.4 oToT o1o1 otot o t-1 o t+1 x1x1 X t-1 xTxT A B B B B B A A A

42. Evaluation problem. Given the HMM M=(A, B,  ) and the observation sequence O=o 1 o 2... o K, calculate the probability that model M has generated sequence O. Decoding problem. Given the HMM M=(A, B,  ) and the observation sequence O=o 1 o 2... o K, calculate the most likely sequence of hidden states s i that produced this observation sequence O. Learning problem. Given some training observation sequences O=o 1 o 2... o K and general structure of HMM (numbers of hidden and visible states), determine HMM parameters M=(A, B,  ) that best fit training data. O=o 1...o K denotes a sequence of observations o k  { v 1,…,v M }. Main issues using HMMs :

43. Markov Models Probabilistic Finite State Automaton Figure – A Markov Model

44. What is the probability of a sequence of states ?

45. Example

46. 46 Markov Chain

47. 47 Markov Chain

48. 48 Markov Chain And then, what does “hidden” means? It results in a Geometric Distribution

49. 49 Extension to Hidden Markov Model

50. 50 The Three Basic Problems

51. Typed word recognition, assume all characters are separated. Character recognizer outputs probability of the image being particular character, P (image | character). 0.5 0.03 0.005 0.31 z c b a Word Recognition example(1). Hidden state Observation

52. Hidden states of HMM = characters. Observations = typed images of characters segmented from the image. Note that there is an infinite number of observations Observation probabilities = character recognizer scores. Transition probabilities will be defined differently in two subsequent models. Word Recognition example(2).

53. If lexicon is given, we can construct separate HMM models for each lexicon word. Amherstam he r s t Buffalobu ff a l o 0.5 0.03 Here recognition of word image is equivalent to the problem of evaluating few HMM models. This is an application of Evaluation problem. Word Recognition example(3). 0.40.6

54. We can construct a single HMM for all words. Hidden states = all characters in the alphabet. Transition probabilities and initial probabilities are calculated from language model. Observations and observation probabilities are as before. a m he r s t b v f o Here we have to determine the best sequence of hidden states, the one that most likely produced word image. This is an application of Decoding problem. Word Recognition example(4).

55. The structure of hidden states is chosen. Observations are feature vectors extracted from vertical slices. Probabilistic mapping from hidden state to feature vectors: 1. Use mixture of Gaussian models 2. Quantize feature vector space. Character Recognition with HMM example.

56. The structure of hidden states: Observation = number of islands in the vertical slice. s1s1 s2s2 s3s3 HMM for character ‘A’ : Transition probabilities: { a ij } = Observation probabilities: { b jk }= .8.2 0   0.8.2   0 0 1  .9.1 0  .1.8.1  .9.1 0  HMM for character ‘B’ : Transition probabilities: { a ij } = Observation probabilities: { b jk }= .8.2 0   0.8.2   0 0 1  .9.1 0   0.2.8  .6.4 0  Exercise: Character Recognition with HMM(1)

57. Suppose that after character image segmentation the following sequence of island numbers in 4 slices was observed: { 1, 3, 2, 1} What HMM is more likely to generate this observation sequence, HMM for ‘A’ or HMM for ‘B’ ? Exercise: Character Recognition with HMM(2)

58. Consider likelihood of generating given observation for each possible sequence of hidden states: HMM for character ‘A’: Hidden state sequenceTransition probabilitiesObservation probabilities s 1  s 1  s 2  s 3.8 .2 .2 .9  0 .8 .9 = 0 s 1  s 2  s 2  s 3.2 .8 .2 .9 .1 .8 .9 = 0.0020736 s 1  s 2  s 3  s 3.2 .2  1 .9 .1 .1 .9 = 0.000324 Total = 0.0023976 HMM for character ‘B’: Hidden state sequenceTransition probabilitiesObservation probabilities s 1  s 1  s 2  s 3.8 .2 .2 .9  0 .2 .6 = 0 s 1  s 2  s 2  s 3.2 .8 .2 .9 .8 .2 .6 = 0.0027648 s 1  s 2  s 3  s 3.2 .2  1 .9 .8 .4 .6 = 0.006912 Total = 0.0096768 Exercise: Character Recognition with HMM(3) .8.2 0   0.8.2   0 0 1  .9.1 0  .1.8.1  .9.1 0 

59. Evaluation problem. Given the HMM M=(A, B,  ) and the observation sequence O=o 1 o 2... o K, calculate the probability that model M has generated sequence O. Trying to find probability of observations O=o 1 o 2... o K by means of considering all hidden state sequences (as was done in example) is impractical: N K hidden state sequences - exponential complexity. Use Forward-Backward HMM algorithms for efficient calculations. Define the forward variable  k (i) as the joint probability of the partial observation sequence o 1 o 2... o k and that the hidden state at time k is s i :  k (i)= P(o 1 o 2... o k, q k = s i ) Evaluation Problem.

60. s1s1 s2s2 sisi sNsN s1s1 s2s2 sisi sNsN s1s1 s2s2 sjsj sNsN s1s1 s2s2 sisi sNsN a 1j a 2j a ij a Nj Time= 1 k k+1 K o 1 o k o k+1 o K = Observations Lattice Representation of an HMM

61. Initialization:  1 (i)= P(o 1, q 1 = s i ) =  i b i (o 1 ), 1<=i<=N. Forward recursion:  k+1 (i)= P(o 1 o 2... o k+1, q k+1 = s j ) =  i P(o 1 o 2... o k+1, q k = s i, q k+1 = s j ) =  i P(o 1 o 2... o k, q k = s i ) a ij b j (o k+1 ) = [  i  k (i) a ij ] b j (o k+1 ), 1<=j<=N, 1<=k<=K-1. Termination: P(o 1 o 2... o K ) =  i P(o 1 o 2... o K, q K = s i ) =  i  K (i) Complexity : N 2 K operations. Forward Recursion for HMM

62. Define the forward variable  k (i) as the joint probability of the partial observation sequence o k+1 o k+2... o K given that the hidden state at time k is s i :  k (i)= P(o k+1 o k+2... o K |q k = s i ) Initialization:  K (i)= 1, 1<=i<=N. Backward recursion:  k (j)= P(o k+1 o k+2... o K | q k = s j ) =  i P(o k+1 o k+2... o K, q k+1 = s i | q k = s j ) =  i P(o k+2 o k+3... o K | q k+1 = s i ) a ji b i (o k+1 ) =  i  k+1 (i) a ji b i (o k+1 ), 1<=j<=N, 1<=k<=K-1. Termination: P(o 1 o 2... o K ) =  i P(o 1 o 2... o K, q 1 = s i ) =  i P(o 1 o 2... o K |q 1 = s i ) P(q 1 = s i ) =  i  1 (i) b i (o 1 )  i Backward Recursion for HMM

63. Decoding problem. Given the HMM M=(A, B,  ) and the observation sequence O=o 1 o 2... o K, calculate the most likely sequence of hidden states s i that produced this observation sequence. We want to find the state sequence Q = q 1 …q K which maximizes P(Q | o 1 o 2... o K ), or equivalently P(Q, o 1 o 2... o K ). Brute force consideration of all paths takes exponential time. Use efficient Viterbi algorithm instead. Define variable  k (i) as the maximum probability of producing observation sequence o 1 o 2... o k when moving along any hidden state sequence q 1 … q k-1 and getting into q k = s i.  k (i) = max P(q 1 … q k-1, q k = s i, o 1 o 2... o k ) where max is taken over all possible paths q 1 … q k-1. Decoding Problem

64. Specification of an HMM N - number of states –Q = {q 1 ; q 2 ; : : : ; q T } - set of states M - the number of symbols (observables) –O = {o 1 ; o 2 ; : : : ;o T } - set of symbols

65. Specification of an HMM A - the state transition probability matrix –aij = P(q t+1 = j | q t = i) B- observation probability distribution –b j (k) = P(o t = k | q t = j) i ≤ k ≤ M π - the initial state distribution Full HMM is thus specified as a triplet:  λ = (A, B, π)

66. Central problems in HMM modelling Problem 1 Evaluation: –Probability of occurrence of a particular observation sequence, O = {o 1,…,o k }, given the model –P(O|λ) –Complicated – hidden states –Useful in sequence classification

67. Central problems in HMM modelling Problem 2 Decoding: –Optimal state sequence to produce given observations, O = {o 1,…,o k }, given model –Optimality criterion –Useful in recognition problems

68. Central problems in HMM modelling Problem 3 Learning: –Determine optimum model, given a training set of observations –Find λ, such that P(O|λ) is maximal

69. General idea: if best path ending in q k = s j goes through q k-1 = s i then it should coincide with best path ending in q k-1 = s i. s1s1 sisi sNsN sjsj a ij a Nj a 1j q k-1 q k  k (i) = max P(q 1 … q k-1, q k = s j, o 1 o 2... o k ) = max i [ a ij b j (o k ) max P(q 1 … q k-1 = s i, o 1 o 2... o k-1 ) ] To backtrack best path keep info that predecessor of s j was s i. Viterbi Algorithm (1)

70. Initialization:  1 (i) = max P(q 1 = s i, o 1 ) =  i b i (o 1 ), 1<=i<=N. Forward recursion:  k (j) = max P(q 1 … q k-1, q k = s j, o 1 o 2... o k ) = max i [ a ij b j (o k ) max P(q 1 … q k-1 = s i, o 1 o 2... o k-1 ) ] = max i [ a ij b j (o k )  k-1 (i) ], 1<=j<=N, 2<=k<=K. Termination: choose best path ending at time K max i [  K (i) ] Backtrack best path. This algorithm is similar to the forward recursion of evaluation problem, with  replaced by max and additional backtracking. Viterbi algorithm (2)

71. Learning problem. Given some training observation sequences O=o 1 o 2... o K and general structure of HMM (numbers of hidden and visible states), determine HMM parameters M=(A, B,  ) that best fit training data, that is maximizes P(O | M). There is no algorithm producing optimal parameter values. Use iterative expectation-maximization algorithm to find local maximum of P(O | M) - Baum-Welch algorithm. Learning Problem (1)

72. If training data has information about sequence of hidden states (as in word recognition example), then use maximum likelihood estimation of parameters: a ij = P(s i | s j ) = Number of transitions from state s j to state s i Number of transitions out of state s j b i (v m ) = P(v m | s i )= Number of times observation v m occurs in state s i Number of times in state s i Learning Problem (2)

73.  http://www.comp.leeds.ac.uk/roger/HiddenMarkovModel s/html_dev/main.html http://www.comp.leeds.ac.uk/roger/HiddenMarkovModel s/html_dev/main.html For links & slides  www.evl.uic.edu/shalini/hmm/ www.evl.uic.edu/shalini/hmm/