1. HMM II: Parameter Estimation

2. Reminder: Hidden Markov Model Markov Chain transition probabilities: p(S i+1 = t|S i = s) = a st Emission probabilities: p(X i = b| S i = s) = e s (b) S1S1 S2S2 S L-1 SLSL x1x1 x2x2 X L-1 xLxL M M M M TTTT For each integer L>0, this defines a probability space in which the simple events are HMM of length L.

3. Hidden Markov Model for CpG Islands The states: Domain(S i )={+, -}  {A,C,T,G} (8 values) In this representation P(x i | s i ) = 0 or 1 depending on whether x i is consistent with s i. E.g. x i = G is consistent with s i =(+,G) and with s i =(-,G) but not with any other state of s i. A-A- T+T+ A T G + G … … … …

4. Most Probable state path S1S1 S2S2 S L-1 SLSL x1x1 x2x2 X L-1 xLxL M M M M TTTT Given an observation sequence x = (x 1,…,x L ), A most probable path s*= (s * 1,…,s * L ) is one which maximizes p(s|x).

5. Viterbi’s algorithm for most probable path s1s1 s2s2 s L-1 sLsL X1X1 X2X2 X L-1 XLXL sisi XiXi For i=1 to L do for each state l : v l (i) = e l (x i ) MAX k {v k (i-1)a kl } ptr i (l)=argmax k {v k (i-1)a kl } [storing previous state for reconstructing the path] Termination: the probability of the most probable path Initialization: v 0 (0) = 1, v k (0) = 0 for k > 0 0 We add the special initial state 0. p(s 1 *,…,s L * ;x 1,…,x l ) =

6. A-A- C-C- T-T- T+T+ A CTT G + G Predicting CpG islands via most probable path: Output symbols: A, C, G, T (4 letters). Markov Chain states: 4 “-” states and 4 “+” states, two for each letter (8 states total). The most probable path (found by Viterbi’s algorithm) predicts CpG islands. Experiment (Durbin et al, p. 60-61) shows that the predicted islands are shorter than the assumed ones. In addition, quite a few “false negatives” are found.

7. Most probable state s1s1 s2s2 s L-1 sLsL X1X1 X2X2 X L-1 XLXL sisi XiXi Given an output sequence x = (x 1,…,x L ), s i is a most probable state (at location i) if: s i =argmax k p(S i =k |x).

8. Finding most probable state 1. The forward algorithm finds {f k (s i ) = P(x 1,…,x i,s i =k): k = 1,...m}. 2. The backward algorithm finds {b k (s i ) = P(x i+1,…,x L |s i =k ): k = 1,...m}. 3. Return {p(S i =k|x) = f k (s i ) b k (s i ) |k=1,...,m}. To Compute for every i simply run the forward and backward algorithms once, and compute {f k (s i ) b k (s i )} for every i, k. s1s1 s2s2 s L-1 sLsL X1X1 X2X2 X L-1 XLXL sisi XiXi f l (i) = p(x 1,…,x i,s i =l ), the probability of a path which emits (x 1,..,x i ) and in which state s i =l. b l (i)= p(x i+1,…,x L,s i =l), the probability of a path which emits (x i+1,..,x L ) and in which state s i =l.

9. Finding the probability that a letter is in a CpG island via the algorithm for most probable state: The probability that the occurrence of G in the i-th location is in a CpG island (+ state) is: ∑ s + p(S i =s + |x) = ∑ s + F(S i =s + )B(S i =s + ) Where the summation is formally over the 4 “+” states, but actually only state G + need to be considered (why?) A-A- C-C- T-T- T+T+ A CTT G + G i

10. Parameter Estimation for HMM An HMM model is defined by the parameters: a kl and e k (b), for all states k,l and all symbols b. Let θ denote the collection of these parameters. s1s1 s2s2 s L-1 sLsL X1X1 X2X2 X L-1 XLXL sisi XiXi lk b a kl ek(b)ek(b)

11. Parameter Estimation for HMM To determine the values of (the parameters in) θ, use a training set = {x 1,...,x n }, where each x j is a sequence which is assumed to fit the model. Given the parameters θ, each sequence x j has an assigned probability p(x j |θ). s1s1 s2s2 s L-1 sLsL X1X1 X2X2 X L-1 XLXL sisi XiXi

12. Maximum Likelihood Parameter Estimation for HMM The elements of the training set {x 1,...,x n }, are assumed to be independent, p(x 1,..., x n |θ) = ∏ j p (x j |θ). ML parameter estimation looks for θ which maximizes the above. The exact method for finding or approximating this θ depends on the nature of the training set used.

13. Data for HMM Possible properties of (the sequences in) the training set: 1.For each x j, the information on the states s j i 2.The size (number of sequences) of the training set S1S1 S2S2 S L-1 SLSL x1x1 x2x2 X L-1 xLxL M M M M TTTT

14. Case 1: ML when state paths are fully known We know the complete structure of each sequence in the training set {x 1,...,x n }. We wish to estimate a kl and e k (b) for all pairs of states k, l and symbols b. By the ML method, we look for parameters θ* which maximize the probability of the sample set: p(x 1,...,x n | θ*) =MAX θ p(x 1,...,x n | θ). s1s1 s2s2 s L-1 sLsL X1X1 X2X2 X L-1 XLXL sisi XiXi

15. Case 1: State paths are fully known s1s1 s2s2 s L-1 sLsL X1X1 X2X2 X L-1 XLXL sisi XiXi Let m kl = |{i: s i-1 =k,s i =l}| (in x j ). m k (b)=|{i:s i =k,x i =b}| (in x j ). For each x j we have:

16. Case 1 s1s1 s2s2 s L-1 sLsL X1X1 X2X2 X L-1 XLXL sisi XiXi By the independence of the x j ’s, p(x 1,...,x n | θ)=∏ j p(x j |θ). Thus, if A kl = #(transitions from k to l) in the training set, and E k (b) = #(emissions of symbol b from state k) in the training set, we have:

17. Case 1 (cont) So we need to find a kl ’s and e k (b)’s which maximize: Subject to:

18. Case 1 Rewriting, we need to maximize:

19. Case 1 (cont) Then we will maximize also F. Each of the above is a simpler ML problem, which is similar to ML parameters estimation for a die, treated next.

20. MLE for n outcomes The MLE is given by the relative frequencies:

21. Apply the ML method to HMM Let A kl = #(transitions from k to l) in the training set. E k (b) = #(emissions of symbol b from state k) in the training set. We need to: s1s1 s2s2 s L-1 sLsL X1X1 X2X2 X L-1 XLXL sisi XiXi

22. Apply to HMM (cont.) We apply the previous technique to get for each k the parameters {a kl |l=1,..,m} and {e k (b)|b  Σ}: s1s1 s2s2 s L-1 sLsL X1X1 X2X2 X L-1 XLXL sisi XiXi Which gives the optimal ML parameters

23. Summary of Case 1: State paths are fully known We know the complete structure of each sequence in the training set {x 1,...,x n }. We wish to estimate a kl and e k (b) for all pairs of states k, l and symbols b. Summary: when everything is known, we can find the (unique set of) parameters θ* which maximizes p(x 1,...,x n | θ*) =MAX θ p(x 1,...,x n | θ). s1s1 s2s2 s L-1 sLsL X1X1 X2X2 X L-1 XLXL sisi XiXi

24. Adding pseudo counts in HMM If the sample set is too small, we may get a biased result. In this case we modify the actual count by our prior knowledge/belief: r kl is our prior belief and transitions from k to l. r k (b) is our prior belief on emissions of b from state k. s1s1 s2s2 s L-1 sLsL X1X1 X2X2 X L-1 XLXL sisi XiXi

25. Case 2: State paths are unknown For a given θ we have: p(x 1,..., x n |θ)= p(x 1 | θ)    p (x n |θ) (since the x j are independent) s1s1 s2s2 s L-1 sLsL X1X1 X2X2 X L-1 XLXL sisi XiXi For each sequence x, p(x|θ)=∑ s p(x,s|θ), The sum taken over all state paths s which emit x.

26. Case 2: State paths are unknown Thus, for the n sequences (x 1,..., x n ) we have: p(x 1,..., x n |θ)= ∑ p(x 1,..., x n, s 1,..., s n |θ), Where the summation is taken over all tuples of n state paths (s 1,..., s n ) which generate (x 1,..., x n ). For simplicity, we will assume that n=1. s1s1 s2s2 s L-1 sLsL X1X1 X2X2 X L-1 XLXL sisi XiXi (s 1,..., s n )

27. Case 2: State paths are unknown So we need to maximize p(x|θ)=∑ s p(x,s|θ), where the summation is over all the sequences S which produce the output sequence x. Finding θ * which maximizes ∑ s p(x,s|θ) is hard. [Unlike finding θ * which maximizes p(x,s|θ) for a single sequence (x,s).] s1s1 s2s2 s L-1 sLsL X1X1 X2X2 X L-1 XLXL sisi XiXi

28. ML Parameter Estimation for HMM The general process for finding θ in this case is 1.Start with an initial value of θ. 2.Find θ’ so that p(x|θ’) > p(x|θ) 3.set θ = θ’. 4.Repeat until some convergence criterion is met. A general algorithm of this type is the Expectation Maximization algorithm, which we will meet later. For the specific case of HMM, it is the Baum- Welch training.

29. Baum Welch training We start with some values of a kl and e k (b), which define prior values of θ. Then we use an iterative algorithm which attempts to replace θ by a θ * s.t. p( x |θ * ) > p( x |θ) This is done by “imitating” the algorithm for Case 1, where all states are known: s1s1 s2s2 s L-1 sLsL X1X1 X2X2 X L-1 XLXL sisi XiXi

30. Baum Welch training In case 1 we computed the optimal values of a kl and e k (b), (for the optimal θ) by simply counting the number A kl of transitions from state k to state l, and the number E k (b) of emissions of symbol b from state k, in the training set. This was possible since we knew all the states. S i = lS i-1 = k x i-1 = b … … x i = c

31. Baum Welch training When the states are unknown, the counting process is replaced by averaging process: For each edge s i-1  s i we compute the average number of “k to l” transitions, for all possible pairs (k,l), over this edge. Then, for each k and l, we take A kl to be the sum over all edges. S i = ?S i-1 = ? x i-1 = b x i = c ……

32. Baum Welch training Similarly, For each edge s i  b and each state k, we compute the average number of times that s i =k, which is the expected number of “k → b” transmission on this edge. Then we take E k (b) to be the sum over all such edges. These expected values are computed by assuming the current parameters θ: S i = ? x i-1 = b

33. A kl and E k (b) when states are unknown A kl and E k (b) are computed according to the current distribution θ, that is: s1s1 s2s2 s L-1 sLsL X1X1 X2X2 X L-1 XLXL sisi XiXi A kl =∑ s A s kl p(s|x,θ), where A s kl is the number of k to l transitions in the sequence s. E k (b)=∑ s E s k (b)p(s|x,θ), where E s k (b) is the number of times k emits b in the sequence s with output x.

34. Baum Welch: step 1a Count expected number of state transitions For each i, k,l, compute the state transitions probabilities by the current θ: s1s1 SiSi sLsL X1X1 XiXi XLXL S i-1 X i-1.. P(s i-1 =k, s i =l | x,θ) For this, we use the forwards and backwards algorithms

35. Finding state probabilities p(s i =k,x) = F k (s i ) B k (s i ) {F k (i) B k (i)} for every i and k are computed by one run of the backward/forward algorithms. s1s1 s2s2 s L-1 sLsL X1X1 X2X2 X L-1 XLXL sisi XiXi F k (i) = p(x 1,…,x i,s i =k ), the probability that in a path which emits (x 1,..,x i ), state s i =k. B k (i)= p(x i+1,…,x L |s i =k), the probability that a path which emits (x i+1,..,x L ), given that state s i =k.

36. Baum Welch: Step 1a (cont) Claim: By the probability distribution of HMM s1s1 SiSi sLsL X1X1 XiXi XLXL S i-1 X i-1.. (a kl and e l (x i ) are the parameters defined by , and F k (i-1), B k (i) are the forward and backward algorithms)

37. Step 1a: Computing P(s i-1 =k, s i =l | x,θ) P(x 1,…,x L,s i-1 =k,s i =l|  ) = P(x 1,…,x i-1,s i-1 =k|  ) a kl e l (x i ) P(x i+1,…,x L |s i =l,  ) = F k (i-1) a kl e l (x i ) B l (i) Via the forward algorithm Via the backward algorithm s1s1 s2s2 s L-1 sLsL X1X1 X2X2 X L-1 XLXL S i-1 X i-1 sisi XiXi x p(s i-1 =k,s i =l | x,  ) = F k (i-1) a kl e l (x i ) B l (i)

38. Step 1a For each pair (k,l), compute the expected number of state transitions from k to l, as the sum of the expected number of k to l transitions over all L edges :

39. Step 1a for many sequences: When we have n independent input sequences (x 1,..., x n ), then A kl is given by:

40. Baum-Welch: Step 1b count expected number of symbols emissions for state k and each symbol b, for each i where X i =b, compute the expected number of times that X i =b. s1s1 s2s2 s L-1 sLsL X1X1 X2X2 X L-1 XLXL sisi X i =b

41. Baum-Welch: Step 1b For each state k and each symbol b, compute the expected number of emissions of b from k as the sum of the expected number of times that s i = k, over all i’s for which x i = b.

42. Step 1b for many sequences Exercise: when we have n sequences (x 1,..., x n ), the expected number of emissions of b from k is given by:

43. Summary of Steps 1a and 1b: the E part of the Baum Welch training These steps compute the expected numbers A kl of k,l transitions for all pairs of states k and l, and the expected numbers E k (b) of transmitions of symbol b from state k, for all states k and symbols b. The next step is the M step, which is identical to the computation of optimal ML parameters when all states are known.

44. Baum-Welch: step 2 Use the A kl ’s, E k (b)’s to compute the new values of a kl and e k (b). These values define θ *. The correctness of the EM algorithm implies that: p(x 1,..., x n |θ * )  p(x 1,..., x n |θ) i.e, θ * increases the probability of the data This procedure is iterated, until some convergence criterion is met.