1. . The Relative Entropy Rate of Two Hidden Markov Processes Or Zuk Dept. of Phys. Of Comp. Systems Weizmann Inst. Of Science Rehovot, Israel

2. 2 Overview u Introduction u Distance Measures and Relative Entropy rate u Results: Generalization from Entropy Rate. u Future Directions

3. 3 Introduction Hidden Markov Processes are relevant: u Error Correction (Markovian source +noise) u Signal Processing, Speech recognition u Experimental physics -telegraph noise, TLS+noise, quantum jumps. u Bioinformatics -biological sequences, gene expression t Transmission Noise 10% Markov chainHMP Quantum jumps 02004006008001000 7.0 8.0 9.0 R (Mohm) Mesoscopic wires

4. 4 HMP - Definitions Markov Process: X – Markov Process M λ – Transition Matrix u m λ (i,j) = Pr(X n+1 = j| X n = i) Hidden Markov Process : Y – Noisy Observation of X R λ – Noise/Emission Matrix u r λ (i,j) = Pr(Y n = j| X n = i) MλMλ RλRλ RλRλ XnXn X n+1 Y n+1 YnYn Models are denoted by λ and µ.

5. 5 Example: Binary HMP 0 1 p(1|0) p(0|1) p(1|1) p(0|0) 0 1 q(0|0) q(1|0) q(0|1) q(1|1) Transition Emission

6. 6 Example: Binary HMP (Cont.) u A simple, Symmetric Binary HMP : u M = R = u All properties of the process depend on two parameters, p and . Assume w.l.og. p,  < ½

7. 7 Overview u Introduction u Distance Measures and Relative Entropy rate u Results: Generalization from Entropy Rate. u Future Directions

8. 8 Distance Measures for Two HMPs u Why important ? u Often, one learns a HMP from data. It is important to know how different is the learned model from the true model. u Sometimes, many HMPs may represent different sources (e.g. different authors, different protein families etc.), and we wish to know which sources are similar. u What distance measure to use? u Look at joint distributions of N consecutive Y symbols P λ (N) and P µ (N).

9. 9 Relative Entropy (RE) Rate u Notation : u Relative Entropy for finite (N-symbol) distributions: u Take the limit to get the RE-rate: u Alternative definition, using conditional relative entropy:

10. 10 Relative Entropy (RE) Rate u First proposed for HMPs by [Juang&Rabiner 85]. u Not a norm (not symmetric, no triangle inequality). u Still it has several natural interpretations: -If one generates data from λ, and gives likelihood score to µ, then D(λ || µ) is the average likelihood-loss per symbol (compared to the optimal model λ). -If one compresses data generated λ, assuming erroneously it was generated by µ, then one ‘looses’ on average D(λ || µ) per symbol. u For Markov chains, D(λ || µ) is easily given by:

11. 11 Relative Entropy (RE) Rate u For HMPs, D(λ || µ) is difficult to compute. So far only bounds [Silva&Narayanan] or approximation algorithms [Li et al. 05, Do 03, Mohammad&Tranter 05] are known. u D(λ || µ) generalizes the concept of the Shannon entropy rate, using: H(λ) = log s – D(λ || u) Where u is the uniform model, s is the alphabet size of Y. u The entropy rate H for an HMP is a Lyapunov Exponent, which is hard to compute generally. [Jacquet et al 04] u What is known (for H) ? Lyapunov exponent representation, analyticity, asymptotic expansions in different Regimes. u Generalize results and techniques to the RE-rate.

12. 12 Why is calculating D(λ || µ) difficult? Markov Chains: -All states with the same no. of flips have the same prob. Polynomial number of types (probs). 2N2N X X X Y 2N2N X 2N2N Y HMPs :Many Markov chains, {X} contributes to the same Y. Different {Y}s have different probs. Exponential number of types (probs). Method of types does not work here.

13. 13 Overview u Introduction u Distance Measures and Relative Entropy rate u Results: Generalization from Entropy Rate. u Future Directions

14. 14 RE-Rate and Lyapunov Exponents u What is Lyapunov exponent? u Arises in Dynamical Systems, Control Theory, Statistical Physics etc. Measures the stability of the system. u Take two (square) matrices A,B. Choose each time at random A (with prob. p) or B (w.p. 1-p). Look at the norm: (1/N) log ||ABBBAABAB…BA|| The limit: -Exists a.s. [Furstenberg&Kesten 60] -Called Top Lyaponov Exponent. -Independent of Matrix Norm chosen. u HMP entropy rate is given as a Lyaponov Exponent [Jacquet et al. 04]

15. 15 RE-Rate and Lyapunov Exponents u What about RE-rate? u Given as the difference of two Lyapunov Exponents: -The G’s are random matrices, which are simply obtained from M and R using the forward equations. -Different matrices appear in the two Lyapunov exponents, but the probabilities selecting the matrices are the same.

16. 16 Analyticity of the RE-Rate u Is the RE-rate continuous, ‘smooth’, or even analytic in the parameters governing the HMPs? u For Lyapunov exponents: Known analyticity in the matrix entries [Rulle 79], and their probabilities [Peres 90,91] separately. u For HMP entropy rate, analyticity was recently shown by [Han&Marcus 05].

17. 17 Analyticity of the RE-Rate u Using both results, we are able to show: Thm: The RE-rate is analytic in the HMPs parameters. u Analyticity is shown only in the interior of the parameters domain (i.e. strictly positive probabilities). u Behavior on the boundaries is more complicated. Sometimes analyticity remains on the boundaries (and beyond). Sometimes we encounter singularities. Full characterization is still lacking [Marcus&Han 05].

18. 18 RE-Rate Taylor Series Expansion u While in general the RE-rate is not known, there are specific parameters values for which it is easily given in closed-form (e.g. for Markov-Chains). Perhaps we can ‘expand’ around these values, and get asymptotic results near them. u Similar approach was used for Lyapunov exponents [Derrida], and for HMP entropy rate [Jacquet et al. 04, Weizmann&Ordenlich 04, Zuk et al. 05] giving first-order asymptotics in various regimes.

19. 19 Different Regimes – Binary Case p -> 0, p -> ½ (  fixed)  -> 0,  -> ½ (p fixed) We concentrate on the ‘High-SNR regime’  -> 0, and ‘almost-memoryless regime’ p-> ½. p  0 0 ½ ½ For High-SNR (η= λ,µ) : Solution can be given as a power-series in  :

20. 20 RE-Rate Taylor Series Expansion u In [Zuk,Domany,Kanter&Aizenman 06] we give a procedure for calculating the full Taylor-Series Expansion for the HMP entropy rate, in the ‘High SNR’, and ‘almost memoryless’ regime. u Main observation: Finite systems give the correct RE rate up to a given order: u Was discovered using computer experiments (symbolic computation in Maple). u Stronger result holds for the entropy rate (orders ‘settle’ for N ≥ (k+3)/2) u Does not hold for any regime. For some regimes (e.g. p->0), even first order never settles.

21. 21 Proof Outline ( with M. Aizenman ) X Y (k+3)/2 H(p,  ) up to O(  k ) Two main Ideas: A.To distinguish between noise at different site    2  3  ….  j …. B. When  m =0, the observation Y m =X m, conditioning back to the past is ‘blocked’  m =0 k+2 H(λ) D(λ||µ)

22. 22 Overview u Introduction u Distance Measures and Relative Entropy rate u Results: Generalization from Entropy Rate. u Future Directions

23. 23 RE-Rate Taylor Series Expansion u First order : u Higher orders were computed for the binary symmetric case. u Similar results for the ‘almost-memoryless’ regime. u Radius of convergence seems larger for the latter expansion, albeit no rigorous results are known.

24. 24 Future Directions oStudy other regimes. (e.g. two ‘close’ models). oBehavior of the EM algorithm. oGeneralizations (e.g. different alphabets sizes, continuous case). oPhysical realization of HMPs (mesoscopic systems, quantum jumps) oDomain of Analyticity - Radius of convergence.

25. 25 Thanks oEytan Domany (Weizmann Inst.) oIdo Kanter (Bar-Ilan Univ.) oMichael Aizenman (Princeton Univ.) oLibi Hertzberg (Weizmann Inst.)