1. Phase Transitions in the Information Distortion NIPS 2003 workshop on Information Theory and Learning: The Bottleneck and Distortion Approach December 13, 2003 Albert E. Parker Department of Mathematical Sciences Center for Computational Biology Montana State University Collaborators: Tomas Gedeon, Alex Dimitrov, John Miller, and Zane Aldworth

2. The Goal: To determine the phase transitions or the bifurcation structure of solutions to clustering problems of the form max q  G(q) constrained by D(q)  I 0 where  is the set of valid conditional probabilities in R NK. G and D are sufficiently smooth in . G and D have symmetry: they are invariant to relabelling of the classes of Z. The Hessians  q G and  q D are block diagonal. XZ q(Z|X) K objectsN clusters

3. A similar formulation: Using the method Lagrange multipliers, the goal of determining the bifurcation structure of solutions of the optimization problem can be rephrased as finding the bifurcation structure of stationary points of the problem max q  (G(q)+  D(q)) where   [0,  ).  is the set of valid conditional probabilities in R NK. G and D are sufficiently smooth in . G and D have symmetry: they are invariant to relabelling of the classes of Z. The Hessian  q (G+  D) is block diagonal, and satisfies a set of regularity conditions at bifurcation: (e.g. the kernel of each block is one dimensional) XZ q(Z|X) K objectsN clusters

4. How: Use the Symmetries By capitalizing on the symmetries of the cost functions, we have described the bifurcation structure of stationary points to problems of the form max q  G(q) constrained by D(q)  I 0 or max q  (G(q)+  D(q)) where   [0,  ).  is the set of valid conditional probabilities in R NK. G and D are sufficiently smooth in . G and D have symmetry: they are invariant to relabelling of the classes of Z. The Hessian  q (G+  D) is block diagonal, and satisfies a set of regularity conditions at bifurcation: (e.g. the kernel is one dimensional)

5. Rate Distortion Theory (Shannon 1950’s) Minimal Informative Compression min I(X,Z) constrained by D(X,Z)  D 0 Deterministic Annealing (Rose 1990’s) A Clustering Algorithm max H(Z|X) constrained by D(X,Z)  D 0 Examples optimizing at a distortion level D(Y,Z)  D 0

6. Rate Distortion Theory (Shannon 1950’s) Minimal Informative Compression max -I(X,Z) constrained by D(X,Z)  D 0 Deterministic Annealing (Rose 1998) A Clustering Algorithm max H(Z|X) constrained by D(X,Z)  D 0 I(X,Z)=H(Z) – H(Z|X) Examples optimizing at a distortion level D(Y,Z)  D 0

7. Y X InputsOutputs Z q(Z|X) Clustered Outputs K objects {x i } N objects {z i }L objects {y i } p(X,Y) Inputs and Outputs and Clustered Outputs

8. Y X InputsOutputs Z q(Z|X) Clustered Outputs K objects {x i } N objects {z i }L objects {y i } p(X,Y) Inputs and Outputs and Clustered Outputs

9. Information Bottleneck Method (Tishby, Pereira, Bialek 1999) min I(X,Z) constrained by D I (X,Z)  D 0 max –I(X,Z) +  I(Y;Z) Information Distortion Method (Dimitrov and Miller 2001) max H(Z|X) constrained by D I (X,Z)  D 0 max H(Z|X) +  I(Y;Z) Two methods which use an information distortion function to cluster

10. Information Bottleneck Method (Tishby, Pereira, Bialek 1999) min I(X,Z) constrained by D I (X,Z)  D 0 max –I(X,Z) +  I(Y;Z) Information Distortion Method (Dimitrov and Miller 2001) max H(Z|X) constrained by D I (X,Z)  D 0 max H(Z|X) +  I(Y;Z) Two methods which use an information distortion function to cluster The Hessian is always singular … (-I(X,Z) is not strictly concave) The theory which follows does not apply

11. Information Bottleneck Method (Tishby, Pereira, Bialek 1999) min I(X,Z) constrained by D I (X,Z)  D 0 max –I(X,Z) +  I(Y;Z) Information Distortion Method (Dimitrov and Miller 2001) max H(Z|X) constrained by D I (X,Z)  D 0 max H(Z|X) +  I(Y;Z) Two methods which use an information distortion function to cluster The Hessian is always singular … (I(X,Z) is not strictly concave) The theory which follows does not apply H(Z|X) is strictly concave) The theory which follows does apply

12. A basic annealing algorithm to solve max q  (G(q)+  D(q)) Let q 0 be the maximizer of max q G(q), and let  0 =0. For k  0, let (q k,  k ) be a solution to max q G(q) +  D(q ). Iterate the following steps until  K =  max for some K. 1.Perform  -step: Let  k+1 =  k + d k where d k >0 2.The initial guess for q k+1 at  k+1 is q k+1 (0) = q k +  for some small perturbation . 3.Optimization: solve max q (G(q) +  k+1 D(q)) to get the maximizer q k+1, using initial guess q k+1 (0).

13. Application of the annealing method to the Information Distortion problem max q  (H(Z|X) +  I(X;Z)) when p(X,Y) is defined by four gaussian blobs Y, Inputs X, Outputs YX K=52 outputs L=52 inputs p(X,Y) XZ q(Z|X) K=52 outputsN=4 clustered outputs X, Outputs Z, Clustered Outputs

14. Evolution of the optimal clustering: Observed Bifurcations for the Four Blob problem: We just saw the optimal clusterings q * at some  * =  max. What do the clusterings look like for  <  max ?? I(Y,Z) bits

15. ?????? Why are there only 3 bifurcations observed? In general, are there only N-1 bifurcations? What kinds of bifurcations do we expect: pitchfork-like, transcritical, saddle-node, or some other type? How many bifurcating branches are there? What do the bifurcating branches look like? Are they 1 st order phase transitions (subcritical) or 2 nd order phase transitions (supercritical) ? What is the stability of the bifurcating branches? Is there always a bifurcating branch which contains solutions of the optimization problem? Are there bifurcations after all of the classes have resolved ?  q* Conceptual Bifurcation Structure Observed Bifurcations for the 4 Blob Problem I(Y,Z) bits

16. Recall the Symmetries: To better understand the bifurcation structure, we capitalize on the symmetries of the function G(q)+  D(q) X Z q(Z|X) : a clustering K objects {x i } N objects {z i } class 1 class 3

17. X Z q(Z|X) : a clustering K objects {x i } N objects {z i } class 3 class 1 Recall the Symmetries: To better understand the bifurcation structure, we capitalize on the symmetries of the function G(q)+  D(q)

18. The symmetry group of all permutations on N symbols is S N.

19. A partial subgroup lattice for S N when N=4.

20. A partial lattice of the maximal subgroups S 2 x S 2 of S 4

21. This Group Structure determines the Bifurcation Structure

22. Define a Gradient Flow Goal: To determine the bifurcation structure of stationary points of max q  (G(q) +  D(q)) Method: Study the equilibria of the of the flow Equilibria of this system (in R NK+K ) are possible solutions of the optimization problem The Jacobian  q, L (q *,  * ) is symmetric, and so only bifurcations of equilibria can occur. The first equilibrium is q * (  0 = 0)  1/N. If w T  q F(q *,  ) w < 0 for every w  ker J, then q * (  ) is a maximizer of. The first equilibrium is q*(  0 = 0)  1/N.

23. Symmetry Breaking Bifurcations  q*

24. Symmetry Breaking Bifurcations  q*

25. Symmetry Breaking Bifurcations  q*

26. Symmetry Breaking Bifurcations  q*

27. Symmetry Breaking Bifurcations  q*

28. Symmetry Breaking Bifurcations  q*

29. Existence Theorems for Bifurcating Branches Given a bifurcation at a point fixed by S N, Equivariant Branching Lemma The Smoller-Wasserman Theorem (Vanderbauwhede and Cicogna 1980-1) (Smoller and Wasserman 1985-6) There are N bifurcating branches, each which have symmetry S N-1. There are N!/(2m!n!) bifurcating branches which have symmetry S m x S n if N=m+n.  q*

30.  Existence Theorems for Bifurcating Branches Given a bifurcation at a point fixed by S N-1, Equivariant Branching Lemma The Smoller-Wasserman Theorem (Vanderbauwhede and Cicogna 1980-1) (Smoller and Wasserman 1985-6) There are N-1 bifurcating branches, each which have symmetry S N-2. There are (N-1)!/(2m!n!) bifurcating branches which have symmetry S m x S n if N-1=m+n.

31. Group Structure Observed Bifurcation Structure

32. Group Structure  q* Observed Bifurcation Structure The Equivariant Branching Lemma shows that the bifurcation structure contains the branches …

33. Group Structure  q* Observed Bifurcation Structure The subgroups {S 2 x S 2 } give additional structure …

34. Group Structure  q* Observed Bifurcation Structure The subgroups {S 2 x S 2 } give additional structure …

35.  q* Theorem: There are at exactly K bifurcations on the branch (q 1/N,  ) whenever G(q 1/N ) is nonsingular There are K=52 bifurcations on the first branch Observed Bifurcation Structure

36. A partial subgroup lattice for S 4 and the corresponding bifurcating directions given by the Equivariant Branching Lemma

37. A partial subgroup lattice for S 4 and the corresponding bifurcating directions corresponding to subgroups isomorphic to S 2 x S 2.

38. This theory enables us to answer the questions previously posed …

39. ?????? Why are there only 3 bifurcations observed? In general, are there only N-1 bifurcations? What kinds of bifurcations do we expect: pitchfork-like, transcritical, saddle-node, or some other type? How many bifurcating solutions are there? What do the bifurcating branches look like? Are they subcritical or supercritical ? What is the stability of the bifurcating branches? Is there always a bifurcating branch which contains solutions of the optimization problem? Are there bifurcations after all of the classes have resolved ?  q* Conceptual Bifurcation Structure Observed Bifurcations for the 4 Blob Problem

40. Why are there only 3 bifurcations observed? In general, are there only N-1 bifurcations? There are N-1 symmetry breaking bifurcations from S M to S M-1 for M  N. What kinds of bifurcations do we expect: pitchfork-like, transcritical, saddle-node, or some other type? How many bifurcating solutions are there? There are at least N from the first bifurcation, at least N-1 from the next one, etc. What do the bifurcating branches look like? They are subcritical or supercritical depending on the sign of the bifurcation discriminator  (q *,  *,u k ). What is the stability of the bifurcating branches? Is there always a bifurcating branch which contains solutions of the optimization problem? No. Are there bifurcations after all of the classes have resolved ? Generically, no. Conceptual Bifurcation Structure  q*

41. Continuation techniques numerically illustrate the theory using the Information Distortion

42.  q* I(Y,Z) bits

43. Bifurcating branches with symmetry S 2 x S 2 =  q* I(Y,Z) bits

44. Additional structure!! I(Y,Z) bits

46. A closer look …  q* I(Y,Z) bits

47. Bifurcation from S 4 to S 3 …  q* I(Y,Z) bits

48. The bifurcation from S 4 to S 3 is subcritical … (the theory predicted this since the bifurcation discriminator  (q 1/4,  *,u)<0 ) I(Y,Z) bits

49. (4) R H (I 0 ) = max q  H(Z|X) constrained by I(Y,Z)  I 0 (7) max q  (H(Z|X) +  I(Y,Z)) What does this mean regarding solutions of the original problems? I(Y,Z) bits

50. Theorem: dR/dI 0 = -  (I 0 ) d 2 R/dI 0 2 = -d  (I 0 )/dI 0 (4) R H (I 0 ) = max q  H(Z|X) constrained by I(Y,Z)  I 0 (7) max q  (H(Z|X) +  I(Y,Z))

51. Theorem: dR/dI 0 = -  (I 0 ) d 2 R/dI 0 2 = d  (I 0 )/dI 0 R H as a function of I 0 R H (I 0 ) = max q  H(Z|Y) constrained by I(X;Z)  I 0 is not convex and not concave is a monotonically decreasing, continuous function RHRH

52. Consequences?? Analogue for the Information Distortion R H (I 0 ) = max q  H(Z|X) constrained by I(Y;Z)  I 0 is neither concave nor convex since subcritical bifurcations and saddle nodes exist. Rate Distortion Function (from Information Theory) R(D 0 ) = min q  I(X;Z) constrained by D(X,Z)  D 0 is convex if D(Y,Z) is linear in q (Rose, 1994; Cover and Thomas; Grey). Relevance Compression Function (for Information Bottleneck) R I (I 0 ) = min q  I(X;Z) constrained by I(Y;Z)  I 0 is convex if N>K+1 (Witsenhausen and Wyner 1975, Bachrach et al 2003)

53. Analogue for the Information Distortion R H (I 0 ) = max q  H(Z|X) constrained by I(Y;Z)  I 0 is neither concave nor convex since subcritical bifurcations and saddle nodes exist. Relevance Compression Function (for Information Bottleneck) R I (I 0 ) = min q  I(X;Z) constrained by I(Y;Z)  I 0 is convex if N>K+1 (Bachrach et al 2003) R I (I 0 ) and R H (I 0 ) are related by I(X;Z) = H(Z) - H(Z|X). The Information Bottleneck can not have a subcritical bifurcation when N > K+1. Are there subcritical bifurcations when N<K+1 ? Is R H (I 0 ) convex when N>K+1 ? That would mean that the subcritical bifurcations go away when considering the gradient flow in R (K+2)K instead of R NK. So What??

55. Application to cricket sensory data E(Y|Z): stimulus means conditioned on each of the classes spike patterns optimal clustering

56. Conclusions … We have a complete theoretical picture of how the clusterings evolve for a class of annealing problems of the form max q  (G(q)+  D(q)) subject to the assumptions stated earlier. oWhen clustering to N classes, there are N-1 bifurcations. oIn general, there are only pitchfork and saddle-node bifurcations. oWe can determine whether pitchfork bifurcations are either subcritical or supercritical (1 st or 2 nd order phase transitions) oWe know the explicit bifurcating directions SO WHAT?? There are theoretical consequences … This suggests an algorithm for solving the annealing problem … (NIPS 2002)