Physics Fluctuomatics (Tohoku University) 1 Physical Fluctuomatics 7th~10th Belief propagation Appendix Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University

Physics Fluctuomatics (Tohoku University) 2 Textbooks Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese), Chapter 5. References H. Nishimori: Statistical Physics of Spin Glasses and Information Processing, ---An Introduction, Oxford University Press, H. Nishimori, G. Ortiz: Elements of Phase Transitions and Critical Phenomena, Oxford University Press, M. Mezard, A. Montanari: Information, Physics, and Computation, Oxford University Press, 2010.

Physics Fluctuomatics (Tohoku University) 3 Probabilistic Model for Ferromagnetic Materials

Physics Fluctuomatics (Tohoku University) 4 Probabilistic Model for Ferromagnetic Materials Prior probability prefers to the configuration with the least number of red lines. > > =

Physics Fluctuomatics (Tohoku University) 5 More is different in Probabilistic Model for Ferromagnetic Materials Disordered State Ordered State Sampling by Markov Chain Monte Carlo method Small p Large p More is different. Critical Point (Large fluctuation)

Physics Fluctuomatics (Tohoku University) 6 Fundamental Probabilistic Models for Magnetic Materials Since h is positive, the probablity of up spin is larger than the one of down spin ． +1 11 h ： External Field Variance Average

Physics Fluctuomatics (Tohoku University) 7 Fundamental Probabilistic Models for Magnetic Materials Since J is positive, (a 1,a 2 )=(+1,+1) and (  1,  1) have the largest probability ． J ： Interaction Variance Average +1 11 11 11 11

Physics Fluctuomatics (Tohoku University) 8 Fundamental Probabilistic Models for Magnetic Materials Translational Symmetry J J h h E ： Set of All the neighbouring Pairs of Nodes Problem: Compute

Physics Fluctuomatics (Tohoku University) 9 Fundamental Probabilistic Models for Magnetic Materials Problem: Compute Translational Symmetry J J h h Spontaneous Magnetization

Physics Fluctuomatics (Tohoku University) 10 Mean Field Approximation for Ising Model We assume that the probability for configurations satisfying i Jm h are large.

Physics Fluctuomatics (Tohoku University) 11 Mean Field Approximation for Ising Model Fixed Point Equation of m We assume that all random variables a i are independent of each other, approximately.

Physics Fluctuomatics (Tohoku University) 12 Fixed Point Equation and Iterative Method Fixed Point Equation

Physics Fluctuomatics (Tohoku University) 13 Fixed Point Equation and Iterative Method Fixed Point Equation Iterative Method

Physics Fluctuomatics (Tohoku University) 14 Fixed Point Equation and Iterative Method Fixed Point Equation Iterative Method

Physics Fluctuomatics (Tohoku University) 15 Fixed Point Equation and Iterative Method Fixed Point Equation Iterative Method

Physics Fluctuomatics (Tohoku University) 16 Fixed Point Equation and Iterative Method Fixed Point Equation Iterative Method

Physics Fluctuomatics (Tohoku University) 17 Fixed Point Equation and Iterative Method Fixed Point Equation Iterative Method

Physics Fluctuomatics (Tohoku University) 18 Fixed Point Equation and Iterative Method Fixed Point Equation Iterative Method

Physics Fluctuomatics (Tohoku University) 19 Marginal Probability Distribution in Mean Field Approximation i Jm h Jm ： Mean Field

Physics Fluctuomatics (Tohoku University) 20 Advanced Mean Field Method h h h Bethe Approximation Kikuchi Method (Cluster Variation Meth) ： Effective Field Fixed Point Equation for  J

Physics Fluctuomatics (Tohoku University) 21 Average of Ising Model on Square Grid Graph (a)Mean Field Approximation (b)Bethe Approximation (c)Kikuchi Method (Cluster Variation Method) (d)Exact Solution （ L. Onsager ， C.N.Yang ） J J h h

Physics Fluctuomatics (Tohoku University) 22 Model Representation in Statistical Physics Gibbs Distribution Partition Function Free Energy Energy Function

Physics Fluctuomatics (Tohoku University) 23 Gibbs Distribution and Free Energy Gibbs Distribution Variational Principle of Free Energy Functional F[Q] under Normalization Condition for Q(a) Free Energy Functional of Trial Probability Distribution Q(a) Free Energy

Physics Fluctuomatics (Tohoku University) 24 Explicit Derivation of Variantional Principle for Minimization of Free Energy Functional Normalization Condition

Physics Fluctuomatics (Tohoku University) 25 Kullback-Leibler Divergence and Free Energy

Physics Fluctuomatics (Tohoku University) 26 Interpretation of Mean Field Approximation as Information Theory and Marginal Probability Distributions Q i (a i ) are determined so as to minimize D[Q|P] Minimization of Kullback-Leibler Divergence between

Physics Fluctuomatics (Tohoku University) 27 Interpretation of Mean Field Approximation as Information Theory Problem: Compute Translational Symmetry J J h h Magnetization

Physics Fluctuomatics (Tohoku University) 28 Kullback-Leibler Divergence in Mean Field Approximation for Ising Model

Physics Fluctuomatics (Tohoku University) 29 Minimization of Kullback-Leibler Divergence and Mean Field Equation Fixed Point Equations for {Q i |  i  V} Variation i Set of all the neighbouring nodes of the node i

Physics Fluctuomatics (Tohoku University) 30 Orthogonal Functional Representation of Marginal Probability Distribution of Ising Model

Physics Fluctuomatics (Tohoku University) 31 Conventional Mean Field Equation in Ising Model Fixed Point Equation J J Translational Symmetry h h

Physics Fluctuomatics (Tohoku University) 32 Interpretation of Bethe Approximation (1) Translational Symmetry J J h h Compute and

Interpretation of Bethe Approximation (2) Free Energy KL Divergence 33 Physics Fluctuomatics (Tohoku University)

Interpretation of Bethe Approximation (3) Bethe Free Energy Free Energy KL Divergence 34 Physics Fluctuomatics (Tohoku University)

Interpretation of Bethe Approximation (4) 35 Physics Fluctuomatics (Tohoku University)

Interpretation of Bethe Approximation (5) Lagrange Multipliers to ensure the constraints 36 Physics Fluctuomatics (Tohoku University)

Interpretation of Bethe Approximation (6) Extremum Condition 37 Physics Fluctuomatics (Tohoku University)

Interpretation of Bethe Approximation (7) Extremum Condition 38 Physics Fluctuomatics (Tohoku University)

Interpretation of Bethe Approximation (8) Extremum Condition 39 Physics Fluctuomatics (Tohoku University)

Interpretation of Bethe Approximation (9) Message Update Rule 40 Physics Fluctuomatics (Tohoku University)

Interpretation of Bethe Approximation (10) = Message Passing Rule of Belief Propagation It corresponds to Bethe approximation in the statistical mechanics. 41 Physics Fluctuomatics (Tohoku University)

Interpretation of Bethe Approximation (11) 42 Physics Fluctuomatics (Tohoku University) Translational Symmetry

Physics Fluctuomatics (Tohoku University) 43 Summary Statistical Physics and Information Theory Probabilistic Model of Ferromagnetism Mean Field Theory Gibbs Distribution and Free Energy Free Energy and Kullback-Leibler Divergence Interpretation of Mean Field Approximation as Information Theory Interpretation of Bethe Approximation as Information Theory