Statistical-Mechanical Approach to Probabilistic Image Processing -- Loopy Belief Propagation and Advanced Mean-Field Method -- Kazuyuki Tanaka and Noriko Yoshiike Graduate School of Information Sciences Tohoku University kazu@statp.is.tohoku.ac.jp http://www.statp.is.tohoku.ac.jp/~kazu/ I will talk about Statistical-Mechanical Approach to Probabilistic Image Processing. In this talk, we apply the Bethe approximation to the Bayesian image analysis. The Bethe approximation is one of advanced mean field methods And the extremum condition of the Bethe free energy is equivalent to the belief propagation in probabilistic inference. A part of the present work is research collaboration with Professors Jun-ichi Inoue and Mike Titterington. Collaborators: J. Inoue (Hokkaido University), D. M. Titterington (University of Glasgow)

Image Processing and Magnetic Material Regular lattice consisting of a lot of nodes. Interactions among neighboring nodes Output images are determined from a priori information and given data. Ordered states are determined from interactions and external fields. Similarity Image processing and magnetic material have two similar theoretical structures with each other. The similarities are very simple. They are constructed from many elements. The many elements interact with each other. As you know, the classical spin systems have a lot of spins and each spin interacts with the neighboring spins. The ordered state is determined from interactions and external fields. On the other hand, images are constructed from a lot of pixels. Output of each pixel are determined from the input values of the neighboring pixels. Output images are determined from a priori information and given data. Thus, the conventional image processing theory and the classical spin system have the similar structure. In the classical spin system, the phase transition occurs at a critical temperature. Below the critical temperature, the classical spin system has a disordered state, for example Paramagnetic state. Above the critical temperature, the classical spin system has an ordered state, for example Ferromagnetic state. In statistical mechanics, many researchers investigate the classical spin systems near the critical temperature. Near the critical temperature, fluctuation is enhanced. Statistical mechanics has a lot of techniques to treat fluctuation near the critical temperature. On the other hand, it is difficult for conventional filters to treat fluctuation in data systematically. In the present study, the statistical mechanical strategies are employed to treat fluctuation in data in image processing. Para Critical Ferro It is difficult for conventional filters to treat fluctuation in data. Fluctuation is enhanced near critical temperature.

Probabilistic Model and Image Restoration Noise Transmission Original Image Degraded Image First of all, I explain the framework of Bayesian analysis for image restoration. First we have an original image and the original image has been degraded by a noise for example in a transmission. The image restoration is a problem to estimate the original image from the given degraded image. Now we assume that the original images that we treat in the real world are produced by a probability which is called a priori probability. And we assume that the degradation process also can be expressed in terms of a conditional probability of the degraded image when the original image is given. By using Bayes formula, the conditional probability of the original image when the degraded image is given is constructed as a posteriori probability. The original image can be estimated by means of the a posteriori probability.

Degradation Process and A Priori Probability in Binary Image Restoration Degradation Process (Binary Symmetric Channel) A Priori Probability Now we consider a binary image restoration. The original image and degraded image are denoted by f and g, respectively. In the degradation process, it is assumed that the degraded image g is generated from the original image f by changing the intensity of each pixel to another intensity with the same probability p, independently of the other pixels. The probability that the intensity is unchanged is therefore 1 - (Q - 1)p. The conditional probability associated with the degradation process when the original image is f is given as Pr{G=g|F=f}. Moreover, the a priori probability that the original image is f is assumed to be Pr{F=f}.

A Priori Probability in Binary Image Restoration We know that the a priori probability is the spin-1/2 Ising model And alpha is a ferromagnetic interaction. Three original images f are simulated from the a priori probability for the spin-1/2 Ising model. The value of critical point for alpha is 0.4406…

Bayes Formula and A Posteriori Probability By substitute the degradation process and the a priori probability to Bayes formula, We obtain the explicit expression of the a posteriori probability. This model is corresponding to the two-dimensional Ising model with the nearest-neighbour interactions and spatially non-uniform external fields.

Maximization of Posterior Marginal We introduce the marginal probabilities for the a posteriori probability Pr{F=f|G=g}. The marginal probabilities are corresponding to one-body distribution function and two-body distribution function in the statistical mechanics. The restored image are determined so as to maximize the posterior marginal probability at each pixel. In this framework, we have to calculate the marginal probability at each pixel. Maximization of Posterior Marginal

Deterministic Equation of Loopy Belief Propagation In the Bethe approximation, the marginal probabilities are assumed to be the following form in terms of the messages from the neighboring pixels to the pixel. These marginal probabilities satisfy the reducibility conditions at each pixels and each nearest-neighbor pair of pixels. The messages are determined so as to satisfy the reducibility conditions.

Message Update Rule of Loopy Belief Propagation Fixed-Point Equations The reducibility conditions can be rewritten as the following fixed point equations. This fixed point equations is corresponding to the extremum condition of the Bethe free energy. And the fixed point equations can be numerically solved by using the natural iteration. The algorithm is corresponding to the loopy belief propagation. Natural Iteration

Free Energy of A Priori Probabilitic Model in Loopy Belief Propagation (Bethe Approx.) 0.5 0.4 Second Order Phase Transition -0.5 0.3 -1.0 0.2 We know that the a priori probability is the spin-1/2 Ising model And alpha is a ferromagnetic interaction. Three original images f are simulated from the a priori probability for the spin-1/2 Ising model. The value of critical point for alpha is 0.4406… -1.5 0.1 -2.0 0.5 1.0 0.5 1.0

Binary Image Restoration Original images are generated by Monte Carlo simulations in the a priori probability. Degraded Image (p=0.2) Original Image Restored Image Three original images f are simulated from the a priori probability for the spin-1/2 Ising model. From these three original images, the degraded images are generated by setting p=0.2. By applying the loopy belief propagation algorithm, we obtain the restored image as enough good results.

Hyperparameter Estimation Maximization of Marginal Likelihood Marginalize In the image restoration, we usually have to estimate the hyperparameters alpha and p. In statistics, the maximum likelihood estimation is often employed. In the standpoint of maximum likelihood estimation, the hyperparameters are determined so as to maximize the marginal likelihood defined by marginalize the joint probability for the original image and degraded image with respect to the original image. The marginal likelihood is expressed in terms of the partition functions of the a priori probabilistic model and the a posteriori probabilistic model. We can calculate these partition functions approximately by using the Bethe approximation.

Binary Image Restoration Original images are generated by Monte Carlo simulations in the a priori probability. Degraded Image (p=0.2) Original Image Restored Image These are the corresponding results by employing the maximization of marginal likelihood. We see that the estimates of the hyperparameters are good.

Binary Image Restoration Hyperparameters are determined so as to maximize the marginal likelihood. Loopy Belief Propagation Original Image Degraded Image This is a more practical numerical experiments. The left ones are the original images. The middle ones are the degraded images and the right ones are the restored images.

Degradation Process in Multi-Valued Image Restoration Next, we consider the multi-valued case. The degradation process is assumed to be the extended ones of the binary symmetric channel.

A Priori Probability in Multi-Valued Image Restoration Q-Ising Model Q-state Potts Model As a priori probability, we assume two different kinds of probabilistic models. One is a Q-Ising model and the other one is a Q-state Potts model.

Free Energy of Q-Ising Model (Q=2) in Loopy Belief Propagation (Bethe Approx.) ０ 0.5 0.4 Second Order Phase Transition -0.5 0.3 -1.0 0.2 We know that the a priori probability is the spin-1/2 Ising model And alpha is a ferromagnetic interaction. Three original images f are simulated from the a priori probability for the spin-1/2 Ising model. The value of critical point for alpha is 0.4406… -1.5 0.1 -2.0 ０ 1.0 2.0 ０ 1.0 2.0

Free Energy of Q-Ising Model (Q=4) in Loopy Belief Propagation (Bethe Approx.) 0.5 0.4 Second Order Phase Transition -0.5 0.3 -1.0 0.2 We know that the a priori probability is the spin-1/2 Ising model And alpha is a ferromagnetic interaction. Three original images f are simulated from the a priori probability for the spin-1/2 Ising model. The value of critical point for alpha is 0.4406… -1.5 0.1 -2.0 1.0 2.0 1.0 2.0

Free Energy of Q-state Potts Model (Q=2) in Loopy Belief Propagation (Bethe Approx.) -0.2 Second Order Phase Transition -1.0 -0.4 -2.0 -0.6 We know that the a priori probability is the spin-1/2 Ising model And alpha is a ferromagnetic interaction. Three original images f are simulated from the a priori probability for the spin-1/2 Ising model. The value of critical point for alpha is 0.4406… -3.0 -0.8 -4.0 -1.0 1.0 2.0 1.0 2.0

Free Energy of Q-state Potts Model (Q=4) in Loopy Belief Propagation (Bethe Approx.) -1.0 -2.0 -4.0 -3.0 -0.2 -0.4 -0.6 -0.8 First Order Phase Transition We know that the a priori probability is the spin-1/2 Ising model And alpha is a ferromagnetic interaction. Three original images f are simulated from the a priori probability for the spin-1/2 Ising model. The value of critical point for alpha is 0.4406… 1.0 2.0 1.0 2.0

Multi-Valued Image Restoration (Q=4) Hyperparameters are determined so as to maximize the marginal likelihood. Q-Ising Model Original Image Degraded Image Restored Image The original images f are simulated from the two a priori probabilities. For these two original images, the degraded images are generated at 30% degraded rate. By applying the loopy belief propagation algorithm and the maximization of marginal likelihood, we obtain the restored image as enough good results. Q-state Potts Model

Multi-Valued Image Restoration (Q=4) Hyperparameters are determined so as to maximize the marginal likelihood. Q-Ising Model Original Image Degraded Image Restored Image This is an alpha and p dependences of the logarithmic of marginal likelihood per pixel in the Q-Ising model.

Multi-Valued Image Restoration (Q=4) Hyperparameters are determined so as to maximize the marginal likelihood. Q-state Potts Model Original Image Degraded Image Restored Image This is an alpha and p dependences of the logarithmic of marginal likelihood per pixel in the Q-state Potts model.

Multi-Valued Image Restoration (Q=4) Hyperparameters are determined so as to maximize the marginal likelihood. Degraded Image（3p=0.3） Q-state Potts Model Q-Ising Model Original Image This is a more practical numerical experiments.

Gray-Level Image Restoration Original Image Degraded Image Belief Propagation Lowpass Filter Median Filter MSE: 2075 MSE: 244 MSE: 217 MSE:135 Finally, we show only the results for the gray-level image restoration. For each numerical experiments, the loopy belief propagation ca give us better results than the ones by conventional filters. MSE: 3469 MSE: 371 MSE: 523 MSE: 395

Summary Probabilistic Image Processing by Bayes Formula and Loopy Belief Propagation Some Numerical Experiments Future Problems Segmentation Image Compression Motion Detection Color Image EM algorithm Statistical Performance Line Fields In this talk, we show the framework of probabilistic image processing by means of the Bayes formula and loopy belief propagation. As future problems, we will apply the present framework to the more practical applications. And, in the standpoint of fundamental theoretical studies, we will investigate the introduction of EM algorithm in the present framework and statistical performance estimation by using the replica method.

Appendix A: Graphical Probabilistic Model

Appendix A: Kullback-Leibler divergence

Appendix A: Bethe Free Energy

Appendix A: Basic Framework of Bethe Approximation

Appendix A: Propagation Rule of Bethe Approximation Update Rule is reduced to Loopy Belief Propagation

Appendix B: A Priori Probability in Image Restoration

Appendix B: Original images are generated by Monte Carlo simulations in the a priori probability. Degraded Image (p=0.2) Mean Field Approximation Bethe Approximation Original Image

Appendix B: Original images are generated by Monte Carlo simulations in the a priori probability. Degraded Image (p=0.2) Mean Field Approx. Original Image Bethe Approx.

Appendix B: Standard Image Mean-Field Approx. Original Image Degraded Image Bethe Approx.

Appendix B: Hyperparameters are determined so as to maximize the marginal likelihood. Mean-Field Approx. Bethe Approx. Original Image Degraded Image

Appendix C: Ising Model

Appendix C: Exact Results in Thermodynamic Limit

Appendix C: Order Parameter 1.0 Mean Field Approx. Exact Bethe Approx. 0.5 Kikuchi Approx. 1 2 3 4 5

Appendix C: Free Energy of A Priori Probabilitic Model in Loopy Belief Propagation (Bethe Approx.) ０ -0.1 Second Order Phase Transition -0.5 -0.2 -1.0 -0.3 -1.5 -0.4 -2.0 -0.5 ０ 0.5 1.0 ０ 0.5 1.0