1. Statistical-Mechanical Approach to Probabilistic Image Processing -- Loopy Belief Propagation and Advanced Mean-Field Method --
Kazuyuki Tanaka and Noriko Yoshiike
Graduate School of Information Sciences
Tohoku University
I will talk about Statistical-Mechanical Approach to Probabilistic Image Processing.
In this talk, we apply the Bethe approximation to the Bayesian image analysis.
The Bethe approximation is one of advanced mean field methods
And the extremum condition of the Bethe free energy is equivalent to the belief propagation
in probabilistic inference.
A part of the present work is research collaboration
with Professors Jun-ichi Inoue and Mike Titterington.
Collaborators: J. Inoue (Hokkaido University),
D. M. Titterington (University of Glasgow)

2. Image Processing and Magnetic Material
Regular lattice consisting of a lot of nodes.
Interactions among neighboring nodes
Output images are determined from a priori information and given data.
Ordered states are determined from interactions and external fields.
Similarity
Image processing and magnetic material have two similar theoretical structures with each other.
The similarities are very simple.
They are constructed from many elements.
The many elements interact with each other.
As you know, the classical spin systems have a lot of spins
and each spin interacts with the neighboring spins.
The ordered state is determined from interactions and external fields.
On the other hand, images are constructed from a lot of pixels.
Output of each pixel are determined from the input values of the neighboring pixels.
Output images are determined from a priori information and given data.
Thus, the conventional image processing theory and the classical spin system have the similar structure.
In the classical spin system, the phase transition occurs at a critical temperature.
Below the critical temperature, the classical spin system has a disordered state, for example Paramagnetic state.
Above the critical temperature, the classical spin system has an ordered state, for example Ferromagnetic state.
In statistical mechanics, many researchers investigate the classical spin systems near the critical temperature.
Near the critical temperature, fluctuation is enhanced.
Statistical mechanics has a lot of techniques to treat fluctuation near the critical temperature.
On the other hand, it is difficult for conventional filters to treat fluctuation in data systematically.
In the present study, the statistical mechanical strategies are employed to treat fluctuation in data in image processing.
Para
Critical
Ferro
It is difficult for conventional filters to treat fluctuation in data.
Fluctuation is enhanced
near critical temperature.

3. Probabilistic Model and Image Restoration
Noise
Transmission
Original Image
Degraded Image
First of all, I explain the framework of Bayesian analysis for image restoration.
First we have an original image and the original image has been degraded by a noise for example in a transmission.
The image restoration is a problem to estimate the original image from the given degraded image.
Now we assume that the original images that we treat in the real world are produced by a probability which is called a priori probability.
And we assume that the degradation process also can be expressed in terms of a conditional probability of the degraded image when the original image is given.
By using Bayes formula, the conditional probability of the original image when the degraded image is given is constructed as a posteriori probability.
The original image can be estimated by means of the a posteriori probability.

4. Degradation Process and A Priori Probability in Binary Image Restoration
Degradation Process (Binary Symmetric Channel)
A Priori
Probability
Now we consider a binary image restoration.
The original image and degraded image are denoted by f and g, respectively.
In the degradation process,
it is assumed that the degraded image g is generated
from the original image f by changing the intensity
of each pixel to another intensity with the same probability p,
independently of the other pixels.
The probability that the intensity is
unchanged is therefore 1 - (Q - 1)p.
The conditional probability associated with the degradation process
when the original image is f is given as Pr{G=g|F=f}.
Moreover, the a priori probability that the original image is f
is assumed to be Pr{F=f}.

5. A Priori Probability in Binary Image Restoration
We know that the a priori probability is the spin-1/2 Ising model
And alpha is a ferromagnetic interaction.
Three original images f are simulated from the a priori probability for the spin-1/2 Ising model.
The value of critical point for alpha is …

6. Bayes Formula and A Posteriori Probability
By substitute the degradation process and the a priori probability to Bayes formula,
We obtain the explicit expression of the a posteriori probability.
This model is corresponding to the two-dimensional Ising model
with the nearest-neighbour interactions and spatially non-uniform external fields.

7. Maximization of Posterior Marginal
We introduce the marginal probabilities for the a posteriori probability Pr{F=f|G=g}.
The marginal probabilities are corresponding to one-body distribution function
and two-body distribution function in the statistical mechanics.
The restored image are determined so as to maximize the posterior marginal probability at each pixel.
In this framework, we have to calculate the marginal probability at each pixel.
Maximization of
Posterior Marginal

8. Deterministic Equation of Loopy Belief Propagation
In the Bethe approximation, the marginal probabilities are assumed to be the following form in terms of the messages from the neighboring pixels to the pixel.
These marginal probabilities satisfy the reducibility conditions at each pixels and each nearest-neighbor pair of pixels.
The messages are determined so as to satisfy the reducibility conditions.

9. Message Update Rule of Loopy Belief Propagation
Fixed-Point Equations
The reducibility conditions can be rewritten as the following fixed point equations.
This fixed point equations is corresponding to the extremum condition of the Bethe free energy.
And the fixed point equations can be numerically solved by using the natural iteration.
The algorithm is corresponding to the loopy belief propagation.
Natural Iteration

10. Free Energy of A Priori Probabilitic Model in Loopy Belief Propagation (Bethe Approx.)
0.5
0.4
Second Order Phase Transition
-0.5
0.3
-1.0
0.2
We know that the a priori probability is the spin-1/2 Ising model
And alpha is a ferromagnetic interaction.
Three original images f are simulated from the a priori probability for the spin-1/2 Ising model.
The value of critical point for alpha is …
-1.5
0.1
-2.0
0.5
1.0
0.5
1.0

11. Binary Image Restoration
Original images are generated by Monte Carlo simulations in the a priori probability.
Degraded Image (p=0.2)
Original Image
Restored Image
Three original images f are simulated from the a priori probability for the spin-1/2 Ising model.
From these three original images, the degraded images are generated by setting p=0.2.
By applying the loopy belief propagation algorithm, we obtain the restored image as enough good results.

12. Hyperparameter Estimation
Maximization of Marginal Likelihood
Marginalize
In the image restoration, we usually have to estimate the hyperparameters alpha and p.
In statistics, the maximum likelihood estimation is often employed.
In the standpoint of maximum likelihood estimation, the hyperparameters are determined so as to maximize the marginal likelihood defined by marginalize the joint probability for the original image and degraded image with respect to the original image.
The marginal likelihood is expressed in terms of the partition functions of the a priori probabilistic model and the a posteriori probabilistic model.
We can calculate these partition functions approximately by using the Bethe approximation.

13. Binary Image Restoration
Original images are generated
by Monte Carlo simulations in the a priori probability.
Degraded Image (p=0.2)
Original Image
Restored Image
These are the corresponding results by employing the maximization of marginal likelihood.
We see that the estimates of the hyperparameters are good.

14. Binary Image Restoration
Hyperparameters are determined
so as to maximize the marginal likelihood.
Loopy Belief Propagation
Original Image
Degraded Image
This is a more practical numerical experiments.
The left ones are the original images.
The middle ones are the degraded images and the right ones are the restored images.

15. Degradation Process in Multi-Valued Image Restoration
Next, we consider the multi-valued case.
The degradation process is assumed to be the extended ones of the binary symmetric channel.

16. A Priori Probability in Multi-Valued Image Restoration
Q-Ising Model
Q-state Potts Model
As a priori probability, we assume two different kinds of probabilistic models.
One is a Q-Ising model and the other one is a Q-state Potts model.

17. Free Energy of Q-Ising Model (Q=2) in Loopy Belief Propagation (Bethe Approx.)０
0.5
0.4
Second Order Phase Transition
-0.5
0.3
-1.0
0.2
We know that the a priori probability is the spin-1/2 Ising model
And alpha is a ferromagnetic interaction.
Three original images f are simulated from the a priori probability for the spin-1/2 Ising model.
The value of critical point for alpha is …
-1.5
0.1
-2.0 ０
1.0
2.0 ０
1.0
2.0

18. Free Energy of Q-Ising Model (Q=4) in Loopy Belief Propagation (Bethe Approx.)
0.5
0.4
Second Order Phase Transition
-0.5
0.3
-1.0
0.2
We know that the a priori probability is the spin-1/2 Ising model
And alpha is a ferromagnetic interaction.
Three original images f are simulated from the a priori probability for the spin-1/2 Ising model.
The value of critical point for alpha is …
-1.5
0.1
-2.0
1.0
2.0
1.0
2.0

19. Free Energy of Q-state Potts Model (Q=2) in Loopy Belief Propagation (Bethe Approx.)
-0.2
Second Order Phase Transition
-1.0
-0.4
-2.0
-0.6
We know that the a priori probability is the spin-1/2 Ising model
And alpha is a ferromagnetic interaction.
Three original images f are simulated from the a priori probability for the spin-1/2 Ising model.
The value of critical point for alpha is …
-3.0
-0.8
-4.0
-1.0
1.0
2.0
1.0
2.0

20. Free Energy of Q-state Potts Model (Q=4) in Loopy Belief Propagation (Bethe Approx.)
-1.0
-2.0
-4.0
-3.0
-0.2
-0.4
-0.6
-0.8
First Order Phase Transition
We know that the a priori probability is the spin-1/2 Ising model
And alpha is a ferromagnetic interaction.
Three original images f are simulated from the a priori probability for the spin-1/2 Ising model.
The value of critical point for alpha is …
1.0
2.0
1.0
2.0

21. Multi-Valued Image Restoration (Q=4)
Hyperparameters are determined so as to maximize the marginal likelihood.
Q-Ising Model
Original Image
Degraded Image
Restored Image
The original images f are simulated from the two a priori probabilities.
For these two original images, the degraded images are generated at 30% degraded rate.
By applying the loopy belief propagation algorithm and the maximization of marginal likelihood, we obtain the restored image as enough good results.
Q-state Potts Model

22. Multi-Valued Image Restoration (Q=4)
Hyperparameters are determined so as to maximize the marginal likelihood.
Q-Ising Model
Original Image
Degraded Image
Restored Image
This is an alpha and p dependences of the logarithmic of marginal likelihood per pixel in the Q-Ising model.

23. Multi-Valued Image Restoration (Q=4)
Hyperparameters are determined so as to maximize the marginal likelihood.
Q-state Potts Model
Original Image
Degraded Image
Restored Image
This is an alpha and p dependences of the logarithmic of marginal likelihood per pixel in the Q-state Potts model.

24. Multi-Valued Image Restoration (Q=4)
Hyperparameters are determined so as to maximize the marginal likelihood.
Degraded Image（3p=0.3）
Q-state Potts Model
Q-Ising Model
Original Image
This is a more practical numerical experiments.

25. Gray-Level Image Restoration
Original Image
Degraded
Image
Belief
Propagation
Lowpass Filter
Median Filter
MSE: 2075
MSE: 244
MSE: 217
MSE:135
Finally, we show only the results for the gray-level image restoration.
For each numerical experiments, the loopy belief propagation ca give us better results than the ones by conventional filters.
MSE: 3469
MSE: 371
MSE: 523
MSE: 395

26. Summary
Probabilistic Image Processing by Bayes Formula and Loopy Belief Propagation
Some Numerical Experiments
Future Problems
Segmentation
Image Compression
Motion Detection
Color Image
EM algorithm
Statistical Performance
Line Fields
In this talk, we show the framework of probabilistic image processing by means of the Bayes formula and loopy belief propagation.
As future problems, we will apply the present framework to the more practical applications.
And, in the standpoint of fundamental theoretical studies, we will investigate the introduction of EM algorithm in the present framework and statistical performance estimation by using the replica method.

27. Appendix A: Graphical Probabilistic Model

28. Appendix A: Kullback-Leibler divergence

29. Appendix A: Bethe Free Energy

30. Appendix A: Basic Framework of Bethe Approximation

31. Appendix A: Propagation Rule of Bethe Approximation
Update Rule is reduced to Loopy Belief Propagation

32. Appendix B: A Priori Probability in Image Restoration

33. Appendix B: Original images are generated by Monte Carlo simulations in the a priori probability.
Degraded Image (p=0.2)
Mean Field
Approximation
Bethe Approximation
Original Image

34. Appendix B: Original images are generated by Monte Carlo simulations in the a priori probability.
Degraded Image (p=0.2)
Mean Field
Approx.
Original Image
Bethe Approx.

35. Appendix B: Standard Image
Mean-Field
Approx.
Original Image
Degraded Image
Bethe Approx.

36. Appendix B: Hyperparameters are determined so as to maximize the marginal likelihood.
Mean-Field Approx.
Bethe Approx.
Original Image
Degraded Image

37. Appendix C: Ising Model

38. Appendix C: Exact Results in Thermodynamic Limit

39. Appendix C: Order Parameter
1.0
Mean Field Approx.
Exact
Bethe Approx.
0.5
Kikuchi Approx. 1 2 3 4 5

40. Appendix C: Free Energy of A Priori Probabilitic Model in Loopy Belief Propagation (Bethe Approx.)０
-0.1
Second Order Phase Transition
-0.5
-0.2
-1.0
-0.3
-1.5
-0.4
-2.0
-0.5 ０
0.5
1.0 ０
0.5
1.0