14 October, 2010LRI Seminar 2010 (Univ. Paris-Sud)1 Statistical performance analysis by loopy belief propagation in probabilistic image processing Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University, Japan Collaborators D. M. Titterington (University of Glasgow, UK) M. Yasuda (Tohoku University, Japan) S. Kataoka (Tohoku University, Japan)

2 Introduction Bayesian network is originally one of the methods for probabilistic inferences in artificial intelligence. Some probabilistic models for information processing are also regarded as Bayesian networks. Bayesian networks are expressed in terms of products of functions with a couple of random variables and can be associated with graphical representations. Such probabilistic models for Bayesian networks are referred to as Graphical Model. 14 October, 2010LRI Seminar 2010 (Univ. Paris-Sud)

3 Probabilistic Image Processing by Bayesian Network Probabilistic image processing systems are formulated on square grid graphs. Averages, variances and covariances of the Bayesian network are approximately computed by using the belief propagation on the square grid graph. 14 October, 2010LRI Seminar 2010 (Univ. Paris-Sud)

10 March, 2010IW-SMI2010 (Kyoto)4 MRF, Belief Propagation and Statistical Performance Nishimori and Wong (1999): Physical Review E Statistical Performance Estimation for MRF (Infinite Range Ising Model and Replica Theory) (Infinite Range Ising Model and Replica Theory) Is it possible to estimate the performance of belief propagation statistically? Tanaka and Morita (1995): Physics Letters A Cluster Variation Method for MRF in Image Processing CVM= Generalized Belief Propagation (GBP) Geman and Geman (1986): IEEE Transactions on PAMI Image Processing by Markov Random Fields (MRF)

14 October, 2010LRI Seminar 2010 (Univ. Paris-Sud)5 Outline 1.Introduction 2.Bayesian Image Analysis by Gauss Markov Random Fields 3.Statistical Performance Analysis for Gauss Markov Random Fields 4.Statistical Performance Analysis in Binary Markov Random Fields by Loopy Belief Propagation 5.Concluding Remarks

14 October, 2010LRI Seminar 2010 (Univ. Paris-Sud)6 Outline 1.Introduction 2.Bayesian Image Analysis by Gauss Markov Random Fields 3.Statistical Performance Analysis for Gauss Markov Random Fields 4.Statistical Performance Analysis in Binary Markov Random Fields by Loopy Belief Propagation 5.Concluding Remarks

Image Restoration by Bayesian Statistics Original Image 14 October, LRI Seminar 2010 (Univ. Paris-Sud)

Image Restoration by Bayesian Statistics Original Image Degraded Image Transmission Noise 14 October, LRI Seminar 2010 (Univ. Paris-Sud)

Image Restoration by Bayesian Statistics Original Image Degraded Image Transmission Noise Estimate 14 October, LRI Seminar 2010 (Univ. Paris-Sud)

10 Image Restoration by Bayesian Statistics Bayes Formula Original Image Degraded Image Transmission Noise Estimate Posterior 14 October, LRI Seminar 2010 (Univ. Paris-Sud)

11 Image Restoration by Bayesian Statistics Assumption 1: Original images are randomly generated by according to a prior probability. Bayes Formula Original Image Degraded Image Transmission Noise Estimate Posterior 14 October, LRI Seminar 2010 (Univ. Paris-Sud)

7 July, Image Restoration by Bayesian Statistics Bayes Formula Assumption 2: Degraded images are randomly generated from the original image by according to a conditional probability of degradation process. Original Image Degraded Image Transmission Noise Estimate Posterior 12 LRI Seminar 2010 (Univ. Paris-Sud) 14 October, 2010

LRI Seminar 2010 (Univ. Paris-Sud) 13 Bayesian Image Analysis Prior Probability Assumption: Prior Probability consists of a product of functions defined on the neighbouring pixels.

14 Bayesian Image Analysis Prior Probability Assumption: Prior Probability consists of a product of functions defined on the neighbouring pixels. 14 October, LRI Seminar 2010 (Univ. Paris-Sud)

15 Bayesian Image Analysis Patterns by MCMC. Prior Probability Assumption: Prior Probability consists of a product of functions defined on the neighbouring pixels. 14 October, LRI Seminar 2010 (Univ. Paris-Sud)

14 October, 2010LRI Seminar 2010 (Univ. Paris-Sud) 16 Bayesian Image Analysis Assumption: Degraded image is generated from the original image by Additive White Gaussian Noise. V:Set of all the pixels

17 Bayesian Image Analysis Assumption: Degraded image is generated from the original image by Additive White Gaussian Noise. 14 October, LRI Seminar 2010 (Univ. Paris-Sud)

18 Bayesian Image Analysis Assumption: Degraded image is generated from the original image by Additive White Gaussian Noise. 14 October, LRI Seminar 2010 (Univ. Paris-Sud)

14 October, 2010LRI Seminar 2010 (Univ. Paris-Sud) 19 Bayesian Image Analysis Original Image Degraded Image Prior Probability Posterior Probability Degradation Process Estimate

20 Bayesian Image Analysis Original Image Degraded Image Prior Probability Posterior Probability Degradation Process Smoothing Data Dominant Estimate 14 October, LRI Seminar 2010 (Univ. Paris-Sud)

11 March, 2010IW-SMI2010 (Kyoto) 21 Bayesian Image Analysis Original Image Degraded Image Prior Probability Posterior Probability Degradation Process Smoothing Data Dominant Bayesian Network Estimate 14 October, LRI Seminar 2010 (Univ. Paris-Sud)

22 Bayesian Image Analysis Original Image Degraded Image Prior Probability Posterior Probability Degradation Process Smoothing Data Dominant Bayesian Network Estimate 14 October, LRI Seminar 2010 (Univ. Paris-Sud)

14 October, 2010LRI Seminar 2010 (Univ. Paris-Sud)23 Image Restorations by Gaussian Markov Random Fields and Conventional Filters MSE Gauss Markov Random Field 315 Lowpass Filter (3x3)388 (5x5)413 Median Filter (3x3)486 (5x5)445 (3x3) Lowpass (5x5) Median Gauss Markov Random Field Original Image Degraded Image RestoredImage V: Set of all the pixels

14 October, 2010LRI Seminar 2010 (Univ. Paris-Sud)24 Outline 1.Introduction 2.Bayesian Image Analysis by Gauss Markov Random Fields 3.Statistical Performance Analysis for Gauss Markov Random Fields 4.Statistical Performance Analysis in Binary Markov Random Fields by Loopy Belief Propagation 5.Concluding Remarks

14 October, 2010LRI Seminar 2010 (Univ. Paris-Sud) 25 Statistical Performance by Sample Average of Numerical Experiments Original Images

26 Statistical Performance by Sample Average of Numerical Experiments Original Images Noise Pr{G|F=f,  } Observed Data 14 October, LRI Seminar 2010 (Univ. Paris-Sud)

11 March, 2010IW-SMI2010 (Kyoto) 27 Statistical Performance by Sample Average of Numerical Experiments Posterior Probability Pr{F|G=g, ,  } Original Images Noise Pr{G|F=f,  } Estimated Results Observed Data 14 October, LRI Seminar 2010 (Univ. Paris-Sud)

28 Statistical Performance by Sample Average of Numerical Experiments Posterior Probability Pr{F|G=g, ,  } Sample Average of Mean Square Error Original Images Noise Pr{G|F=f,  } Estimated Results Observed Data 14 October, LRI Seminar 2010 (Univ. Paris-Sud)

14 October, 2010LRI Seminar 2010 (Univ. Paris-Sud) 29 Statistical Performance Estimation Additive White Gaussian Noise Posterior Probability Restored Image Original ImageDegraded Image Additive White Gaussian Noise

14 October, 2010LRI Seminar 2010 (Univ. Paris-Sud) 30 Statistical Performance Estimation for Gauss Markov Random Fields   =40 

Outline 1.Introduction 2.Bayesian Image Analysis by Gauss Markov Random Fields 3.Statistical Performance Analysis for Gauss Markov Random Fields 4.Statistical Performance Analysis in Binary Markov Random Fields by Loopy Belief Propagation 5.Concluding Remarks 14 October, 2010LRI Seminar 2010 (Univ. Paris-Sud)31

32 Marginal Probability in Belief Propagation 14 October, 2010LRI Seminar 2010 (Univ. Paris-Sud) In order to compute the marginal probability Pr{F 2 |G=g}, we take summations over all the pixels except the pixel 2.

33 Marginal Probability in Belief Propagation 2 14 October, 2010LRI Seminar 2010 (Univ. Paris-Sud)

34 Marginal Probability in Belief Propagation October, 2010LRI Seminar 2010 (Univ. Paris-Sud) In the belief propagation, the marginal probability Pr{F 2 |G=g} is approximately expressed in terms of the messages from the neighbouring region of the pixel 2.

35 Marginal Probability in Belief Propagation October, 2010LRI Seminar 2010 (Univ. Paris-Sud) In order to compute the marginal probability Pr{F 1,F 2 |G=g}, we take summations over all the pixels except the pixels 1 and2.

36 Marginal Probability in Belief Propagation October, 2010LRI Seminar 2010 (Univ. Paris-Sud) In the belief propagation, the marginal probability Pr{F 1,F 2 |G=g} is approximately expressed in terms of the messages from the neighbouring region of the pixels 1 and 2.

Belief Propagation in Probabilistic Image Processing 14 October, 2010LRI Seminar 2010 (Univ. Paris-Sud) 37

Belief Propagation in Probabilistic Image Processing 14 October, 2010LRI Seminar 2010 (Univ. Paris-Sud) 38

14 October, 2010LRI Seminar 2010 (Univ. Paris-Sud) 39 Image Restorations by Gaussian Markov Random Fields and Conventional Filters MSE Gauss MRF (Exact)315 Gauss MRF (Belief Propagation) 327 Lowpass Filter (3x3)388 (5x5)413 Median Filter (3x3)486 (5x5)445 Belief Propagation Exact Original ImageDegraded Image RestoredImage V: Set of all the pixels

40 Gray-Level Image Restoration (Spike Noise) Original Image MSE:135MSE: 217 MSE: 371MSE: 523 MSE: 244 MSE: 3469 MSE: 2075 Degraded Image Belief Propagation Lowpass FilterMedian Filter MSE: October, 2010LRI Seminar 2010 (Univ. Paris-Sud)

14 October, 2010LRI Seminar 2010 (Univ. Paris-Sud) 41 Binary Image Restoration by Loopy Belief Propagation   

42 Statistical Performance by Sample Average of Numerical Experiments Posterior Probability Pr{F|G=g, ,  } Sample Average of Mean Square Error Original Images Noise Pr{G|F=f,  } Estimated Results Observed Data 14 October, LRI Seminar 2010 (Univ. Paris-Sud)

14 October, 2010LRI Seminar 2010 (Univ. Paris-Sud) 43 Statistical Performance Estimation for Binary Markov Random Fields It can be reduced to the calculation of the average of free energy with respect to locally non-uniform external fields g 1, g 2,…,g |V|. Free Energy of Ising Model with Random External Fields Light intensities of the original image can be regarded as spin states of ferromagnetic system. ==> ==>

Statistical Performance Estimation for Binary Markov Random Fields 14 October, 2010LRI Seminar 2010 (Univ. Paris-Sud) 44

14 October, 2010LRI Seminar 2010 (Univ. Paris-Sud) 45 Statistical Performance Estimation for Markov Random Fields   =40  =1  Spin Glass Theory in Statistical Mechanics Loopy Belief Propagation Multi-dimensional Gauss Integral Formulas

14 October, 2010LRI Seminar 2010 (Univ. Paris-Sud) 46 Statistical Performance Estimation for Markov Random Fields

Outline 1.Introduction 2.Bayesian Image Analysis by Gauss Markov Random Fields 3.Statistical Performance Analysis for Gauss Markov Random Fields 4.Statistical Performance Analysis in Binary Markov Random Fields by Loopy Belief Propagation 5.Concluding Remarks 14 October, 2010LRI Seminar 2010 (Univ. Paris-Sud)47

14 October, 2010LRI Seminar 2010 (Univ. Paris-Sud)48 Summary Formulation of probabilistic model for image processing by means of conventional statistical schemes has been summarized. Statistical performance analysis of probabilistic image processing by using Gauss Markov Random Fields has been shown. One of extensions of statistical performance estimation to probabilistic image processing with discrete states has been demonstrated.

Image Impainting by Gauss MRF and LBP 14 October, 2010LRI Seminar 2010 (Univ. Paris-Sud) 49 Gauss MRF and LBP Our framework can be extended to erase a scribbling.

14 October, 2010LRI Seminar 2010 (Univ. Paris-Sud)50 References 1.K. Tanaka and D. M. Titterington: Statistical Trajectory of Approximate EM Algorithm for Probabilistic Image Processing, Journal of Physics A: Mathematical and Theoretical, vol.40, no.37, pp , M. Yasuda and K. Tanaka: The Mathematical Structure of the Approximate Linear Response Relation, Journal of Physics A: Mathematical and Theoretical, vol.40, no.33, pp , K. Tanaka and K. Tsuda: A Quantum-Statistical-Mechanical Extension of Gaussian Mixture Model, Journal of Physics: Conference Series, vol.95, article no , pp.1-9, January K. Tanaka: Mathematical Structures of Loopy Belief Propagation and Cluster Variation Method, Journal of Physics: Conference Series, vol.143, article no , pp.1-18, M. Yasuda and K. Tanaka: Approximate Learning Algorithm in Boltzmann Machines, Neural Computation, vol.21, no.11, pp , S. Kataoka, M. Yasuda and K. Tanaka: Statistical Performance Analysis in Probabilistic Image Processing, Journal of the Physical Society of Japan, vol.79, no.2, article no , 2010.