Presentation is loading. Please wait.

Presentation is loading. Please wait.

Graduate School of Information Sciences, Tohoku University, Japan

Published by进 黄 Modified over 5 years ago

Similar presentations

Presentation on theme: "Graduate School of Information Sciences, Tohoku University, Japan"— Presentation transcript:

1 Graduate School of Information Sciences, Tohoku University, Japan
Generalized Belief Propagation for Gaussian Graphical Model in probabilistic image processing Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University, Japan Reference K. Tanaka, H. Shouno, M. Okada and D. M. Titterington: Accuracy of the Bethe Approximation for Hyperparameter Estimation in Probabilistic Image Processing, J. Phys. A: Math & Gen., 37, 8675 (2004). 6 September, 2005 SPDSA2005 (Roma)

2 Contents Introduction Loopy Belief Propagation
Generalized Belief Propagation Probabilistic Image Processing Concluding Remarks 6 September, 2005 SPDSA2005 (Roma)

3 Bayesian Image Analysis
Noise Transmission Original Image Degraded Image Graphical Model with Loops =Spin System on Square Lattice Bayesian Image Analysis + Belief Propagation → Probabilistic Image Processing 6 September, 2005 SPDSA2005 (Roma)

4 Belief Propagation Belief Propagation Generalized Belief Propagation
Probabilistic model with no loop Belief Propagation = Transfer Matrix (Lauritzen, Pearl) Probabilistic model with some loops Approximation→Loopy Belief Propagation Generalized Belief Propagation (Yedidia, Freeman, Weiss) Loopy Belief Propagation (LBP) = Bethe Approximation Generalized Belief Propagation (GBP) = Cluster Variation Method How is the accuracy of LBP and GBP? 6 September, 2005 SPDSA2005 (Roma)

5 Gaussian Graphical Model
6 September, 2005 SPDSA2005 (Roma)

6 Probabilistic Image Processing by Gaussian Graphical Model and Generalized Belief Propagation
How can we construct a probabilistic image processing algorithm by using Loopy Belief Propagation and Generalized Belief Propagation? How is the accuracy of Loopy Belief Propagation and Generalized Belief Propagation? In order to clarify both questions, we assume the Gaussian graphical model as a posterior probabilistic model 6 September, 2005 SPDSA2005 (Roma)

7 Contents Introduction Loopy Belief Propagation
Generalized Belief Propagation Probabilistic Image Processing Concluding Remarks 6 September, 2005 SPDSA2005 (Roma)

8 Kullback-Leibler Divergence of Gaussian Graphical Model
Entropy Term 6 September, 2005 SPDSA2005 (Roma)

9 Loopy Belief Propagation
Trial Function Tractable Form 6 September, 2005 SPDSA2005 (Roma)

10 Loopy Belief Propagation
Trial Function Marginal Distribution of GGM is also GGM 6 September, 2005 SPDSA2005 (Roma)

11 Loopy Belief Propagation
Bethe Free Energy in GGM 6 September, 2005 SPDSA2005 (Roma)

12 Loopy Belief Propagation
m is exact 6 September, 2005 SPDSA2005 (Roma)

13 Iteration Procedure Fixed Point Equation Iteration 6 September, 2005
SPDSA2005 (Roma)

14 Loopy Belief Propagation and TAP Free Energy
Mean Field Free Energy 6 September, 2005 SPDSA2005 (Roma)

15 Contents Introduction Loopy Belief Propagation
Generalized Belief Propagation Probabilistic Image Processing Concluding Remarks 6 September, 2005 SPDSA2005 (Roma)

16 Generalized Belief Propagation
Cluster: Set of nodes Every subcluster of the element of B does not belong to B. Example: System consisting of 4 nodes 1 2 3 4 1 2 3 4 6 September, 2005 SPDSA2005 (Roma)

17 Selection of B in LBP and GBP
(Bethe Approx.） 1 2 4 5 3 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 4 5 3 6 7 8 9 GBP (Square Approx. in CVM) 6 September, 2005 SPDSA2005 (Roma)

18 Selection of B and C in Loopy Belief Propagation
LBP (Bethe Approx.） The set of Basic Clusters The Set of Basic Clusters and Their Subclusters 6 September, 2005 SPDSA2005 (Roma)

19 Selection of B and C in Generalized Belief Propagation
GBP (Square Approximation in CVM） The set of Basic Clusters The Set of Basic Clusters and Their Subclusters 6 September, 2005 SPDSA2005 (Roma)

20 Generalized Belief Propagation
Trial Function Marginal Distribution of GGM is also GGM 6 September, 2005 SPDSA2005 (Roma)

21 Generalized Belief Propagation
m is exact 6 September, 2005 SPDSA2005 (Roma)

22 Contents Introduction Loopy Belief Propagation
Generalized Belief Propagation Probabilistic Image Processing Concluding Remarks 6 September, 2005 SPDSA2005 (Roma)

23 Bayesian Image Analysis
Noise Transmission Original Image Degraded Image 6 September, 2005 SPDSA2005 (Roma)

24 Bayesian Image Analysis
Degradation Process Additive White Gaussian Noise Transmission Original Image Degraded Image 6 September, 2005 SPDSA2005 (Roma)

25 Bayesian Image Analysis
A Priori Probability Generate Standard Images Similar? 6 September, 2005 SPDSA2005 (Roma)

26 Bayesian Image Analysis
Original Image f Degraded Image g A Posteriori Probability Gaussian Graphical Model 6 September, 2005 SPDSA2005 (Roma)

27 Bayesian Image Analysis
A Priori Probability Degraded Image Degraded Image Original Image Pixels A Posteriori Probability 6 September, 2005 SPDSA2005 (Roma)

28 Hyperparameter Determination by Maximization of Marginal Likelihood
In the image restoration, we usually have to estimate the hyperparameters alpha and p. In statistics, the maximum likelihood estimation is often employed. In the standpoint of maximum likelihood estimation, the hyperparameters are determined so as to maximize the marginal likelihood defined by marginalize the joint probability for the original image and degraded image with respect to the original image. The marginal likelihood is expressed in terms of the partition functions of the a priori probabilistic model and the a posteriori probabilistic model. We can calculate these partition functions approximately by using the Bethe approximation. Marginalization Degraded Image Original Image Marginal Likelihood 6 September, 2005 SPDSA2005 (Roma)

29 Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm
Q-Function In the image restoration, we usually have to estimate the hyperparameters alpha and p. In statistics, the maximum likelihood estimation is often employed. In the standpoint of maximum likelihood estimation, the hyperparameters are determined so as to maximize the marginal likelihood defined by marginalize the joint probability for the original image and degraded image with respect to the original image. The marginal likelihood is expressed in terms of the partition functions of the a priori probabilistic model and the a posteriori probabilistic model. We can calculate these partition functions approximately by using the Bethe approximation. Incomplete Data Equivalent 6 September, 2005 SPDSA2005 (Roma)

30 Iterate the following EM-steps until convergence:
Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm Marginal Likelihood Q-Function In the image restoration, we usually have to estimate the hyperparameters alpha and p. In statistics, the maximum likelihood estimation is often employed. In the standpoint of maximum likelihood estimation, the hyperparameters are determined so as to maximize the marginal likelihood defined by marginalize the joint probability for the original image and degraded image with respect to the original image. The marginal likelihood is expressed in terms of the partition functions of the a priori probabilistic model and the a posteriori probabilistic model. We can calculate these partition functions approximately by using the Bethe approximation. Iterate the following EM-steps until convergence: EM Algorithm A. P. Dempster, N. M. Laird and D. B. Rubin, “Maximum likelihood from incomplete data via the EM algorithm,” J. Roy. Stat. Soc. B, 39 (1977). 6 September, 2005 SPDSA2005 (Roma)

31 Image Restoration The original image is generated from the prior probability. (Hyperparameters: Maximization of Marginal Likelihood) Degraded Image Original Image Loopy Belief Propagation Mean-Field Method Exact Result 6 September, 2005 SPDSA2005 (Roma)

32 Numerical Experiments of Logarithm of Marginal Likelihood
The original image is generated from the prior probability. (Hyperparameters: Maximization of Marginal Likelihood) Original Image Degraded Image MFA -5.0 -5.0 MFA LPB -5.5 Exact LPB Exact -6.0 -5.5 10 20 30 40 50 60 0.0010 0.0020 Mean-Field Method Loopy Belief Propagation Exact Result 6 September, 2005 SPDSA2005 (Roma)

33 Numerical Experiments of Logarithm of Marginal Likelihood
EM Algorithm with Belief Propagation Original Image MF Exact LBP 0.001 50 100 LPB MFA 0.002 Degraded Image 6 September, 2005 SPDSA2005 (Roma)

34 Image Restoration by Gaussian Graphical Model
EM Algorithm with Belief Propagation Original Image Degraded Image MSE: 1512 MSE: 1529 6 September, 2005 SPDSA2005 (Roma)

35 Image Restoration by Gaussian Graphical Model
Original Image Degraded Image Mean Field Method MSE: 1512 MSE:611 Finally, we show only the results for the gray-level image restoration. For each numerical experiments, the loopy belief propagation ca give us better results than the ones by conventional filters. LBP TAP GBP Exact Solution MSE:327 MSE:320 MSE: 315 MSE:315 6 September, 2005 SPDSA2005 (Roma)

36 Image Restoration by Gaussian Graphical Model
Original Image Degraded Image Mean Field Method Finally, we show only the results for the gray-level image restoration. For each numerical experiments, the loopy belief propagation ca give us better results than the ones by conventional filters. MSE: 1529 MSE: 565 LBP TAP GBP Exact Solution MSE:260 MSE:248 MSE:236 MSE:236 6 September, 2005 SPDSA2005 (Roma)

37 Image Restoration by Gaussian Graphical Model
MSE MF 611 26.918 LBP 327 36.302 TAP 320 37.170 GBP 315 37.909 Exact 37.919 Finally, we show only the results for the gray-level image restoration. For each numerical experiments, the loopy belief propagation ca give us better results than the ones by conventional filters. MSE MF 565 26.353 LBP 260 33.998 TAP 248 34.475 GBP 236 34.971 Exact 34.975 6 September, 2005 SPDSA2005 (Roma)

38 Image Restoration by Gaussian Graphical Model and Conventional Filters
MSE MF 611 Lowpass Filter (3x3) 388 LBP 327 (5x5) 413 TAP 320 Median Filter 486 GBP 315 445 Exact Wiener Filter 864 548 GBP Finally, we show only the results for the gray-level image restoration. For each numerical experiments, the loopy belief propagation ca give us better results than the ones by conventional filters. (3x3) Lowpass (5x5) Median (5x5) Wiener 6 September, 2005 SPDSA2005 (Roma)

39 Image Restoration by Gaussian Graphical Model and Conventional Filters
MSE MF 565 Lowpass Filter (3x3) 241 LBP 260 (5x5) 224 TAP 248 Median Filter 331 GBP 236 244 Exact Wiener Filter 703 372 GBP Finally, we show only the results for the gray-level image restoration. For each numerical experiments, the loopy belief propagation ca give us better results than the ones by conventional filters. (5x5) Lowpass (5x5) Median (5x5) Wiener 6 September, 2005 SPDSA2005 (Roma)

40 Contents Introduction Loopy Belief Propagation
Generalized Belief Propagation Probabilistic Image Processing Concluding Remarks 6 September, 2005 SPDSA2005 (Roma)

41 Summary Generalized Belief Propagation for Gaussian Graphical Model
Accuracy of Generalized Belief Propagation Derivation of TAP Free Energy for Gaussian Graphical Model by Perturbation Expansion of Bethe Approximation 6 September, 2005 SPDSA2005 (Roma)

42 Future Problem Hyperparameter Estimation by TAP Free Energy is better than by Loopy Belief Propagation. Effectiveness of Higher Order Terms of TAP Free Energy for Hyperparameter Estimation by means of Marginal Likelihood in Bayesian Image Analysis. TAP Free Energy Mean Field Free Energy 6 September, 2005 SPDSA2005 (Roma)

Download ppt "Graduate School of Information Sciences, Tohoku University, Japan"

Similar presentations

Part 2: Unsupervised Learning

Part 2: Unsupervised Learning
Discrete Optimization Lecture 4 – Part 3 M. Pawan Kumar Slides available online

Discrete Optimization Lecture 4 – Part 3 M. Pawan Kumar Slides available online
Variational Methods for Graphical Models Micheal I. Jordan Zoubin Ghahramani Tommi S. Jaakkola Lawrence K. Saul Presented by: Afsaneh Shirazi.

Variational Methods for Graphical Models Micheal I. Jordan Zoubin Ghahramani Tommi S. Jaakkola Lawrence K. Saul Presented by: Afsaneh Shirazi.
Loopy Belief Propagation a summary. What is inference? Given: –Observabled variables Y –Hidden variables X –Some model of P(X,Y) We want to make some.

Loopy Belief Propagation a summary. What is inference? Given: –Observabled variables Y –Hidden variables X –Some model of P(X,Y) We want to make some.
Introduction to Belief Propagation and its Generalizations. Max Welling Donald Bren School of Information and Computer and Science University of California.

Introduction to Belief Propagation and its Generalizations. Max Welling Donald Bren School of Information and Computer and Science University of California.
3 March, 2003University of Glasgow1 Statistical-Mechanical Approach to Probabilistic Inference --- Cluster Variation Method and Generalized Loopy Belief.

3 March, 2003University of Glasgow1 Statistical-Mechanical Approach to Probabilistic Inference --- Cluster Variation Method and Generalized Loopy Belief.
Belief Propagation by Jakob Metzler. Outline Motivation Pearl’s BP Algorithm Turbo Codes Generalized Belief Propagation Free Energies.

Belief Propagation by Jakob Metzler. Outline Motivation Pearl’s BP Algorithm Turbo Codes Generalized Belief Propagation Free Energies.
1 Bayesian Image Modeling by Generalized Sparse Markov Random Fields and Loopy Belief Propagation Kazuyuki Tanaka GSIS, Tohoku University, Sendai, Japan.

1 Bayesian Image Modeling by Generalized Sparse Markov Random Fields and Loopy Belief Propagation Kazuyuki Tanaka GSIS, Tohoku University, Sendai, Japan.
Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael.

Learning Low-Level Vision William T. Freeman Egon C. Pasztor Owen T. Carmichael.
Understanding Belief Propagation and its Applications Dan Yuan June 2004.

Understanding Belief Propagation and its Applications Dan Yuan June 2004.
Super-Resolution of Remotely-Sensed Images Using a Learning-Based Approach Isabelle Bégin and Frank P. Ferrie Abstract Super-resolution addresses the problem.

Super-Resolution of Remotely-Sensed Images Using a Learning-Based Approach Isabelle Bégin and Frank P. Ferrie Abstract Super-resolution addresses the problem.
24 November, 2011National Tsin Hua University, Taiwan1 Mathematical Structures of Belief Propagation Algorithms in Probabilistic Information Processing.

24 November, 2011National Tsin Hua University, Taiwan1 Mathematical Structures of Belief Propagation Algorithms in Probabilistic Information Processing.
Some Surprises in the Theory of Generalized Belief Propagation Jonathan Yedidia Mitsubishi Electric Research Labs (MERL) Collaborators: Bill Freeman (MIT)

Some Surprises in the Theory of Generalized Belief Propagation Jonathan Yedidia Mitsubishi Electric Research Labs (MERL) Collaborators: Bill Freeman (MIT)
1 物理フラクチュオマティクス論 Physical Fluctuomatics 応用確率過程論 Applied Stochastic Process 第 5 回グラフィカルモデルによる確率的情報処理 5th Probabilistic information processing by means of.

1 物理フラクチュオマティクス論 Physical Fluctuomatics 応用確率過程論 Applied Stochastic Process 第 5 回グラフィカルモデルによる確率的情報処理 5th Probabilistic information processing by means of.
Physics Fluctuomatics (Tohoku University) 1 Physical Fluctuomatics 7th~10th Belief propagation Appendix Kazuyuki Tanaka Graduate School of Information.

Physics Fluctuomatics (Tohoku University) 1 Physical Fluctuomatics 7th~10th Belief propagation Appendix Kazuyuki Tanaka Graduate School of Information.
1 October, 2007 ALT&DS2007 (Sendai, Japan ） 1 Introduction to Probabilistic Image Processing and Bayesian Networks Kazuyuki Tanaka Graduate School of Information.

1 October, 2007 ALT&DS2007 (Sendai, Japan ） 1 Introduction to Probabilistic Image Processing and Bayesian Networks Kazuyuki Tanaka Graduate School of Information.
1 Physical Fluctuomatics 5th and 6th Probabilistic information processing by Gaussian graphical model Kazuyuki Tanaka Graduate School of Information Sciences,

1 Physical Fluctuomatics 5th and 6th Probabilistic information processing by Gaussian graphical model Kazuyuki Tanaka Graduate School of Information Sciences,
3 September, 2009 SSP2009, Cardiff, UK 1 Probabilistic Image Processing by Extended Gauss-Markov Random Fields Kazuyuki Tanaka Kazuyuki Tanaka, Muneki.

3 September, 2009 SSP2009, Cardiff, UK 1 Probabilistic Image Processing by Extended Gauss-Markov Random Fields Kazuyuki Tanaka Kazuyuki Tanaka, Muneki.
Image Restoration using Iterative Wiener Filter --- ECE533 Project Report Jing Liu, Yan Wu.

Image Restoration using Iterative Wiener Filter --- ECE533 Project Report Jing Liu, Yan Wu.
Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 1 Physical Fluctuomatics Applied Stochastic Process 9th Belief propagation Kazuyuki.

Physics Fluctuomatics / Applied Stochastic Process (Tohoku University) 1 Physical Fluctuomatics Applied Stochastic Process 9th Belief propagation Kazuyuki.

Similar presentations

Ads by Google