1. 29 December, 2008 National Tsing Hua University, Taiwan 1 Introduction to Probabilistic Image Processing and Bayesian Networks Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University, Sendai, Japan http://www.smapip.is.tohoku.ac.jp/~kazu/ Collaborators Prof. Mike Titterington (University of Glasgow, UK) Dr. Koji Tsuda (MPI for Biological Cybernetics, Germany) Dr. Muneki Yasuda (Tohoku University, Japan)

2. 29 December, 2008National Tsing Hua University, Taiwan2 Contents 1.Introduction 2.Probabilistic Image Processing 3.Gaussian Graphical Model 4.Belief Propagation 5.Other Application 6.Concluding Remarks

3. 29 December, 2008National Tsing Hua University, Taiwan3 Contents 1.Introduction 2.Probabilistic Image Processing 3.Gaussian Graphical Model 4.Belief Propagation 5.Other Application 6.Concluding Remarks

4. 29 December, 2008National Tsing Hua University, Taiwan4 More is different Probabilistic information processing can give us unexpected capacity in a system constructed from many cooperating elements with randomness. The circumstances are similar to everything in the natural world being made up of a lot of molecules with interactions and fluctuations. The collection of molecules can give rise to unpredictable phenomena. This is often called “More is different” in physics.

5. 29 December, 2008National Tsing Hua University, Taiwan5 Main Interests Information Processing: DataPhysics:Material, Natural Phenomena System of a lot of elements with mutual relation Common Concept between Computer Sciences and Physics Material Molecule Materials are constructed from a lot of molecules. Molecules have interactions of each other. ０,1 101101 110001 01001110111010 10001111100001 10000101000000 11101010111010 1010 Bit Data Data is constructed from many bits A sequence is formed by deciding the arrangement of bits. A lot of elements have mutual relation of each other Some physical concepts in Physical models are useful for the design of computational models in probabilistic information processing.

6. 29 December, 2008National Tsing Hua University, Taiwan6 More is different in informatics as well. Our goal is to establish theoretical paradigms for probabilistic information processing by means of statistical science and statistical physics. The probabilistic information processing is based on both modeling of problems and design of algorithms, which is often realized as graphical models including Bayesian network.

7. 29 December, 2008National Tsing Hua University, Taiwan7 Purpose of My Talk Review of formulation of probabilistic model for image processing by means of conventional statistical schemes. Review of probabilistic image processing by using Gaussian graphical model (Gauss Markov Random Fields) as the most basic example. Review of how to construct a belief propagation algorithm for image processing.

8. 29 December, 2008National Tsing Hua University, Taiwan8 Contents 1.Introduction 2.Probabilistic Image Processing 3.Gaussian Graphical Model 4.Belief Propagation 5.Other Application 6.Concluding Remarks

9. 29 December, 2008National Tsing Hua University, Taiwan9 Bayes Formula and Bayesian Network Posterior Probability Bayes Rule Prior Probability Event B is given as the observed data. Event A corresponds to the original information to estimate. Thus the Bayes formula can be applied to the estimation of the original information from the given data. A B Bayesian Network Data-Generating Process

10. 29 December, 2008 National Tsing Hua University, Taiwan 10 Image Restoration by Probabilistic Model Original Image Degraded Image Transmission Noise Assumption 1: The degraded image is randomly generated from the original image by according to the degradation process. Assumption 2: The original image is randomly generated by according to the prior probability. Bayes Formula

11. 29 December, 2008 National Tsing Hua University, Taiwan 11 Image Restoration by Probabilistic Model Degraded Image i f i : Light Intensity of Pixel i in Original Image Position Vector of Pixel i g i : Light Intensity of Pixel i in Degraded Image i Original Image The original images and degraded images are represented by f = (f 1,f 2,…,f |V| ) T and g = (g 1,g 2,…,g |V| ) T, respectively.

12. 29 December, 2008 National Tsing Hua University, Taiwan 12 Probabilistic Modeling of Image Restoration Random Fields fifi gigi fifi gigi or Assumption 1: A given degraded image is obtained from the original image by changing the state of each pixel to another state by the same probability, independently of the other pixels.

13. 29 December, 2008 National Tsing Hua University, Taiwan 13 Probabilistic Modeling of Image Restoration Random Fields Assumption 2: The original image is generated according to a prior probability. Prior Probability consists of a product of functions defined on the neighbouring pixels. ij Product over All the Nearest Neighbour Pairs of Pixels

14. 29 December, 2008 National Tsing Hua University, Taiwan 14 Prior Probability for Binary Image = = > pp i j Probability of Neigbouring Pixel ij It is important how we should assume the function  (f i,f j ) in the prior probability. We assume that every nearest-neighbour pair of pixels take the same state of each other in the prior probability.

15. 29 December, 2008 National Tsing Hua University, Taiwan 15 Prior Probability for Binary Image Prior probability prefers to the configuration with the least number of red lines. Which state should the center pixel be taken when the states of neighbouring pixels are fixed to the white states? ？ > = = > pp i j Probability of Nearest Neigbour Pair of Pixels

16. 29 December, 2008National Tsing Hua University, Taiwan16 Prior Probability for Binary Image Which state should the center pixel be taken when the states of neighbouring pixels are fixed as this figure? ？-？？-？ = = > pp > > = Prior probability prefers to the configuration with the least number of red lines.

17. 29 December, 2008National Tsing Hua University, Taiwan17 What happens for the case of large number of pixels? p lnp Disordered State Critical Point (Large fluctuation) small plarge p Covariance between the nearest neghbour pairs of pixels Sampling by Marko chain Monte Carlo Ordered State Patterns with both ordered states and disordered states are often generated near the critical point.

18. 29 December, 2008National Tsing Hua University, Taiwan18 Physical model of ferromagnetism and Probabilistic model of image processing pp pp > = = Ising Model Markov Random Field (MRF) Model > = = Up Spin State Down Spin State Black State White State Probabilistic models for image processing has the similar structure as physical models of ferromagnetism Regular graph x y

19. 29 December, 2008National Tsing Hua University, Taiwan19 Pattern near Critical Point of Prior Probability ln p similar small plarge p Covariance between the nearest neghbour pairs of pixels We regard that patterns generated near the critical point are similar to the local patterns in real world images.

20. 29 December, 2008National Tsing Hua University, Taiwan20 Bayesian Image Analysis Original Image Degraded Image Prior Probability Bayes Formulas => Posterior Probability Degradation Process Image processing is reduced to calculations of averages, variances and co-variances in the posterior probability. E ： Set of all the nearest neighbour pairs of pixels V ： Set of All the pixels

21. 29 December, 2008National Tsing Hua University, Taiwan21 Contents 1.Introduction 2.Probabilistic Image Processing 3.Gaussian Graphical Model 4.Belief Propagation 5.Other Application 6.Concluding Remarks

22. 29 December, 2008 National Tsing Hua University, Taiwan 22 Bayesian Image Analysis by Gaussian Graphical Model Patterns are generated by MCMC. Markov Chain Monte Carlo Method Prior Probability E:Set of all the nearest-neghbour pairs of pixels V:Set of all the pixels

23. 29 December, 2008National Tsing Hua University, Taiwan23 Bayesian Image Analysis by Gaussian Graphical Model Histogram of Gaussian Random Numbers Degraded image is obtained by adding a white Gaussian noise to the original image. Degradation Process is assumed to be the additive white Gaussian noise. V:Set of all the pixels

24. 29 December, 2008 National Tsing Hua University, Taiwan 24 Bayesian Image Analysis by Gaussian Graphical Model Multi-Dimensional Gaussian Integral Formula Posterior Probability Average of the posterior probability can be calculated by using the multi- dimensional Gauss integral Formula |V|x|V| matrix E:Set of all the nearest-neghbour pairs of pixels V:Set of all the pixels

25. 29 December, 2008 National Tsing Hua University, Taiwan 25 Bayesian Image Analysis by Gaussian Graphical Model Iteration Procedure of EM algorithm in Gaussian Graphical Model EM

26. 29 December, 2008National Tsing Hua University, Taiwan26 Image Restoration by Gaussian Graphical Model and Conventional Filters MSE Statistical Method 315 Lowpass Filter (3x3)388 (5x5)413 Median Filter (3x3)486 (5x5)445 (3x3) Lowpass (5x5) Median MRF Original Image Degraded Image RestoredImage V:Set of all the pixels

27. 29 December, 2008National Tsing Hua University, Taiwan27 Contents 1.Introduction 2.Probabilistic Image Processing 3.Gaussian Graphical Model 4.Belief Propagation 5.Other Application 6.Concluding Remarks

28. 29 December, 2008National Tsing Hua University, Taiwan28 Graphical Representation for Tractable Models Tractable Model A B C D Intractable Model A B C Tree Graph Cycle Graph It is possible to calculate each summation independently. It is hard to calculate each summation independently.

29. 29 December, 2008National Tsing Hua University, Taiwan29 Loopy Belief Propagation for Graphical Model in Image Processing Graphical model for image processing is represented in terms of the square lattice. Square lattice includes a lot of cycles. Belief propagation are applied to the calculation of statistical quantities as an approximate algorithm. Every graph consisting of a pixel and its four neighbouring pixels can be regarded as a tree graph. Loopy Belief Propagation 124 5 3 124 5 3 3 1 2 5 4 2 1

30. 29 December, 2008National Tsing Hua University, Taiwan30 Loopy Belief Propagation in Image Processing We have four kinds of message passing rules for each pixel. Each massage passing rule includes 3 incoming messages and 1 outgoing message Visualizations of Passing Messages

31. 29 December, 2008National Tsing Hua University, Taiwan31 EM algorithm by means of Belief Propagation Input Output Loopy BP EM Update Rule of Loopy Belief Propagation EM Algorithm for Hyperparameter Estimation 3 1 2 5 4 2 1

32. 29 December, 2008 National Tsing Hua University, Taiwan 32 Probabilistic Image Processing by EM Algorithm and Loopy BP for Gaussian Graphical Model Loopy Belief Propagation Exact MSE:327 MSE:315

33. 29 December, 2008National Tsing Hua University, Taiwan33 Contents 1.Introduction 2.Probabilistic Image Processing 3.Gaussian Graphical Model 4.Belief Propagation 5.Other Application 6.Concluding Remarks

34. 29 December, 2008National Tsing Hua University, Taiwan34 Digital Images Inpainting based on MRF Input Output Markov Random Fields M. Yasuda, J. Ohkubo and K. Tanaka: Proceedings of CIMCA&IAWTIC2005.

35. 29 December, 2008National Tsing Hua University, Taiwan35 Contents 1.Introduction 2.Probabilistic Image Processing 3.Gaussian Graphical Model 4.Belief Propagation 5.Other Application 6.Concluding Remarks

36. 29 December, 2008National Tsing Hua University, Taiwan36 Summary Formulation of probabilistic model for image processing by means of conventional statistical schemes has been summarized. Probabilistic image processing by using Gaussian graphical model has been shown as the most basic example. It has been explained how to construct a belief propagation algorithm for image processing.

37. 29 December, 2008National Tsing Hua University, Taiwan37 References 1.K. Tanaka: “Statistical-Mechanical Approach to Image Processing” (Topical Review), Journal of Physics A: Mathematical and General, vol.35, no.37, pp.R81- R150, 2002. 2.K. Tanaka: “Probabilistic Inference by Means of Cluster Variation Method and Linear Response Theory,” IEICE Transactions on Information and Systems, vol.E86-D, no.7, pp.1228-1242, 2003. 3.K. Tanaka, H. Shouno, M. Okada and D. M. Titterington: “Accuracy of the Bethe Approximation for Hyperparameter Estimation in Probabilistic Image Processing,” Journal of Physics A: Mathematical and General, vol.37, no.36, pp.8675-8696, 2004. 4.J. Ohkubo, M. Yasuda and K. Tanaka: “Statistical-mechanical Iterative Algorithms on Complex Networks,” Physical Review E, vol.72, no.4, Article No.046135, 2005. 5.K. Tanaka and D. M. Titterington: “Statistical Trajectory of Approximate EM Algorithm for Probabilistic Image Processing,” Journal of Physics A: Mathematical and Theoretical, vol.40, no.37, pp.11285-11300, 2007. 6.M. Yasuda and K. Tanaka: “The Mathematical Structure of the Approximate Linear Response Relation,” Journal of Physics A: Mathematical and Theoretical, vol.40, no.33, pp.9993-10007, 2007.

38. 29 December, 2008National Tsing Hua University, Taiwan38 SMAPIP Project MEXT Grant-in Aid for Scientific Research on Priority Areas Period: 2002 –2005 Head Investigator: Kazuyuki Tanaka Period: 2002 –2005 Head Investigator: Kazuyuki Tanaka Member: K. Tanaka, Y. Kabashima, H. Nishimori, T. Tanaka, M. Okada, O. Watanabe, N. Murata,...... Statistical Mechanical Approach to Probabilistic Information Processing

39. 29 December, 2008National Tsing Hua University, Taiwan39 DEX-SMI Project http://dex-smi.sp.dis.titech.ac.jp/DEX-SMI/http://dex-smi.sp.dis.titech.ac.jp/DEX-SMI/ DEX-SMI GO MEXT Grant-in Aid for Scientific Research on Priority Areas Period: 2006 –2009 Head Investigator: Prof. Yoshiyuki Kabashima (Tokyo Institute of Technology) Period: 2006 –2009 Head Investigator: Prof. Yoshiyuki Kabashima (Tokyo Institute of Technology) Deepening and Expansion of Statistical Mechanical Informatics Member: Y. Kabashima, K. Tanaka, H. Nishimori, T. Tanaka, M. Okada, S. Ishii, M. Hayashi,…

40. 29 December, 2008National Tsing Hua University, Taiwan40 CERIES Project http://www.ecei.tohoku.ac.jp/gcoe/http://www.ecei.tohoku.ac.jp/gcoe/ CERIES GO GCOE Program Period: 2007 –2011 Head Investigator: Prof. Fumiyuki Adachi (Tohoku University) Period: 2007 –2011 Head Investigator: Prof. Fumiyuki Adachi (Tohoku University) Center of Education and Research for Information Electronics Systems