Graduate School of Information Sciences, Tohoku University Physical Fluctuomatics Applied Stochastic Process 4th Maximum likelihood estimation and EM algorithm Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University kazu@smapip.is.tohoku.ac.jp http://www.smapip.is.tohoku.ac.jp/~kazu/ Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

Physical Fuctuomatics (Tohoku University) Textbooks Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) , Chapter 4. Physical Fuctuomatics (Tohoku University)

Statistical Machine Learning and Model Selection Inference of Probabilistic Model by using Data Model Selection Statistical Machine Learning Maximum Likelihood Complete Data and Incomplete data EM algorithm Extension Implement by Belief Propagation and Markov Chain Monte Carlo method Akaike Information Criteria, Akaike Bayes Information Criteria, etc. Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

Maximum Likelihood Estimation Parameter Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

Maximum Likelihood Estimation Parameter 1 2 3 4 5 6 7 8 Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

Maximum Likelihood Estimation Data Parameter 1 2 3 4 5 6 7 8 Data Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

Maximum Likelihood Estimation Data Parameter 1 2 3 4 5 6 7 8 Data Histogram Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

Maximum Likelihood Estimation Data Parameter 1 2 3 4 5 6 7 8 Data Histogram Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

Maximum Likelihood Estimation Data Parameter 1 2 3 4 5 6 7 8 Data Histogram Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

Maximum Likelihood Estimation Data Parameter Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given. Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

Maximum Likelihood Estimation Data Parameter Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given. Extremum Condition Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

Maximum Likelihood Estimation Data Parameter Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given. Extremum Condition Sample Deviation Sample Average Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

Maximum Likelihood Estimation Data Parameter Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given. Extremum Condition Sample Deviation Sample Average Histogram Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

Maximum Likelihood is unknown Data Hyperparameter Marginal Likelihood Extremum Condition Parameter Bayes Formula Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

Probabilistic Model for Signal Processing Noise i fi i gi Transmission Source Signal Observable Data Bayes Formula Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

Prior Probability for Source Signal j E：Set of all the links One dimensional Data Image Data 1 2 3 4 5 1 2 3 4 6 7 8 9 21 22 23 24 5 10 25 11 12 13 14 16 17 18 19 15 20 = 1 2 X 2 3 X 3 4 X 4 5 Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

Data Generating Process Additive White Gaussian Noise V：Set of all the nodes Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

Probabilistic Model for Signal Processing Parameter データ i fi i gi Hyperparameter Posterior Probability Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

Maximum Likelihood in Signal Processing Incomplete Data Parameter Data Marginal Likelihood Hyperparameter Extremum Condition Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

Maximum Likelihood and EM algorithm Incomplete Data Parameter Data Marginal Likelihood Q function Hyperparameter Expectation Maximization (EM) algorithm provide us one of Extremum Points of Marginal Likelihood. Extemum Condition Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

Model Selection in One Dimensional Signal Expectation Maximization (EM) Algorithm 127 255 100 200 Original Signal Degraded Signal Estimated Signal 0.04 0.03 α(t) 0.02 0.01 α(0)=0.0001, σ(0)=100 Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

Model Selection in Noise Reduction MSE 327 0.000611 36.30 Estimate of Original Image Original Image Degraded Image EM algorithm and Belief Propagation α(0)=0.0001 σ(0)=100 MSE 260 0.000574 34.00 Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

Summary Maximum Likelihood and EM algorithm Statistical Inference by Gaussian Graphical Model Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

Practice 4-1 ,　 , ,　 Let us suppose that data {gi |i=0,1,...,N-1} are generated by according to the following probability density function: , where Prove that estimates for average m and variance s2 of the maximum likelihood , are given as Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)