1. Graduate School of Information Sciences, Tohoku University
Physical Fluctuomatics Applied Stochastic Process 4th Maximum likelihood estimation and EM algorithm
Kazuyuki Tanaka
Graduate School of Information Sciences, Tohoku University
Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

2. Physical Fuctuomatics (Tohoku University)
Textbooks
Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) , Chapter 4.
Physical Fuctuomatics (Tohoku University)

3. Statistical Machine Learning and Model Selection
Inference of Probabilistic Model by using Data
Model Selection
Statistical Machine Learning
Maximum Likelihood
Complete Data and Incomplete data
EM algorithm
Extension
Implement by Belief Propagation and Markov Chain Monte Carlo method
Akaike Information Criteria, Akaike Bayes Information Criteria, etc.
Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

4. Maximum Likelihood Estimation
Parameter
Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

5. Maximum Likelihood Estimation
Parameter 1 2 3 4 5 6 7 8
Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

6. Maximum Likelihood Estimation
Data
Parameter 1 2 3 4 5 6 7 8
Data
Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

7. Maximum Likelihood Estimation
Data
Parameter 1 2 3 4 5 6 7 8
Data
Histogram
Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

8. Maximum Likelihood Estimation
Data
Parameter 1 2 3 4 5 6 7 8
Data
Histogram
Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

9. Maximum Likelihood Estimation
Data
Parameter 1 2 3 4 5 6 7 8
Data
Histogram
Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

10. Maximum Likelihood Estimation
Data
Parameter
Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given.
Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

11. Maximum Likelihood Estimation
Data
Parameter
Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given.
Extremum Condition
Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

12. Maximum Likelihood Estimation
Data
Parameter
Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given.
Extremum Condition
Sample Deviation
Sample Average
Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

13. Maximum Likelihood Estimation
Data
Parameter
Probability density function for data with average μ and variance σ2 is regarded as likelihood function for average μ and variance σ2 when data is given.
Extremum Condition
Sample Deviation
Sample Average
Histogram
Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

14. Maximum Likelihood is unknown Data Hyperparameter Marginal Likelihood
Extremum Condition
Parameter
Bayes Formula
Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

15. Probabilistic Model for Signal Processing
Noise i fi i gi
Transmission
Source Signal
Observable Data
Bayes Formula
Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

16. Prior Probability for Source Signalj
E：Set of all the links
One dimensional Data
Image Data 1 2 3 4 5 1 2 3 4 6 7 8 9 21 22 23 24 5 10 25 11 12 13 14 16 17 18 19 15 20 = 1 2 X 2 3 X 3 4 X 4 5
Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

17. Data Generating Process
Additive White Gaussian Noise
V：Set of all the nodes
Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

18. Probabilistic Model for Signal Processing
Parameter
データ i fi i gi
Hyperparameter
Posterior Probability
Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

19. Maximum Likelihood in Signal Processing
Incomplete Data
Parameter
Data
Marginal Likelihood
Hyperparameter
Extremum Condition
Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

20. Maximum Likelihood and EM algorithm
Incomplete Data
Parameter
Data
Marginal Likelihood
Q function
Hyperparameter
Expectation Maximization (EM) algorithm provide us one of Extremum Points of Marginal Likelihood.
Extemum Condition
Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

21. Model Selection in One Dimensional Signal
Expectation Maximization (EM) Algorithm
127
255
100
200
Original Signal
Degraded Signal
Estimated Signal
0.04
0.03
α(t)
0.02
0.01
α(0)=0.0001, σ(0)=100
Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

22. Model Selection in Noise Reduction
MSE
327
36.30
Estimate of Original Image
Original Image
Degraded Image
EM algorithm and Belief Propagation
α(0)=0.0001
σ(0)=100
MSE
260
34.00
Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

23. Summary Maximum Likelihood and EM algorithm
Statistical Inference by Gaussian Graphical Model
Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)

24. Practice 4-1 ,　 , ,　
Let us suppose that data {gi |i=0,1,...,N-1} are generated by according to the following probability density function: ,
where
Prove that estimates for average m and variance s2 of the maximum likelihood ,
are given as
Phisical Fluctuomatics / Applied Stochastic Process (Tohoku University)