1. Parameter Estimation
主講人：虞台文

2. Contents Introduction Maximum-Likelihood Estimation
Bayesian Estimation

3. Parameter Estimation
Introduction

4. Bayesian Rule
We want to estimate the parameters of class-conditional densities if its parametric form is known, e.g.,

5. Methods The Method of Moments Maximum-Likelihood Estimation
Not discussed in this course
Maximum-Likelihood Estimation
Assume parameters are fixed but unknown
Bayesian Estimation
Assume parameters are random variables
Sufficient Statistics

6. Maximum-Likelihood Estimation
Parameter Estimation
Maximum-Likelihood Estimation

7. Samples 1 2 D1 D2
The samples in Dj are drawn independently according to the probability law p(x|j). D3
Assume that p(x|j) has a known parametric form with parameter vector j. 3
e.g., j

8. Goal D1 D2 D3 1 2 Use Dj to estimate the unknown parameter vector j
The estimated version will be denoted by
Goal 1 2 D1 D2 D3
Use Dj to estimate the unknown parameter vector j 3

9. Problem Formulation D Now the problem is:
Because each class is consider individually, the subscript used before will be dropped.
Now the problem is: D
Given a sample set D, whose elements are drawn independently from a population possessing a known parameter form, say p(x|), we want to choose a that will make D to occur most likely.

10. Criterion of ML MLE By the independence assumption, we have
Likelihood function:
MLE

11. Criterion of ML
Often, we resort to maximize the log-likelihood function
How?
MLE

12. Criterion of ML
Example:
How?

13. Differential Approach if Possible
Find the extreme values using the method in differential calculus.
Let f() be a continuous function, where =(1, 2,…, n)T.
Gradient
Operator
Find the extreme values by solving

14. Preliminary
Let

15. Preliminary
Let
(xf )T

16. The Gaussian Population
Two cases:
Unknown 
Unknown  and 

17. The Gaussian Population: Unknown 

18. The Gaussian Population: Unknown 
Set
Sample Mean

19. The Gaussian Population: Unknown  and 
Consider univariate normal case

20. The Gaussian Population: Unknown  and 
Consider univariate normal case
unbiased
Set
biased

21. The Gaussian Population: Unknown  and 
For multivariate normal case
The MLE of  and  are:
unbiased
biased

22. Unbiasedness Unbiased Estimator Consistent Estimator
(Absolutely unbiase)
Consistent Estimator
(asymptotically unbiased)

23. MLE for Normal Population
Sample Mean
Sample Covariance Matrix

24. Parameter Estimation
Bayesian Estimation

25. Comparison MLE (Maximum-Likelihood Estimation) Bayesian Estimation
to find the fixed but unknown parameters of a population.
Bayesian Estimation
Consider the parameters of a population to be random variables.

26. Heart of Bayesian Classification
Ultimate Goal:
Evaluate
What can we do if prior probabilities and class-conditional densities are unknown?

27. Helpful Knowledge Functional form for unknown densities
e.g., Normal, exponential, …
Ranges for the values of unknown parameters
e.g., uniform distributed over a range
Training Samples
Sampling according to the states of nature.

28. Posterior Probabilities from Sample

29. Posterior Probabilities from Sample
Each class can be considered independently

30. Problem Formulation D This the central problem of Bayesian Learning.
Let D be a set of samples drawn independently according to the fixed but known distribution p(x).
We want to determine D
This the central problem of Bayesian Learning.

31. Parameter Distribution
Assume p(x) is unknown but knowing it has a fixed form with parameter vector .
is complete known
Assume  is a random vector, and p() is a known a priori.

32. Class-Conditional Density Estimation

33. Class-Conditional Density Estimation
The posterior density we want to estimate
The form of distribution is assumed known

34. Class-Conditional Density Estimation
If p(|D) has a sharp peak at

35. Class-Conditional Density Estimation

36. The Univariate Gaussian: Unknown 
distribution form is known
assume  is normal distributed

37. The Univariate Gaussian: Unknown 

38. The Univariate Gaussian: Unknown 
Comparison

39. The Univariate Gaussian: Unknown 

40. The Univariate Gaussian: Unknown 

41. The Univariate Gaussian: p(x|D)

42. The Univariate Gaussian: p(x|D)

43. The Univariate Gaussian: p(x|D)=?

44. The Multivariate Gaussian: Unknown 
distribution form is known
assume  is normal distributed

45. The Multivariate Gaussian: Unknown 

46. The Multivariate Gaussian: Unknown 

47. General Theory 1.
the form of class-conditional density is known. 2.
knowledge about the parameter distribution is available.
samples are randomly drawn according to the unknown probability density p(x). 3.

48. General Theory 1.
the form of class-conditional density is known. 2.
knowledge about the parameter distribution is available.
samples are randomly drawn according to the unknown probability density p(x). 3.

49. Incremental Learning
Recursive

50. 1.
Example 2. 3.

51. 1.
Example 2. 3. 4 2 4 6 8 10 
p(|Dn) 3 2 1