Parameter Estimation 主講人：虞台文

Contents Introduction Maximum-Likelihood Estimation Bayesian Estimation

Parameter Estimation Introduction

Bayesian Rule We want to estimate the parameters of class-conditional densities if its parametric form is known, e.g.,

Methods The Method of Moments Maximum-Likelihood Estimation Not discussed in this course Maximum-Likelihood Estimation Assume parameters are fixed but unknown Bayesian Estimation Assume parameters are random variables Sufficient Statistics

Maximum-Likelihood Estimation Parameter Estimation Maximum-Likelihood Estimation

Samples 1 2 D1 D2 The samples in Dj are drawn independently according to the probability law p(x|j). D3 Assume that p(x|j) has a known parametric form with parameter vector j. 3 e.g., j

Goal D1 D2 D3 1 2 Use Dj to estimate the unknown parameter vector j The estimated version will be denoted by Goal 1 2 D1 D2 D3 Use Dj to estimate the unknown parameter vector j 3

Problem Formulation D Now the problem is: Because each class is consider individually, the subscript used before will be dropped. Now the problem is: D Given a sample set D, whose elements are drawn independently from a population possessing a known parameter form, say p(x|), we want to choose a that will make D to occur most likely.

Criterion of ML MLE By the independence assumption, we have Likelihood function: MLE

Criterion of ML Often, we resort to maximize the log-likelihood function How? MLE

Criterion of ML Example: How?

Differential Approach if Possible Find the extreme values using the method in differential calculus. Let f() be a continuous function, where =(1, 2,…, n)T. Gradient Operator Find the extreme values by solving

Preliminary Let

Preliminary Let (xf )T

The Gaussian Population Two cases: Unknown  Unknown  and 

The Gaussian Population: Unknown 

The Gaussian Population: Unknown  Set Sample Mean

The Gaussian Population: Unknown  and  Consider univariate normal case

The Gaussian Population: Unknown  and  Consider univariate normal case unbiased Set biased

The Gaussian Population: Unknown  and  For multivariate normal case The MLE of  and  are: unbiased biased

Unbiasedness Unbiased Estimator Consistent Estimator (Absolutely unbiase) Consistent Estimator (asymptotically unbiased)

MLE for Normal Population Sample Mean Sample Covariance Matrix

Parameter Estimation Bayesian Estimation

Comparison MLE (Maximum-Likelihood Estimation) Bayesian Estimation to find the fixed but unknown parameters of a population. Bayesian Estimation Consider the parameters of a population to be random variables.

Heart of Bayesian Classification Ultimate Goal: Evaluate What can we do if prior probabilities and class-conditional densities are unknown?

Helpful Knowledge Functional form for unknown densities e.g., Normal, exponential, … Ranges for the values of unknown parameters e.g., uniform distributed over a range Training Samples Sampling according to the states of nature.

Posterior Probabilities from Sample

Posterior Probabilities from Sample Each class can be considered independently

Problem Formulation D This the central problem of Bayesian Learning. Let D be a set of samples drawn independently according to the fixed but known distribution p(x). We want to determine D This the central problem of Bayesian Learning.

Parameter Distribution Assume p(x) is unknown but knowing it has a fixed form with parameter vector . is complete known Assume  is a random vector, and p() is a known a priori.

Class-Conditional Density Estimation

Class-Conditional Density Estimation The posterior density we want to estimate The form of distribution is assumed known

Class-Conditional Density Estimation If p(|D) has a sharp peak at

Class-Conditional Density Estimation

The Univariate Gaussian: Unknown  distribution form is known assume  is normal distributed

The Univariate Gaussian: Unknown 

The Univariate Gaussian: Unknown  Comparison

The Univariate Gaussian: Unknown 

The Univariate Gaussian: Unknown 

The Univariate Gaussian: p(x|D)

The Univariate Gaussian: p(x|D)

The Univariate Gaussian: p(x|D) =?

The Multivariate Gaussian: Unknown  distribution form is known assume  is normal distributed

The Multivariate Gaussian: Unknown 

The Multivariate Gaussian: Unknown 

General Theory 1. the form of class-conditional density is known. 2. knowledge about the parameter distribution is available. samples are randomly drawn according to the unknown probability density p(x). 3.

General Theory 1. the form of class-conditional density is known. 2. knowledge about the parameter distribution is available. samples are randomly drawn according to the unknown probability density p(x). 3.

Incremental Learning Recursive

1. Example 2. 3.

1. Example 2. 3. 4 2 4 6 8 10  p(|Dn) 3 2 1