CS479/679 Pattern Recognition Dr. George Bebis Parameter Estimation: Maximum Likelihood Estimation Chapter 3 (Duda et al.) – Sections 3.1-3.2 CS479/679 Pattern Recognition Dr. George Bebis

Parameter Estimation Bayesian Decision Theory allows us to design an optimal classifier given that we have estimated P(i) and p(x/i) first: Estimating P(i) is usually not very difficult. Estimating p(x/i) could be more difficult: Dimensionality of feature space is large. Number of samples is often too small.

Parameter Estimation (cont’d) We will make the following assumptions: A set of training samples D ={x1, x2, ...., xn} is given, where the samples were drawn according to p(x|wj). p(x|wj) has some known parametric form: Parameter estimation problem: e.g., p(x /i) ~ N(μ i , i) also denoted as p(x/q) where q=(μi , Σi) Given D, find the best possible q

Main Methods in Parameter Estimation Maximum Likelihood (ML) Bayesian Estimation (BE)

Main Methods in Parameter Estimation Maximum Likelihood (ML) Best estimate is obtained by maximizing the probability of obtaining the samples D ={x1, x2, ...., xn} actually observed: ML assumes that θ is fixed and makes a point estimate:

Main Methods in Parameter Estimation (cont’d) Bayesian Estimation (BE) Assumes that θ is a set of random variables that have some known a-priori distribution p(θ). Estimates a distribution rather than making a point estimate (i.e., like ML): Note: the BE solution p(x/D) might not be of the parametric form assumed (e.g., p(x/q)).

ML Estimation - Assumptions Consider c classes and c training data sets (i.e., one for each class): Samples in Dj are drawn independently according to p(x/ωj). Problem: given D1, D2, ...,Dc and a model for p(x/ωj) ~ p(x/q), estimate: D1, D2, ...,Dc q1, q2,…, qc

ML Estimation - Problem Formulation If the samples in Dj provide no information about qi ( ), we need to solve c independent problems (i.e., one for each class). The ML estimate for D={x1,x2,..,xn} is the value that maximizes p(D / q) (i.e., best supports the training data). Using independence assumption, we can simplify p(D / q) :

ML Estimation - Solution How can we find the maximum of p(D/ q) ? where (gradient)

ML Estimation Using Log-Likelihood Taking the log for simplicity: Maximizes ln p(D/ θ): log-likelihood

Example p(D / θ) =μ ln p(D/ θ) training data: unknown mean, known variance p(D / θ) =μ ln p(D/ θ)

ML for Multivariate Gaussian Density: Case of Unknown θ=μ Assume Computing the gradient, we have:

ML for Multivariate Gaussian Density: Case of Unknown θ=μ (cont’d) Setting we have: The solution is given by The ML estimate is simply the “sample mean”.

Special Case: Maximum A-Posteriori Estimator (MAP) Assume that θ is a random variable with known p(θ). Maximize p(θ/D) or p(D/θ)p(θ) or ln p(D/ θ)p(θ): Consider:

Special Case: Maximum A-Posteriori Estimator (MAP) (cont’d) What happens when p(θ) is uniform? MAP is equivalent to ML

MAP for Multivariate Gaussian Density: Case of Unknown θ=μ Assume Maximize ln p(μ /D) = ln p(D/ μ)p(μ): and (both are known)

MAP for Multivariate Gaussian Density: Case of Unknown θ=μ (cont’d) If , then What happens when

ML for Univariate Gaussian Density: Case of Unknown θ=(μ,σ2) Assume θ =(θ1,θ2)=(μ,σ2) p(xk/θ) p(xk/θ) p(xk/θ)

ML for Univariate Gaussian Density: Case of Unknown θ=(μ,σ2) (cont’d) p(xk/ θ)=0 =0 =0 The solutions are given by: sample mean sample variance

ML for Multivariate Gaussian Density: Case of Unknown θ=(μ,Σ) In the general case (i.e., multivariate Gaussian) the solutions are: sample mean sample covariance

Biased and Unbiased Estimates An estimate is unbiased when The ML estimate is unbiased, i.e., The ML estimates and are biased:

Biased and Unbiased Estimates (cont’d) The following are unbiased estimates of and

Comments ML estimation is simpler than alternative methods. ML provides more accurate estimates as the number of training samples increases. If the model for p(x/ θ) is correct, and the independence assumptions among samples are true, ML will work well.