236607 Visual Recognition Tutorial Maximum likelihood – an example Maximum likelihood – another example Bayesian estimation Expectation Maximization Algorithm Jensen’s inequality EM for a mixture model 236607 Visual Recognition Tutorial

Bayesian Estimation: General Theory Bayesian leaning considers (the parameter vector to be estimated) to be a random variable. Before we observe the data, the parameters are described by a prior which is typically very broad. Once we observed the data, we can make use of Bayes’ formula to find posterior. Since some values of the parameters are more consistent with the data than others, the posterior is narrower than prior. This is Bayesian learning 236607 Visual Recognition Tutorial

Bayesian parametric estimation Density function for x, given the training data set (it was defined in the Lect.2) From the definition of conditional probability densities The first factor is independent of X(n) since it just our assumed form for parameterized density. Therefore 236607 Visual Recognition Tutorial

Bayesian parametric estimation Instead of choosing a specific value for , the Bayesian approach performs a weighted average over all values of If the weighting factor , which is a posterior of peaks very sharply about some value we obtain . Thus the optimal estimator is the most likely value of given the data and the prior of . 236607 Visual Recognition Tutorial

Bayesian decision making Suppose we know the distribution of possible values of that is a prior Suppose we also have a loss function which measures the penalty for estimating when actual value is Then we may formulate the estimation problem as Bayesian decision making: choose the value of which minimizes the risk Note that the loss function is usually continuous. 236607 Visual Recognition Tutorial

Maximum A-Posteriori (MAP) Estimation Let us look at : the optimal estimator is the most likely value of q given the data and the prior of q This “most likely value” is given by 236607 Visual Recognition Tutorial

Maximum A-Posteriori (MAP) Estimation since the data is i.i.d. We can disregard the normalizing factor when looking for the maximum 236607 Visual Recognition Tutorial

236607 Visual Recognition Tutorial MAP - continued So, the we are looking for is 236607 Visual Recognition Tutorial

236607 Visual Recognition Tutorial Maximum likelihood In MAP estimator, the larger n (the size of the data), the less important is in the expression It can motivate us to omit the prior. What we get is the maximum likelihood (ML) method. Informally: we don’t use any prior knowledge about the parameters; we seek those values that “explain” the data in the best way . is a log-likelihood of with respect to X(n) . We seek a maximum of the likelihood function, log-likelihood, or their monotonically increasing function. 236607 Visual Recognition Tutorial

Maximum likelihood – an example Let us find the ML estimator for the parameter of the exponential density function : so we are actually looking for the maximum of log-likelihood. Observe: The maximum is achieved where We have got the empirical mean (average) 236607 Visual Recognition Tutorial

Maximum likelihood – another example Let us find the ML estimator for Observe: The maximum is at where This is the median of the sampled data. 236607 Visual Recognition Tutorial

Bayesian estimation -revisited We saw Bayesian estimator for 0/1 loss function (MAP). What happens when we assume other loss functions? Example 1: (q is unidimensional). The total Bayesian risk here: We seek its minimum: 236607 Visual Recognition Tutorial

Bayesian estimation -continued At the which is a solution we have That is, for the the optimal Bayesian estimator for the parameter is the median of the distribution Example 2: (squared error). Total Bayesian risk: Again, in order to find the minimum, let the derivative be equal 0: 236607 Visual Recognition Tutorial

Bayesian estimation -continued The optimal estimator here is the conditional expectation of q given the data X(n) . 236607 Visual Recognition Tutorial

236607 Visual Recognition Tutorial Jensen’s inequality Definition: function is convex over (a,b) if Convex Concave Jensen’s inequality: For convex function 236607 Visual Recognition Tutorial

236607 Visual Recognition Tutorial Jensen’s inequality For d.r.v.with two mass points Let Jensen’s inequality is right for k-1 mass points, then due to induction assumption due to convexity 236607 Visual Recognition Tutorial

Jensen’s inequality corollary Let Function log is concave, so from Jensen inequality we have: 236607 Visual Recognition Tutorial

236607 Visual Recognition Tutorial EM Algorithm EM is iterative technique designed for probabilistic models. We have: two sample spaces: X which are observed Y which are missing Vector of parameters q which gives a distribution of X. We should find or 236607 Visual Recognition Tutorial

236607 Visual Recognition Tutorial EM Algorithm The problem is that to calculate Is difficult, but calculation of is relatively easy We define: The algorithm makes cyclically two steps: E: Compute (see (10) below) M: 236607 Visual Recognition Tutorial

236607 Visual Recognition Tutorial EM Algorithm EM is iterative technique designed for probabilistic models. Maximizing a function with lower-bound approximation vs. linear approximation 236607 Visual Recognition Tutorial

236607 Visual Recognition Tutorial EM Algorithm Gradient descend makes linear approximation to the objective function (O.F.), Newton’s method makes quadratic approx. But optimal step is not known. EM instead makes a local approx. that is lower bound (l.b.) to the O.F. Choosing a new guess to maximize the l.b. will always be an improvement, if gradient is not zero. Thus two steps: E – compute a l.b., M-maximize the l.b. The bound used by EM is following from Jensen’s inequality. 236607 Visual Recognition Tutorial

The General EM Algorithm We should make maximization of the function where X is a matrix of observed data. If f(q) is simple, we find maximum by equating its gradient to zero But if f(q) is a mixture (of simple functions) it is difficult. This is a situation for the EM. Given a guess for q find lower bound for f(q) with a function g(q, q(y)), parameterized by free variables q(y). 236607 Visual Recognition Tutorial

236607 Visual Recognition Tutorial EM Algorithm Gradient descend makes linear approximation to the provided Define If we want the lower bound g(q,q) to touch f at the current guess for q , we choose q to maximize G(q, q) . 236607 Visual Recognition Tutorial

236607 Visual Recognition Tutorial EM Algorithm Adding the Lagrange multiplier to the constraint on q gives: For this choice the bound becomes So indeed it touches the objective f(q) . 236607 Visual Recognition Tutorial

236607 Visual Recognition Tutorial EM Algorithm Finding q to get a good bound is the “E” step. To get the next guess for q, we maximize the bound over q (this is the “M” step). It is problem-dependent. The relevant term of G is It may be difficult and also it isn’t strictly necessary to maximize the bound over q . This is sometimes called “generalized EM”. It is clear from the figure that the derivative of g at the current guess is identical to the derivative of f . 236607 Visual Recognition Tutorial

236607 Visual Recognition Tutorial EM for a mixture model We have a mixture of two one-dimensional Gaussians (k=2). Let mixture coefficients be equal: Let variances be The problem is to find We have sample set 236607 Visual Recognition Tutorial

236607 Visual Recognition Tutorial EM for a mixture model To use an algorithm of EM define hidden random variables (indicators) Thus for every i we have: We define every hidden variables: The aim is to calculate and to maximize Q. 236607 Visual Recognition Tutorial

236607 Visual Recognition Tutorial EM for a mixture model For every xi we have: From the assumption of iid for the sample set we have: We see that an expression is linear in . 236607 Visual Recognition Tutorial

236607 Visual Recognition Tutorial EM for a mixture model STEP E: We want to calculate an expected value relative to 236607 Visual Recognition Tutorial

236607 Visual Recognition Tutorial EM for a mixture model STEP M: Differentiating and equating to zero we’ll have: Thus 236607 Visual Recognition Tutorial

EM mixture of Gaussians In what follows we use j instead of y because missing variables are discrete in this example. Model density is a linear combination of component densities p(x | j,q) : where M is a number of basis functions (parameter of the model), P(j) are mixing parameters. They actually are prior probabilities of the data point having been generated from component j of the mixture. 236607 Visual Recognition Tutorial

EM mixture of Gaussians They satisfy The component density function p(x | j) are normalized: We shall use Gaussians for p(x | j) We should find 236607 Visual Recognition Tutorial

EM mixture of Gaussians STEP E: calculate when . (See formulas (8) and (10)) We have: We maximize (17) with constrain (12): 236607 Visual Recognition Tutorial

EM mixture of Gaussians STEP M: Derivative of (18) with respect to Pnew(j): Thus Using (12) we shall have So from (21) and (20) : 236607 Visual Recognition Tutorial

EM mixture model. General case By calculating derivatives from(18) due to and we’ll have: 236607 Visual Recognition Tutorial

EM mixture model. General case Algorithm for calculating p(x) (formula (11)). For every x begin initialize do fixed number of times Calculate formulas (22),(23),(24) return formula (11). end 236607 Visual Recognition Tutorial