EC 331 The Theory of and applications of Maximum Likelihood Method Burak Saltoğlu

outline Maximum Liklelihood Principle Estimating population parameters via ML method Properties of ML OLS vs ML

This represents the joint density of y’s given parameter 1 Maximum Likelihood ML method is based on the principle that the parameter estimates can be obtained by maximising the likelihood of the selected sample to reflect the population. We choose the parameters in a way that we maximize the joint likelihood of representing the population. Suppose we are given iid observed sample of y and also a parameter vector (of k dimesion) can be represented as This represents the joint density of y’s given parameter

Likelihood Function Joint likelihood function then can be written as the joint probability of observing y’s drawn from f(.) Likelihood function is Maximizing above function w.r.t. will yield a special value that maximizes the probability of obtaining sample values that have actually observed. In most applications it is convinient to work with loglikelihood function, which is

Likelihood Function Note that Also note that above equation is known as score .

Example-1 Poisson distribbution due to Siméon Denis Poisson expresses the probability of a given number of events occurring in a fixed interval of time these events occur with a known average rate and independently of the time since the last event use: defaults of countries, customers,

Example-1

Example-1

Numerical example

Numerical example

Likelihood profile (lambda in the horizontal axis)

Likelihood and log-likelihood for Poisson (rescaled Graph)

Example-2 It describes the time between events in a Poisson process

Example-2

Example-2

Example-3

Example-4

Convergence in Probability Definition : Let xn be a sequence random variable where n is sample size, the random variable xn converges in probability to a constant c if the values that the x may take that are not close to c become increasingly unlikely as n increases. If xn converges to c, then we say, All the mass of the probability distribution concentrates around c.

Properties of MLE Consistency: Asymtotic Normality: where information matrix is that is, the hessian of log-likelihood function.

3.3 Properties of MLE Asymtotic Efficiency: Assumimg that we are dealing with only one parameter θ; which states that if there is another consistent and asymtotically normal estimator of to θ then , Invariance:

4 Estimation of the Linear Regression Model

Matrix notation

3.4 Estimation of the Linear Regression Model Parameter vector is

3.4 Estimation of the Linear Regression Model To calculate variance matrix of parameters, we need hessian of likelihood parameters. İf we take ot second derivatives Taking expectations,

3.4 Estimation of the Linear Regression Model

3.4 Estimation of the Linear Regression Model So, the information matrix is The inverse of the information matrix will give us the variance-covariance matrix of the MLE estimators,

Testing in Maximum Likelihood Framework

Example from Poisson example

Example This ratio is always between 0 and 1 and the less likely the assumption is, the smaller this ratio 

Likelihood Ratio Test If we want to test Restricted likelihood ratio defined as can be used with decision rule Restricted Unrestricted q:#restrictions

Likelihood Ratio Test Don’t reject the null

More on LR test in the context of Linear Regression

Likelihood Ratio Test