01/20151 EPI 5344: Survival Analysis in Epidemiology Maximum Likelihood Estimation: An Introduction March 10, 2015 Dr. N. Birkett, School of Epidemiology, Public Health & Preventive Medicine, University of Ottawa

01/20152 Objectives MLE was introduced by me in EPI5340 Likely covered in other courses too. Won’t cover much on the basics. Parameter estimation using maximum likelihood Using MLE to estimate variance and do statistical testing.

01/20153 Intro (1) Conduct an experiment –Toss a coin 10 times and observe 6 heads –What is the probability of getting a head when tossing this coin? –NOTE: we do not know that the coin is fair! Let p = prob(head). Assume binomial dist’n:

01/20154 Intro (3) We can give a formula for how likely the data is, given a specific value of ‘p’:

01/20155

6 Intro (4) For mathematical ease, one usually works with the logarithm of the likelihood –Has the same general shape –Has the same maximum point

01/20157

8 Intro (5) What value of ‘p’ makes the log(L) as large as possible? Log(L) curves have the same general shape –An inverted ‘U’ Have one point which is the maximum. Use calculus to find it To find maximum, find ‘p’ which makes this equal to ‘0’

01/20159 Intro (6) To find maximum, find ‘p’ which makes this equal to ‘0’

01/ Intro (7) Suppose we re-do experiment and get 600 heads in 1,000 tosses. What is p MLE ? –600/1000 = 0.6 (the same) Do we gain anything by doing 100 times for tosses? –Plot the log(L) curve

01/ Much narrower

01/ MLE (1) Likelihood –how likely is the observed data given that the parameter(s) assume a fixed value(s) It is not the probability of the observed data Assumes –We have a parametric model for the data –Usually assumes independent observations Coin tosses are independent, each with a Bernoulli Dist'n When plotted, scale on y-axis is arbitrary Usually work with ln(L): the natural logarithm of L

01/ MLE (2) Ln(L) curve is nearly always an inverted ‘U’ (inverted parabola) The value of the parameter which makes the curve as high as possible makes the observed data the most likely. –Maximum Likelihood Estimator (MLE)

01/ MLE (3) The width of the ln(L) curve relates to the variance of the parameter estimate –More precisely, the variance is related to: slope of the slope of the ln(L) curve at the MLE Referred to as: Fisher’s Information

01/201515

01/201516

Another example: incidence rate 01/ # of observed events (D) follows a Poisson Distribution:

01/ To find the MLE, set this slope to ‘0’ The formula for the incidence rate from epidemiology

Normal(Gaussian) 1 observation only 01/ We will assume that σ is known To find MLE, set = 0

Normal(Gaussian) ‘N’ observations 01/ Previous may not seem useful – who does a study with one data point? So, let’s suppose we have ‘N’ observations: x 1 …x N All normally distributed with common mean and variance Assume that σ is known

Normal(Gaussian) ‘N’ observations 01/

Normal(Gaussian) ‘N’ observations 01/ To find MLE, set

01/ Approximations (1) All likelihoods have a similar shape –Inverted ‘U’, with one peak Over some range of parameter values (near the MLE), all likelihood curves look like a parabola –Larger sample size  larger range of fit We can approximate any likelihood curve with a parabola  Normal approximation. This is useful since it provides statistical tests.

01/ Approximations (2) General Idea –Assume that true likelihood is based on one parameter θ –θ MLE is most likely value of θ –We want to find a normal likelihood with a peak at the same point and which ‘looks similar’ around the MLE point: True ln(L) Normal approx

01/ Approximations (3) For a Gaussian curve, we have (ignoring the constant: We have seen that, for this situation, Our ‘true’ curve has an MLE of To have the same peak, we need to set:

01/ Approximations (4) What do we mean by ‘similar shape’? –Can’t use ‘slope’ since it is always ‘0’ at MLE Many criteria could be used. We will use ‘curvature’

01/ Approximations (5) Curvature = - second derivative of log(L) = - Information Curvature –The slope of the slope of the likelihood curve at the MLE Rate at which the slope is changing at the MLE Peeked curves have higher values It is always < 0

01/ Approximations (6) What is the curvature at the peak (MLE) for a Gaussian? Which is a constant! Set to the curvature of ‘real’ curve to get approximate curve

Approximations (7) To get a ‘good’ normal approximation in the region of the MLE, here’s what we need to do Set the ‘mean’ of the normal curve to Set the variance of the normal curve to the negative of the reciprocal of the curvature of the target: 01/ How to do this depends on the ‘target’

01/ Approximations (8) Approximation to binomial dist’n ‘N’ events ‘D’ are positive Want to find a normal approximation to use around the MLE

01/ Approximations (9) We need the curvature at the MLE. So, make these 2 substitutions: This gives: So, the normal approximation uses:

01/201532

01/ Hypothesis tests (1) Simple hypothesis test: –H 0 : mean = μ 0 We’ll do this using a Likelihood approach Based off the real curve, not an approximation (for now) Determine the likelihood at: –Null hypothesis –MLE (the observed data) –Subtract likelihoods (‘MLE’ from ‘null’)

01/ p MLE Null

01/ Difference in log-likelihood = -18 p MLE Null

01/ p MLE Null Difference in log-likelihood = -0.1

01/ Hypothesis tests (2) We want to test Sample: x 1, x 2,…,x n iid~N(μ, σ 2 ), σ 2 is assumed ‘known’. We know that: Likelihood ratio test of null hypothesis NOTE: for convenience, I have scaled the ln(L) axes so the the value at the MLE is ‘0’. In reality, the ln(L) value at the MLE is not ‘0’.

01/ Hypothesis tests (3) Likelihood Curve

01/ Hypothesis tests (4) But, it again is easier to work with logs. So, the test is based on:

01/ Hypothesis tests (5) First, remember that for a normal distribution, we have: So, at the null hypothesis, we have: And at the MLE point, we have:

01/ Hypothesis tests (6) Distributed as Should recognize this test from Biostats 1 After a bit of algebra

Likelihood ratio test = -2ΔLR ~ –If x’s are normal, test is exact –If x’s are not normal, test is not exact but isn’t bad. Assumes that we know the true shape of the likelihood curve. What if we don’t? Use an approximation Two main methods –Wald –Score 01/ Hypothesis tests (7)

01/ Hypothesis tests (8) Wald test –Assumes that the true and normal curves have: the same peak value (the MLE) Same curvature at the peak value –Is an approximate test which is best around the MLE Good for 95% confidence intervals. –Tends to under-estimate the LR test value.

01/ Wald approximation Wald True

01/ True LR test Wald LR test Wald True

01/ Hypothesis tests (9) Score test –Assumes that the true and normal curves have: Same slope and curvature at the null value –Implies that the peaks are not the same the MLEs are also not the same –Is an approximate test which is best around the Null hypothesis

01/201547

01/ Hypothesis tests (10) Regression models –can be fit using MLE methods –most common approach used for logistic regression Cox regression Poisson regression Data will be iid and normally distributed with:

01/ Hypothesis tests (11) Can use MLE to estimate the Betas Fitted model will have a ln(L) value. Now, fit two models: –one with x –one without x. Each model will have a ln(L) –ln(L with x ) –ln(L without x )

01/ Hypothesis tests (12) Likelihood ratio test of is given by: Complicated way to test one Beta Easily extended to more complex models Very similar to using Partial F-tests which you covered when learning linear regression

01/201551