1. AGC DSP AGC DSP Professor A G Constantinides© Estimation Theory We seek to determine from a set of data, a set of parameters such that their values would yield the highest probability of obtaining the observed data. The unknown parameters may be seen as deterministic or random variables There are essentially two alternatives to the statistical case When no a priori distribution assumed then Maximum Likelihood When a priori distribution known then Bayes

2. AGC DSP AGC DSP Professor A G Constantinides© Maximum Likelihood Principle: Estimate a parameter such that for this value the probability of obtaining an actually observed sample is as large as possible. I.e. having got the observation we “look back” and compute probability that the given sample will be observed, as if the experiment is to be done again. This probability depends on a parameter which is adjusted to give it a maximum possible value. Reminds you of politicians observing the movement of the crowd and then move to the front to lead them?

3. AGC DSP AGC DSP Professor A G Constantinides© Estimation Theory Let a random variable have a probability distribution dependent on a parameter The parameter lies in a space of all possible parameters Let be the probability density function of Assume the the mathematical form of is known but not

4. AGC DSP AGC DSP Professor A G Constantinides© Estimation Theory The joint pdf of sample random variables evaluated at each the sample points Is given as The above is known as the likelihood of the sampled obserevation

5. AGC DSP AGC DSP Professor A G Constantinides© Estimation Theory The likelihood function is a function of the unknown parameter for a fixed set of observations The Maximum Likelihood Principle requires us to select that value of that maximises the likelihood function The parameter may also be regarded as a vector of parameters

6. AGC DSP AGC DSP Professor A G Constantinides© Estimation Theory It is often more convenient to use The maximum is then at

7. AGC DSP AGC DSP Professor A G Constantinides© An example Let be a random sample selected from a normal distribution The joint pdf is We wish to find the best and

8. AGC DSP AGC DSP Professor A G Constantinides© Estimation Theory Form the log-likelihood function Hence or

9. AGC DSP AGC DSP Professor A G Constantinides© Fisher and Cramer-Rao The Fisher Information helps in placing a bound on estimators Cramer-Rao Lower Bound:“If is any unbiased estimator of based on maximum likelihood then Ie provides a lower bound on the covariance matrix of any unbiased estimator

10. AGC DSP AGC DSP Professor A G Constantinides© Estimation Theory It can be seen that if we model the observations as the output of an AR process driven by zero mean Gaussian noise then the Maximum Likelihood estimator for the variance is also the Least Squares Estimator.