1. Estimation
A major purpose of statistics is to estimate some characteristics of a population.
Take a sample from the population under study and
Compute sample statistics to estimate the desired characteristics.
We generally will have a rule, called an estimator which is just a function defined on the sample.
Computed value of the estimator is an estimate.
A single number estimate of a parameter from a sample is a point estimate statistic
We want our estimator to be a “sensible rule”
No unique way to do this. We’ll use:
Integration followed by algebra: Method of Moments
Differentiation followed by algebra: Maximum Likelihood Estimation
Simulation: The Bootstrap
There are also Bayes estimators, but we won’t get to them unfortunately.

2. General concepts of point estimation

3. Estimator Rule of Thumb
Simplest rule for choosing an estimator:
pick the sample analog of the parameter to be estimated
estimate the population expected mean with the sample mean.
estimate the population standard deviation with the sample standard deviation.
estimate the population median with the sample median.
estimate a population proportion, use the sample proportion.
The rule isn’t rigorous but it is quite useful.

4. Example
Consider the following random sample of observations on coating thickness for low-viscosity paint. Assume that the distribution of coating thickness is normal and we want some properties of coating thickness.
0.83
0.88
1.04
1.09
1.12
1.29
1.31
1.48
1.49
1.59
1.62
1.65
1.71
1.76
1.83
Calculate a point estimate for the mean value of coating thickness. (Best guess for mean)
Calculate a point estimate for the median of coating thickness.(Best guess for median)
Calculate a point value for the 90th percentile of coating thickness. (Best guess for 90th percentile)
Estimate P(X < 1.5), proportion of all thicknesses below 1.5 (Best guess for probability)

6. Unbiasedsness and bias
If is not unbiased then the difference:
is called the bias of
Look at the bias of some estimators below

7. Variances of estimators

8. The standard error of an estimator

9. Formulating estimators

10. Example
In his last 20 shots on goal, Fred Sasakamoose
had the following record:
What is Fred’s estimated probability of making a goal, and what is the standard error?

11. Method of Moments
Definitions:

12. Deriving the expression for the mth moment involves integration:
IF the integrals can be evaluated, the we’re left with (probably non-linear) algebraic equations for the parameters

13. Find estimators for the parameters of the Gamma distribution via the method of moments.
First moment of the Gamma distribution (the mean)

14. Find estimators for the parameters of the Gamma distribution via the method of moments.
First moment of the Gamma distribution

15. So now we have two equations and two unknowns.
Plug in data to get numbers for these expectation values
a and b (the parameters) are “unknowns”

16. The method of moments estimates for the parameters of the Gamma distribution:

17. Example
Compute the method of moments estimates for the parameters of a gamma distribution using the data:
0.95, 1.38, 1.11, 2.20, 1.54, 5.77, 1.43, 3.82, 1.32, 0.74

18. Maximum Likelihood
Log-likelihood is often easier to work with

19. Maximum likelihood estimators

20. Maximum likelihood estimation in practice

21. Maximum likelihood estimation in practice
Deriving the expression for maximum log-likelihood involves differentiation with respect to the pdf’s parameters:
Setting the derivatives = 0 locates the parameters that give the maximum likelihood, and a set of equations to find them.
This is still not an easy task.

22. Maximum Likelihood
Find estimators for the parameters of the Gamma distribution via maximum likelihood.
Log-likelihood of the Gamma pdf for a set of data
Simplified slightly

23. Maximum Likelihood
Now to Mathematica. To keep things simple for now just assum 3 data points:
Log-likelihood:

24. Maximum Likelihood Substituting for b:
If we generalized to n data points:
We can’t solve this for a directly. We solve it iteratively instead:
YUK!

25. Maximum Likelihood Luckily, we can do this all numerically in R
Example: On a particular month, the average scaled load on a web server is 1. For subsequent months the average scaled loads on the same server appear below.
0.95, 1.38, 1.11, 2.20, 1.54, 5.77, 1.43, 3.82, 1.32, 0.74
Compute the maximum likelihood estimates for the parameters of a gamma distribution given this data.
What is the distributions mean and standard deviation?
Plot the distribution.

26. Maximum Likelihood Luckily, we can do this all numerically in R
Compute the maximum likelihood estimates for the parameters of a gamma distribution given this data.
What is the distributions mean and standard deviation?
Plot the distribution.

27. The Bootstrap Method
Moral of the story: Neither the maximum likelihood nor the method of moments is an easy thing to do by hand but:
They are both principled ways of finding estimators for distribution parameters
Sometimes they yield closed form equations
Numerical MLE is available in R (and many other software packages)
If:
You have a limited sample of data and want an estimate of the sampling distributions of the population parameters
You have a parameter that has a difficult to deal with or unknown/undefined distribution
Use the simulation method called “the bootstrap”

28. The Parametric Bootstrap Method
Using the same server load data from the previous example:
0.95, 1.38, 1.11, 2.20, 1.54, 5.77, 1.43, 3.82, 1.32, 0.74
Estimate the parameters for it’s gamma distribution using the parametric bootstrap method. Plot the parameters estimated sampling distributions.
1. Look up or determine the distribution’s parameters as a function of easy to compute sample statistics (usually mean and variance):

29. The Parametric Bootstrap Method
2. Using these formulas estimate the parameters from the data:

30. The Parametric Bootstrap Method
3. Using the estimates obtain a random sample the same size as the original sample.
Called a (parametric) bootstrap sample
Compute estimates of the parameters from this new sample

31. The Parametric Bootstrap Method
4. Repeat this process many times. Typically I use 2000 bootstrap replications:
Loop to repeat bootstrap calculation
Bootstrap estimates
Estimates of sampling uncertainty in the parameters (i.e. the bootstrap standard deviations)

32. The Parametric Bootstrap Method
Estimated sampling distributions of the parameters via the bootstrap.