1. July 3, 2015 1 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden Minimal sufficient statistic

2. July 3, 2015 2 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden A statistic defines a partition of the sample space of (X 1, …, X n ) into classes satisfying T(x 1, …, x n ) = t for different values of t. If such a partition puts the sample x = (x 1, …, x n ) and y = (y 1, …, y n ) into the same class if and only if then T is minimal sufficient for 

3. July 3, 2015 3 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden Example

4. July 3, 2015 4 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden Rao-Blackwell theorem

5. July 3, 2015 5 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden The Exponential family of distributions A random variable X belongs to the (k-parameter) exponential family of probability distributions if the p.d.f. of X can be written What about  N( ,  2 ) ?  Po( ) ?  U(0,  ) ?

6. July 3, 2015 6 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden For a random sample x = (x 1, …, x n ) from a distribution belonging to the exponential family

7. July 3, 2015 7 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden Exponential family written on the canonical form:

8. July 3, 2015 8 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden Completeness Let x 1, …, x n be a random sample from a distribution with p.d.f. f (x;  )and T = T (x 1, …, x n ) a statistic Then T is complete for  if whenever h T (T ) is a function of T such that E[h T (T )] = 0 for all values of  then Pr(h T (T )  0) = 1 Important lemmas from this definition:  Lemma 2.6: If T is a complete sufficient statistic for  and h (T ) is a function of T such that E[h (T ) ] = , then h is unique (there is at most one such function)  Lemma 2.7: If there exists a Minimum Variance Unbiased Estimator (MVUE) for  and h (T ) is an unbiased estimator for , where T is a complete minimal sufficient statistic for , then h (T ) is MVUE  Lemma 2.8: If a sample is from a distribution belonging to the exponential family, then (  B 1 (x i ), …,  B k (x i ) ) is complete and minimal sufficient for  1, …,  k

9. July 3, 2015 9 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden Maximum-Likelihood estimation Consider as usual a random sample x = x 1, …, x n from a distribution with p.d.f. f (x;  ) (and c.d.f. F(x;  ) ) The maximum likelihood point estimator of  is the value of  that maximizes L(  ; x ) or equivalently maximizes l(  ; x ) Useful notation: With a k-dimensional parameter:

10. July 3, 2015 10 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden Complete sample case: If all sample values are explicitly known, then Censored data case: If some ( say n c ) of the sample values are censored, e.g. x i k 2, then where

11. July 3, 2015 11 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden When the sample comes from a continuous distribution the censored data case can be written In the case the distribution is discrete the use of F is also possible: If k 1 and k 2 are values that can be attained by the random variables then we may write where

12. July 3, 2015 12 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden Example

13. July 3, 2015 13 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

14. July 3, 2015 14 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden Example

15. July 3, 2015 15 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden Too complicated to find an analytical solutions. Solve by a numerical routine!

16. July 3, 2015 16 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden Exponential family distributions: Use the canonical form (natural parameterization): Let Then the maximum likelihood estimators (MLEs) of  1, …,  k are found by solving the system of equations

17. July 3, 2015 17 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden Example

18. July 3, 2015 18 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

19. July 3, 2015 19 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden Computational aspects When the MLEs can be found by evaluating numerical routines for solving the generic equation g(  ) = 0 can be used. Newton-Raphson method Fisher’s method of scoring (makes use of the fact that under regularity conditions: ) This is the multidimensional analogue of Lemma 2.1 ( see page 17)

20. July 3, 2015 20 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden When the MLEs cannot be found the above way other numerical routines must be used: Simplex method EM-algorithm For description of the numerical routines see textbook. Maximum Likelihood estimation comes into natural use not for handling the standard case, i.e. a complete random sample from a distribution within the exponential family, but for finding estimators in more non- standard and complex situations.

21. July 3, 2015 21 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden Example

22. July 3, 2015 22 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

23. July 3, 2015 23 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden Properties of MLEs Invariance: Consistency: Under some weak regularity conditions all MLEs are consistent

24. July 3, 2015 24 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden Efficiency: Under the usual regularity conditions: (Asymptotically efficient and normally distributed) Sufficiency:

25. July 3, 2015 25 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden Example

26. July 3, 2015 26 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

27. July 3, 2015 27 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

28. July 3, 2015 28 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden Invariance property 

29. July 3, 2015 29 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

30. July 3, 2015 30 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

31. July 3, 2015 31 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden i.e. the two MLEs are asymptotically uncorrelated (and by the normal distribution independent)

32. July 3, 2015 32 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden Modifications and extensions Ancillarity and conditional sufficiency:

33. July 3, 2015 33 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden Profile likelihood: This concept has its main use in cases where  1 contains the parameters of “interest” and  2 contains nuisance parameters. The same ML point estimator for  1 is obtained by maximizing the profile likelihood as by maximizing the full likelihood function

34. July 3, 2015 34 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden Marginal and conditional likelihood: Again, these concepts have their main use in cases where  1 contains the parameters of “interest” and  2 contains nuisance parameters.

35. July 3, 2015 35 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden Penalized likelihood: MLEs can be derived subjected to some criteria of smoothness. In particular this is applicable when the parameter is no longer a single value (one- or multidimensional), but a function such as an unknown density function or a regression curve. The penalized log-likelihood function is written

36. July 3, 2015 36 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden Method of moments estimation (MM )

37. July 3, 2015 37 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

38. July 3, 2015 38 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden The method of moments point estimator of  = (  1, …,  k ) is obtained by solving for  1, …,  k the systems of equations

39. July 3, 2015 39 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden Example

40. July 3, 2015 40 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden

41. July 3, 2015 41 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden Method of Least Squares (LS) First principles: Assume a sample x where the random variable X i can be written The least-squares estimator of  is the value of  that minimizes i.e.

42. July 3, 2015 42 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden A more general approach Assume the sample can be written (x, z ) where x i represents the random variable of interest (endogenous variable) and z i represent either an auxiliary random variable (exogenous) or a given constant for sample point i The least squares estimator of  is then

43. July 3, 2015 43 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden Special cases The ordinary linear regression model: The heteroscedastic regression model:

44. July 3, 2015 44 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden The first-order auto-regressive model:

45. July 3, 2015 45 Department of Computer and Information Science (IDA) Linköpings universitet, Sweden The conditional least-squares estimator of  (given  ) is