1. STATISTICS POINT ESTIMATION Professor Ke-Sheng Cheng Department of Bioenvironmental Systems Engineering National Taiwan University

2. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 2

3. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Is it originated from a normal (or gamma) distribution? 3

4. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Statistical inference( ) Given a random sample from the distribution of a population, we often are interested in making inferences about the population. Two important statistical inferences are – Estimation ( ) – Test of hypotheses ( ). Normal distribution? 4

5. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Parameter estimation Assume that some characteristics of the elements in a population can be represented by a RV X with pdf f X ( ;θ), where the form of the density is assumed known except that it contains some unknown parameters θ. We want to estimate an unknown parameter θ or some function of the unknown parameter, τ(θ), using observed values of a random sample, x 1,x 2, …,x n. 5

6. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Point estimation Let the value of some statistic, say, represent the unknown parameter θ or τ(θ); such a statistic is called a point estimator. For example, and respectively are point estimator of and. 6

7. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Interval estimation Two statistics and, where, so that (, ) constitutes an interval for which the probability can be determined that it contains the unknown parameter θ or τ(θ). 7

8. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Point Estimation of Distribution Parameters What parameters are to be estimated? – Parameters are variables that characterize distributions. For example mean and standard deviation are parameters. How can we estimate the parameters ? – methods of finding estimators – desired properties of point estimators 8

9. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. To estimate parameters we need to have a random sample. For example, a random sample (x 1,x 2, …,x n ) of f( ;θ) is collected in order to estimate the parameter θ. Therefore, There can be many (or infinite) ways of estimation and we need to establish some kind of criteria in order to have an adequate estimator. 9

10. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Estimators and Estimates Estimator: Any statistic whose values are used to estimate τ(θ), where τ( ) is some function of the parameter θ, is defined to be an estimator of. Note that an estimator is a random variable. Estimate: The estimated value given by an estimator. 10

11. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Methods of finding estimator Assume that x 1,x 2, …,x n is a random sample from a density f( ;θ), where the form of the density is known but the parameter θ is unknown. Further assume that θ is a vector of real numbers, say θ=(θ 1,θ 2, …,θ k ). We often let, called the parameter space, denote the set of possible values that the parameter θ can assume. Our objective is to find statistics to be used as estimators of certain functions, say, of θ= (θ 1, …, θ k ). 11

12. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Method of moments Let f( ;θ 1,θ 2, …,θ k ) be the density of random variable X which has k parameters. In general will be a known function of the k parameters, i.e. Let x 1, x 2, …, x n be a random sample from the density f( ; ). From the k equations, we have 12

13. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. We can solve a solution of expressed in terms of x 1, x 2, …, x n. The solution, i.e., is called an estimator of. 13

14. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 14

15. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Given a random sample of a normal distribution with mean μ and varianceσ 2. Estimate the parameters μ and σ by the method of moments. 15

16. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Note that the method of moment estimator of σ 2 is NOT the sample variance. 16

17. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Let x 1, x 2, …, x n be a random sample from a Poisson distribution with parameter λ. Estimate using MOM. 17

18. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Let be a random sample from a uniform distribution on. What are the method of moments estimator of and ? 18

19. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Maximum Likelihood Method Definition of the likelihood function 19

20. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 20

21. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Rationale of using likelihood function for parameter estimation 21

22. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 22

23. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 23

24. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 24

25. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Definition of the maximum likelihood estimator 25

26. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 26

27. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 27

28. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Example 28

29. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 29

30. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 30

31. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Example 31

32. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Example 32

33. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Lower bound for y 1 Upper bound for y n 33

34. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 34

35. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 35

36. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Example 36

37. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 37

38. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 38

39. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 39

40. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 40

41. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 41

42. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Parameter estimators of various distributions 42

43. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 43

44. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Properties of point estimators Among different estimators, we want to know whether one estimator is better than others or what properties an estimator may or may not possess. Consider the case that we estimate using a statistic of a random sample from a density. Intuitively, we look for an estimator that is close to. However, what is the definition of closeness ? (Or, how do we measure closeness?) 44

45. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Most concentrated estimator 45

46. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Pitman-closer and Pitman- closest 46

47. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Unbiased estimator An estimator ( ) of is said to be unbiased if An unbiased estimator is said to be more efficient than any other unbiased estimator of, if for all. 47

48. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Mean squared error of an estimator 48

49. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Naturally, we would prefer to find an estimator that has the smallest mean-squared error, however, such estimators rarely exist. In general, the mean-squared error of an estimator depends on. What should we look for if a uniformly minimum MSE estimator rarely exists? 49

50. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Uniformly minimum MSE estimator 50

51. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Let s define = bias of T. [Note: is an estimator of.] Then, 51

52. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 52

53. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 53

54. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 54 Bias

55. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. We look for an estimator that has a uniformly minimum MSE within the class of unbiased estimators. Such an estimator is called a uniformly minimum-variance unbiased estimator (UMVUE). 55

56. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Example 56

57. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 57

58. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. The MSE of is 58

59. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 59

60. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Standard Error The standard error of a statistic is the standard deviation of its sampling distribution. If the standard error involves unknown parameters whose values can be estimated, substitution of those estimates into the standard error results in an estimated standard error. 60

61. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 61

62. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. Mean-squared-error consistency We discuss the mean-squared error of an estimator derived from a random sample of fixed size n. Properties of point estimators that are defined for a fixed sample size are referred to as small-sample properties, whereas properties that are defined for increasing sample size are referred to as large-sample properties. 62

63. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 63

64. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 64

65. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 65

66. Properties of the maximum likelihood estimators Asymptotic properties of the MLEs 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 66

67. Invariance property of the MLEs The invariance property does not hold for unbiasedness. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 67

68. 1/31/2014 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 68