1. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. Lecture 3: Monte Carlo Simulations (Chapter 2.8–2.10)

2. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-2 Review We want an estimator to form a “best guess” of the slope of a line through the origin. We want an estimator that works well across many different samples: low average error, low average absolute error, low squared errors….

3. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-3 Review (cont.) We have brainstormed 4 “best guesses”:

4. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-4 Review (cont.) We will compare these estimators in “horseraces” across thousands of computer-generated datasets. We get to specify the underlying relationship between Y and X. We can see how each estimator does.

5. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-5 Review (cont.) We choose all the rules for how our data are created. The underlying rules are the “Data Generating Process” (DGP). We choose to use the Gauss– Markov Assumptions.

6. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-6 Review (cont.) What is our Data Generating Process (DGP)? – E(  i ) = 0 – Var(  i ) =   2 – Cov(  i,  k ) = 0 i ≠ k –X 1, X 2, …, X n are fixed across samples GAUSS–MARKOV ASSUMPTIONS

7. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-7 How to Build a Race Track… n = ? – How big is each sample?  = ? – What slope are we estimating? Set X 1, X 2, ….., X n – Do it once, or for each sample? Draw  1,  2, …..,  n – Must draw randomly each sample Form Y 1, Y 2, ….., Y n –Y i =  X i +  i Compute all four estimators Save errors, absolute errors, squared errors Do for 10,000 samples and compare

8. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-8 What Will We Get? We will get precise estimates of: 1.Mean Error of Each Estimator 2.Mean Absolute Error of Each Estimator 3.Mean Squared Error of Each Estimator 4.Distribution of Each Estimator By running different racetracks, we check the robustness of our results.

9. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-9 Agenda for Today Off to the Races! Today we run the simulations and report the results. How well does each estimator do? Does one estimator outperform the others in all settings? Next time, we move from computational methods to analytical methods.

10. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-10 What Shall We Explore? Mean Error, Mean Absolute Error, Mean Square Error, & the distribution of each  g Would 1,000 samples do well enough? Does sample size matter? – Try n = 6, n = 100

11. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-11 What Shall We Explore? (cont.) Does normality matter? – For n = 6? For n = 100? Do X ’s matter? – Try low X ’s & high X ’s Does  matter? – Try  = 2 &  = -3 &  = 20

12. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-12 A Scorecard

13. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-13 EstimatorMean EstimateStandard Deviation  g 1 2.069615.59  g 2 2.052013.11  g 3 1.647142.84  g 4 2.037912.52 A First Monte Carlo: n = 6 ;  = 2 ;  ~ N( 0,500 2 ) ; X ’s = { 10, 20, 10, 20, 10, 20 } How close does each estimator come to  = 2? –Results from 10,000 samples

14. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-14 A Scorecard

15. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-15 A First Monte Carlo (cont.) : n = 6 ;  = 2  ~ N( 0,500 2 ) ; X ’s = { 10, 20, 10, 20, 10, 20 } What is the Mean Error? – Results from 1000 samples EstimatorMean Error Standard Deviation of Error  g 1 0.0696 15.59  g 2 0.0520 13.11  g 3 -0.3529 42.84  g 4 0.0379 12.52 What do you conclude?

16. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-16 A First Monte Carlo (cont.) : n = 6 ;  = 2  ~ N( 0,500 2 ) ; X ’s = { 10, 20, 10, 20, 10, 20 } Results from 1000 samples Estimator Mean Abs. Error  g 1 12.20  g 2 10.24  g 3 34.30  g 4 9.78 What do you conclude?

17. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-17 A First Monte Carlo (cont.) : n = 6 ;  = 2  ~ N( 0,500 2 ) ; X ’s = { 10, 20, 10, 20, 10, 20 } Results from 1000 samples Estimator Mean Sq. Error  g 1 243.05  g 2 171.83  g 3 1834.35  g 4 156.85 What do you conclude?

18. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-18 A Scorecard

19. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-19 A First Monte Carlo Pictures are often very helpful in econometrics. What do the errors look like for each estimator? Let’s look at some histograms.

20. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-20 Figure 2.6 The Distributions of  g 1,  g 2 and  g 3 for Several Sample Sizes With Normal Disturbances (1 of 2)

21. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-21 Figure 2.6 The Distributions of  g 1,  g 2 and  g 3 for Several Sample Sizes With Normal Disturbances (2 of 2)

22. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. Figure 2.6 The Distributions of  g 1,  g 2 and  g 3 for Several Sample Sizes With Normal Disturbances

23. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-23 Figure 2.7 The Distributions of  g 1,  g 2 and  g 3 for Several Sample Sizes With Skewed Disturbances (1 of 2)

24. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-24 Figure 2.7 The Distributions of  g 1,  g 2 and  g 3 for Several Sample Sizes With Skewed Disturbances (2 of 2)

25. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. Figure 2.7 The Distributions of  g 1,  g 2 and  g 3 for Several Sample Sizes With Skewed Disturbances

26. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-26 A Scorecard

27. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-27 Who’s Winning? Does one of the estimators seem to be doing better than the others? Does the relative performance of the estimators depend on the racetrack? Let’s vary: – The number of samples we draw – The size of each sample – The distribution of the error term – The X ’s – 

28. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-28 10,000 SAMPLES Estimator Mean Absolute Error  g 1 12.90  g 2 10.86  g 3 33.96  g 4 10.28 EstimatorMean Squared Error  g 1 261.42  g 2 186.04  g 3 1808.19  g 4 167.29 1000 SAMPLES Estimator Mean Absolute Error  g 1 12.20  g 2 10.24  g 3 34.30  g 4 9.78 Estimator Mean Squared Error  g 1 243.05  g 2 171.83  g 3 1834.35  g 4 156.81 What if We Draw Only 1,000 Samples? What do you conclude?

29. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-29 A Scorecard

30. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-30 Histogram of Errors of  g 1 1000 Samples 10,000 Samples Mean Error of  g 1 -54.2905 50.3459 0.079 Mean Error of  g 1 -57.290168.5268 0.077 Do We Need 10,000 Samples?

31. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-31 Mean Error of  g 2 -49.224653.3472 0.0787 Mean Error of  g 2 -48.1282 47.473 0.094 Histogram of Errors of  g 2 1000 Samples 10,000 Samples Do We Need 10,000 Samples? (cont.)

32. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-32 Mean Error of  g 3 -128.237132.589 0.075 Mean Error of  g 3 -160.287154.563 0.0728 Histogram of Mean Error of  g 3 1000 Samples10,000 Samples Do We Need 10,000 Samples? (cont.)

33. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-33 Mean Error of  g 4 -43.883246.2664 0.0713 Mean Error of  g 4 -43.198345.1747 0.089 Histogram of Mean Error of  g 4 1000 Samples10,000 Samples Do We Need 10,000 Samples? (cont.)

34. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-34 n = 6 Estimator Mean Error Std. Dev.  g 1 0.130 16.16  g 2 0.078 13.64  g 3 -0.271 42.52  g 4 0.037 12.93 n = 100 Estimator Mean Error Std. Dev.  g 1 0.014 3.92  g 2 0.001 3.32  g 3 -0.087 10.05  g 4 -0.007 3.17 Does the Sample Size of Each Dataset Matter? n = 6 vs. n = 100 What do you conclude?

35. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-35 Does n Matter? n = 6 vs. n = 100 n = 100 EstimatorMean Absolute Error  g 1 3.14  g 2 2.66  g 3 8.04  g 4 2.54 EstimatorMean Squared Error  g 1 15.39  g 2 11.06  g 3 101.10  g 4 10.08 n = 6 Estimator Mean Absolute Error  g 1 12.90  g 2 10.86  g 3 33.96  g 4 10.28 Estimator Mean Squared Error  g 1 261.42  g 2 186.04  g 3 808.19  g 4 167.29 What do you conclude?

36. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-36 A Scorecard

37. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-37 Does Normality of Errors Matter? Our Monte Carlo DGP has assumed  i ~ N( 0,500 2 ) What if we draw the error term from a different distribution? We will consider two distributions, one bimodal and one skewed.

38. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-38 Normal Errors vs. Bimodal Errors: n = 6  ~ Normal Estimator Mean Absolute Error  g 1 12.90  g 2 10.86  g 3 33.96  g 4 10.28 Estimator Mean Squared Error  g 1 261.42  g 2 186.04  g 3 1808.19  g 4 167.29  ~ Bimodally Estimator Mean Absolute Error  g 1 12.18  g 2 10.28  g 3 32.44  g 4 9.80 Estimator Mean Squared Error  g 1 258.75  g 2 184.91  g 3 1854.32  g 4 167.23

39. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-39  ~ Skewed Estimator Mean Absolute Error  g 1 12.64  g 2 10.23  g 3 33.12  g 4 10.13 Estimator Mean Squared Error  g 1 255.12  g 2 182.07  g 3 1792.70  g 4 164.53  ~ Normal Estimator Mean Absolute Error  g 1 12.90  g 2 10.86  g 3 33.96  g 4 10.28 Estimator Mean Squared Error  g 1 261.42  g 2 186.04  g 3 1808.19  g 4 167.29 Normal Errors vs. Skewed Errors: n = 6

40. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-40 A Scorecard

41. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-41 Error of  g 1,  ~ Normal -57.290168.5268.077 Error of  g 1,  ~ Bimodally -78.04880.9158.0728 Error of  g 1,  ~ Skewed -52.526.25.2116 Does Normality Matter? n = 6 Normal Bimodal Skewed

42. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-42 Error of  g 4,  ~ Normal -43.883246.2664.0713 Error of  g 4,  ~ Bimodally -61.60878.4728.0756 Error of  g 4,  ~ Skewed -4221.2094 Does Normality Matter? n = 6 (cont.) Normal Bimodal Skewed

43. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-43 Skewed Errors The histogram for the errors of the bimodal distribution of  looks approximately normal. The histogram for the errors of the skewed distribution of  does NOT look approximately normal.

44. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-44 Skewed Errors (cont.) What happens to the histogram of errors if we increase the sample size of each dataset from n = 6 to n = 100?

45. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-45 Error of  g 1,  ~ Skewed, n = 100 -16.814.7 0.102 Error of  g 1,  ~ Skewed, n = 6 -52.5 26.25 0.2116 Does n Matter for Normality When Disturbances Are Skewed? n = 6 n = 100

46. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-46 Error of  g 4,  ~ Skewed, n = 100 -12.611.76 0.1015 Error of  g 4,  ~ Skewed, n = 6 -4221 0.2094 Does n Matter for Normality When Disturbances Are Skewed? (cont.) n = 6 n = 100

47. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-47 Sample Size and Errors The error of our estimator starts to look more normal as we increase n. Later in the class, we will learn about the “Central Limit Theorem.” When we do hypothesis testing, we want to assume the error term is normally distributed. In large samples, this assumption is not so bad.

48. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-48 Does X Matter? So far, X has been either 10 or 20. What happens to our errors if we use a larger X ?

49. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-49 Small X vs. Large X X ’s 10 and 20 Estimator Mean Absolute Error  g 1 12.90  g 2 10.86  g 3 33.96  g 4 10.28 Estimator Mean Squared Error  g 1 261.42  g 2 186.04  g 3 1808.19  g 4 167.29 X ’s 40 and 50 Estimator Mean Absolute Error  g 1 3.69  g 2 3.62  g 3 33.96  g 4 3.59 Estimator Mean Squared Error  g 1 21.45  g 2 20.67  g 3 1808.19  g 4 20.41

50. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-50 Does  Matter? Estimator Mean Squared Error  = 2  g 1 261.42  g 2 186.04  g 3 1808.19  g 4 167.29  = -3  g 1 261.42  g 2 186.04  g 3 1808.19  g 4 167.29  = 20  g 1 261.42  g 2 186.04  g 3 1808.19  g 4 167.29

51. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-51 A Scorecard

52. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-52 Review What have we assumed? What is our Data Generating Process (DGP)? – E(  i ) = 0 – Var(  i ) =  2 – Cov(  i,  k ) = 0 i ≠ k –X 1, X 2, …, X n are fixed across samples GAUSS–MARKOV ASSUMPTIONS

53. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-53 Review (cont.)  g 4 dominates other three estimators  g 2 runs second n matters (bigger n, smaller errors) Biases seem “small” for  g 1,  g 2,  g 4 ? Normal  → normal  g ’s, any n Non-normal  → approximately normal  g ’s for large n → not so normal  g ’s for small n X matters (bigger X ’s, smaller errors)  itself doesn’t matter

54. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-54 Why Did We Do the Monte Carlos? 1.To emphasize estimators have sampling distributions. 2.To emphasize sampling properties (Mean Error, Mean Absolute Error, Mean Squared Error) are criteria for selecting estimators. 3.To emphasize that estimators’ traits depend on where the data come from.

55. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-55 What’s Next? We have seen how our estimators do in a number of specific settings that we have simulated computationally. Are these conclusions robust to a broader range of settings? We could run many, many more simulations. Can we find a more general method for exploring the properties of estimators?

56. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-56 What’s Next? (cont.) We “guessed” four estimators. One of them did very well (  g 4 ) Could there be other estimators that would do even better? Is there a more systematic method for finding good estimators?

57. Copyright © 2006 Pearson Addison-Wesley. All rights reserved. 3-57 What’s Next? (cont.) Computer simulations let you see what happens in the worlds you design. For broader findings, we turn to mathematics. Next time, we will review some basic mathematical tools. Then we will use these tools to derive general properties of estimators. We can also use these tools to derive the “best” estimators.