1. Automated stress tests for econometric models Presentation to Stata 2008 Fall North American Users Group Meeting by Roy J. Epstein, PhD Adjunct Professor of Finance, Boston College November 14, 2008

2. Roy J. Epstein PhD 2 Need for Greater Quality Control and Transparency in Applied Econometrics  Econometric models now part of mainstream in policy analysis, legal proceedings, and other advocacy situations  But assessing reliability of estimation results often difficult Lack of transparency  Risk of bad decisions  Enhanced regression software—a virtual referee—is key to improving the practice of econometrics

3. Roy J. Epstein PhD 3 Regression Software as Virtual Referee for Econometric “Best Practices”  Journal referee process and professional dialogues routinely probe effects of different: Explanatory variables Data Statistical procedures  Similar quality control process can/should be built into regression software Promotes neutral and transparent assessment of reliability

4. Roy J. Epstein PhD 4 Automating Regression Stress Tests  Software can anticipate reasonable perturbations of a model  Program then can assess sensitivity of original regression to the perturbations  Can be standard part of regression output Much as we now expect to see t-statistics

5. Roy J. Epstein PhD 5 Extending Stata for Stress Tests  Macro facility allows design of highly flexible testing strategy  Can implement as stand-alone post-estimation routine  Probably more advantageous to incorporate as part of standard output  Can reflect ongoing professional discussion of “best practice” stress tests in econometrics

6. Roy J. Epstein PhD 6 A Taxonomy of Stress Tests  Algebraic sign of regression coefficient  Magnitude  Statistical significance  Panel models: Validity of pooling assumptions Cross-sectional error correlation  Autocorrelation  Temporal aggregation  Lagged variables  Dropping variables  Outliers  Functional form  Weighting

7. Roy J. Epstein PhD 7 Stata Command (Prototype) stresstest price focuscoef cost realgdp i.firm i.density i.month, timeseries(year month) panel(firm density) w(pounds);  Based on “regress” command syntax  Parameters identify time, panel, and weighting variables as appropriate  Implemented as conventional Stata 10 ado file

8. Case Studies

9. Roy J. Epstein PhD 9 Conwood v. US Tobacco  Econometric analysis of tobacco sales Logic of plaintiff model was such that the more negative the estimated slope, the greater the damages.  $1 billion claimed damages, after trebling  Sustained after review by Supreme Court  Timely use of stress tests probably would have vindicated defendant

10. Roy J. Epstein PhD 10 The Conwood Data and Regression Line Washington, DC

11. Roy J. Epstein PhD 11 Stress Test Easily Identifies Effect of Outlier Negative estimated slope, $1 billion damages Positive estimated slope, (excludes Washington, DC) Zero damages

12. Roy J. Epstein PhD 12 Confidential Antitrust Case  Alleged price-fixing conspiracy  Panel model of average prices (by firm, product, month)  Nearly 100 control variables  Claimed damages of $330 million based on “statistically significant” coefficient of dummy variable for conspiracy period

13. Roy J. Epstein PhD 13 Why the Virtual Referee Would Have Helped  Plaintiff expert: Did not test validity of pooling data for 4 different firms Did not report results from aggregated model but admitted having estimated it Ignored serial correlation in the residuals Did not report effect of excluding control variables with dubious justification  Stress tests quickly cut through complexity of model to identify serious reliability problems

14. Roy J. Epstein PhD 14 Stress Tests Reveal High Sensitivity to Pooling Data from Different Firms

15. Roy J. Epstein PhD 15 Stress Tests Reveal High Sensitivity to Aggregating Across the Panel

16. Roy J. Epstein PhD 16 Stress Tests Reveal High Sensitivity to Autocorrelation

17. Roy J. Epstein PhD 17 Stress Tests Reveal High Sensitivity to Presence of Certain Variables

18. Roy J. Epstein PhD 18 Summary  Software designers have important role in improving the practice of econometrics  Regression results not reliable until sensitivities identified and explained  Software can and should automatically invoke appropriate best practice stress tests  Software will act as virtual referee

19. For Further Information… Roy J. Epstein, PhD Expert economic analysis for complex litigation 1280 Massachusetts Ave., 2nd Fl. Cambridge, MA 02138 rje@royepstein.com (617) 489-3818 Adjunct Professor of Finance, Boston College