1. 1 A Rorschach Test

2. S. Stanley Young, NISS Jessie Q. Xia, NISS Banff, Canada Dec 15, 2011 Variable Importance in Environmental Studies

3. Current Challenges in Statistical Learning 1. Statistical methods 2. Data quality 3. Invalid claims a. Multiple testing b. Multiple modeling c. Bias

4. Great Smog of '52 or Big Smoke 12,000 estimated deaths

5. Pope et al. 2009

6. 6 Studied Variables Life Expectancy life-table methods Per capita income (in thousands of $) Lung Cancer (Age standardized death rate) COPD (Age standardized death rate) High-school graduates (proportion of population) PM2.5 (μg/m3) Black population (proportion of population) Population (in hundreds of thousands) 5-Year in-migration (proportion of population) Hispanic population (proportion of population) Urban residence (proportion of population)

7. 7 First Analysis, Regression VariableSS FirstSS Last Income31.815.6 Lung Cancer22.4 5.1 COPD21.5 4.1 High School15.9 0.0 Population 9.4 5.2 PM2.5 9.4 5.8 Hispanic 4.3 2.4 Black 3.1 1.7 Urban 1.4 0.8 Migration 0.0 0.8

8. 8 Recursive Partitioning

9. 9 Variable Importance VariableRegressionRP Income0.33900.2108 COPD0.16210.1199 Lung Cancer0.17680.1467 PM2.50.07320.1302 High School0.09970.1066 %Black0.05370.0319 Pop Density0.04180.0793 %Hispanic0.01770.0136 Migration0.02280.0202 Urban0.01330.0105

10. 10 East versus West  Krewski et al. 2000 Health Effects In.  Enstrom 2005 Inhalation Toxicology  Bell et al. 2007 Env Health Pers  Smith et al. 2009 Inhalation Toxicology  Jerrett 2010 CARB workshop

11. 11 Fine particles and Mortality Pope co-author, 2000.

12. 12 Ozone and Mortality

13. 13 Variable Importance Regression Recursive Partitioning

14. 14 Longevity versus PM2.5 East : Gray West : Red

15. 15 Longevity versus Income

16. 16 Hans Rosling's 200 Countries, 200 Years http://www.youtube.com/watch?v=jbkSRLYSojo

17. 17 Summary to this point  Income is very important.  PM2.5 is 4th or 5th in importance.  PM2.5 is not important in West.  Pope knew or should have known the East/West heterogeneity.

18. 18 E1: Breakfast cereal and boy babies

19. 19 P-value plot

20. E2 : Peto, NEJM, statins and cancer Hypothesis: The (SEAS) trial has raised the hypothesis that adding ezetimibe to statin therapy might increase the incidence of cancer.

21. The claim fails to replicate. The relative risk is wide (95% CI, 1.13 to 2.12; 99% CI, 1.02 to 2.33; uncorrected P = 0.006 before any allowance is made for this being the hypothesis-generating result. NB: 16 x 0.006 = 0.098. SEAS New Studies

22. E3: A multiple testing and modeling train wreck 1. 275 chemicals 2. 32 medical outcomes 3. 10 demographic covariates 275 x 32 = 8800 x 2^10 = ~9 million A CDC “systems” train wreck in progress! JAMA

23. Author Interpretation There exists an increased risk of myocardial infarction in patients exposed to abacavir and didanosine within the preceding 6 months. E4 : Bias Example: Lancet DAD study

24. First drug use (Text, page 1422, and Table 3)

25. 25 E4 : BMJ versus JAMA (1) Conclusion: The risk of oesophageal cancer increased with 10 or more prescriptions for oral bisphosphonates and with prescriptions over about a five year period. BMJ 2010; 341:c4444

26. 26 E4: BMJ versus JAMA (2) Conclusion: Oral bisphosphonates was not significantly associated with incident of esophageal or gastric cancer. JAMA 2010; 304(6): 657‐663

27. 27 A Rorschach Test With large, complex data sets, there is enough flexibility to get what you want/need.

28. 28 Consumer Wishes  Honest science  Valid claims  Claims in context  + and – of data and methods

29. 29 What do we have? (Deming)  A systems failure.  Essentially no process control.  Journals operating by “quality by inspection”.  Workers are happy.  Management failure.

30. 30 What to do?  Funding agencies need to require data access on publication.  Editors need to give up quality by inspection require split sample strategy require number of claims at issue.

31. 31 Statisticians Eventually society will figure it out; Scientific claims are (most) often wrong. Essentially all claims are supported by statistics. Society will ask, “Where were the statisticians?”

32. 32 Contact  Stan Young  www.niss.org  young@niss.org young@niss.org  919 685 9328