1. Statistical Pitfalls Stephen Senn (c) Stephen Senn1Statistical Pitfalls

2. (c) Stephen Senn2 Four to watch out for Regression to the mean Simpson’s paradox Invalid inversion –The error of the transposed conditional Selective sampling Statistical Pitfalls

3. There are Three Kinds of Statistician Those who can count Those who can’t (c) Stephen Senn3Statistical Pitfalls

4. (c) Stephen Senn4 Regression to the mean Powerful phenomenon causing apparent change over time If individuals are selected for treatment because extreme when measured again on average they will be closer to the mean Discovered by Francis Galton (1822-1911) Statistical Pitfalls

5. (c) Stephen Senn5Statistical Pitfalls

6. (c) Stephen Senn6Statistical Pitfalls

7. (c) Stephen Senn7Statistical Pitfalls

8. (c) Stephen Senn8 Consequences Spontaneous improvement over time easy to produce Is a consequence of the way data are studied not the phenomenon being studied Always compare to the control group Quite possibly the explanation of the placebo effect Statistical Pitfalls

9. (c) Stephen Senn9 Examples Remedial treatment for accident blackspots –Sacrificing a chicken would work The placebo effect –How do you know that nothing at all would not work Horace Secrist’s discovery of the decline in profitability of the most profitable US firms –Harold Hotelling put him right And he originally studied journalism! Statistical Pitfalls

10. 10 How to Prove Spells Against Rain Work Wait for a very rainy day Say these words –“Rain rain go away, come again another day” By some mystical pluvioincantory process the weather will be drier at some stage in the future than it was on the day the spell was uttered Conclusion –The spell works against rain (c) Stephen SennStatistical Pitfalls

11. 11 Simpson’s Paradox Graduate Admissions to Berkeley 1973 A Bias against Women? Per cent admission by sex (c) Stephen SennStatistical Pitfalls Simpson, E. H. (1951). "The interpretation of interaction in contingency tables." Journal of the Royal Statistical Society, Series B 13: 238-241.

12. 12 Graduate Admissions to Berkeley 1973 The Bias Disappears? (c) Stephen SennStatistical Pitfalls

13. 13 Simpson’s Paradox The Berkeley Data Women were more likely to target arts faculty departments These department had lower admission rates Hence, admission rates for women were lower overall Despite fact that department by department they were not (c) Stephen SennStatistical Pitfalls

14. The Origin of Babies Rival theories –Mulberry bushes –Doctors’ bags –Storks We shall begin a statistical investigation of the last of these (c) Stephen Senn14Statistical Pitfalls

15. Source Matthews, 2000 Teaching Statistics, 22, 36- 28 (c) Stephen Senn15Statistical Pitfalls

16. Storks and Babies Larger countries tend to have more storks Larger countries tend to have more babies Hence the size of the country may be a third factor responsible for the correlation (c) Stephen Senn16Statistical Pitfalls

17. (c) Stephen Senn17Statistical Pitfalls

18. Correlations: S per A, B Rate Pearson correlation of S per A and B Rate = 0.161 P-Value = 0.536 (c) Stephen Senn18Statistical Pitfalls

19. 19 Morals Watch out for confounding variables Where possible (it is not always possible) design the study so that these are accounted for –For example in experiment have controls and randomise Take care in jumping to conclusions (c) Stephen SennStatistical Pitfalls

20. (c) Stephen Senn20 Invalid Inversion Most women do not get breast cancer However most breast cancer victims are women You cannot reverse probability statements It is not generally true, for example, that the probability of the evidence given innocence is the same as the probability of innocence given evidence Statistical Pitfalls

21. 21 OJ Simpson’s Paradox “Let me begin with a refrain constantly repeated by attorney Alan Dershowitz during the trial. He declared that since fewer than 1 in a 1000 women who are abused by their mates go on to be killed by them, the spousal abuse in the Simpsons' marriage was irrelevant to the case.” John Allen Paulos “the issue is whether a history of spousal abuse is necessarily a prelude to murder”. Alan Dershowitz (c) Stephen SennStatistical Pitfalls

22. 22 That Calculation About 2000 women are murdered annually by a current or former mate in the USA About 2 million spousal assaults occur annually. The ratio of one to the other is one in thousand. Therefore a woman in an abusive relationship has only a 1 in 1000 chance of being murdered by their mate each year (c) Stephen SennStatistical Pitfalls

23. 23 Mariage and Murder ‘Dershowitz had stated in the L.A. Times article that “the issue is whether a history of spousal abuse is necessarily a prelude to murder”. He’s Wrong - The issue is not whether abuse leads to murder but whether a history of abuse helps identify the murderer.’ Kevin Hayes, University of Limerick http://www.ul.ie/elements/Issue5/Oj.htm http://www.maths.ul.ie/KH.htm The following data taken from Haye’s website show data on women murdered in the USA in 1992 (c) Stephen SennStatistical Pitfalls

24. 24 ‘Marriage’ and Murder (c) Stephen Senn Statistical Pitfalls

25. 25 Abuse and Murder (c) Stephen SennStatistical Pitfalls

26. 26 OJ Simpson Revisited “Given certain reasonable factual assumptions, it can be easily shown using probability theory that if a man abuses his wife and she is later murdered, the batterer is the murderer more than 80% of the time. (A nice demonstration of this by Jon Merz and Jonathan Caulkins appeared in a recent issue of Chance magazine.) Thus, without any further evidence, there was mathematical warrant for immediate police suspicion of Mr. Simpson.” John Allen Paulos (c) Stephen SennStatistical Pitfalls

27. Selective Sampling We often make an assumption that the data arrive without ‘side’ This is not necessarily true One may have to think carefully about the data-generation process (c) Stephen Senn27Statistical Pitfalls

28. 28 Abraham Wald (1902-1950) Rumanian/Hungarian/American, mathematical statistician Inventor of decision theory –brilliant and seminal paper of 1939 Also innovator for sequential analysis Died in a plane crash in India Ironically, was employed by US military to advise on plane safety in World War II (c) Stephen SennStatistical Pitfalls

29. 29 Wald’s Problem Returning planes were examined to see where they had been hit Engines were rarely hit Fuel tanks very often Extra armour could be placed but not everywhere Where should it be placed? (c) Stephen SennStatistical Pitfalls

30. 30 Wald The Military and The Aircraft The US Military decided to reinforce the fuel tanks That was where the most shots were They argued that therefore the fuel tanks needed protection (c) Stephen SennStatistical Pitfalls

31. 31 Wald and the Aircraft Wald argued that the pattern of shots received ought to be random The fact that it was not, indicated that this sample was not random If the shots hit the fuel tank, the plane returned safely If it hit the engine, it did not Solution: reinforce the engines not the fuel tanks! (c) Stephen SennStatistical Pitfalls

32. Moral Think carefully about the process that has led to the data in hand There may be subtle effects at work Don’t jump to conclusions (c) Stephen Senn32Statistical Pitfalls

33. Question Studies have shown that if popes from the 13 th to 19 th century are compared to artists of the same era, they died older It has been claimed that this shows the effect of status in society on longevity Is there a snag? (c) Stephen SennStatistical Pitfalls33 Carrieri, M. P. and D. Serraino (2005). "Longevity of popes and artists between the 13th and the 19th century.“ Int J Epidemiol 34(6): 1435-1436.

34. Key Questions You should always ask What happened to the controls? Are there any hidden confounders? Has the probability statement been framed the right way round? Is there a bias in the way the data are collected?