The Presentation of Statistics in Clinical and Health Psychology Research Jeremy Miles Department of Health Sciences Susanne Hempel Centre for Reviews and Dissemination

Introduction Statistics in clinical and health psychology –Appropriate statistics used –Statistics appropriately presented Graphical display Verbal presentation

Methodology Reviewed 2003 volumes (4 issues) of –British Journal of Clinical Psychology –British Journal of Health Psychology Looking for –Errors of statistical presentation / interpretation –Potential areas of improvement

Results BJCP: 29 papers reviewed BJHP: 31 papers reviewed –5 excluded (qualitative, narrative review) Wide range of problems identified Emerging themes –P-values –Inferential statistics –Effect Sizes –Reliability –Other Issues 2 papers with no issues

Statistical Significance

Confusing and controversial issue –Misunderstood by students, researchers, teachers, textbook authors (Broadly) two rival approaches to probability: –Fisher: report exact significance value –Neyman-Pearson: <0.05, or not These are incompatible(!) (Ignoring Bayes; ignoring meanings of probability)

A Bastardised Approach (From Gigerenzer, 1992) The two approaches are misunderstood, and combined –“We must report the exact p” –“We must present results as <0.xx” Recommended: –Exact probability values (e.g. Wilkinson, et al, 1999)

Results of p-value reporting BJCP: 8 out of 29 reported exact p-values –1 used strict N-P approach BJHP: 4 out of 26 reported exact p-values

More on P-Values 2 papers reported p < 0 (.00) –True values were , Several reported arbitrary cutoffs –<0.07, <0.02 Incorrect, but not deceptive

Misleading? Not using exact p-values sometimes appears fishy: –Exact p-values for all except where p = 0.049, reported as p < 0.05 –Gave p > 0.05 (p = 0.057), p < 0.05 (p = 0.048) –P < 0.01 when p = 1 * (others in same paper reported as p < 0.001) –p = , described as “< 0.01”, p = described as “<0.05”

Finally: Mistakes Good old errors –Very hard for readers and reviewers to spot, but still … –“F (1, 69) = 4.58, p < 0.001” No, p = –“F (1.76, ) = 3.026, p =.058.” No, p = –F = 4.02, (df not given, but are 2, 62), p = (information in table) No, p = 0.022

Inferential Statistics

Reporting Test Statistics Most people can’t interpret a test statistic –Even fewer are interested –Why report a test statistic exactly, and not the exact p? “[no] significant interaction of both variables, F (1,67) =.289.” No p-value given (it’s 0.59) –F without df No use at all (unless df can be worked out, but can be tricky or ambiguous)

Standard Errors Standard error is the standard deviation of the sampling distribution –Used to calculate t (and hence p-value) and CIs 95% CIs given by: Value depends on df –df = 5, t  /2 = 2.57 –df = 100, t  /2 = 1.98 Standard error has little use.

Graph shows mean +/- 1 SE. SE Mean is not showing anything useful

Graph shows mean +/- standard error. Data are repeated measures.

Confidence Intervals Generally recommended that confidence intervals are reported –Better idea of the likely value in the population –Not significant ≠ no effect Appropriate confidence intervals: –BJCP: 3 (of 29) –BJHP: 4 (of 26)

Inappropriate Confidence Intervals / Standard Errors Compare two groups –Appropriate standard error / confidence interval is of the difference, not of each group

Independent groups study: Significant difference? Yes. t = 2.7, df = 18, p = 0.016, difference 2.7, 95% CIs = 0.60, 4.80

Repeated measures study: Significant difference? Trick question. It’s the same graph, and I haven’t given you enough information t = 2.25, df = 9, p = Difference = 2.7, 95% CIs -0.02, 2.25

Effect Sizes

More statistically significant = larger, more important effect? –No Effect sizes describe the size of the effect –r, d,  2, R 2 YesNo BJCP416 BJHP710

Reliability Reporting

Small, but important Reliability is not a property of a test –It is a property of a test, in a population, at a particular time Reliability should always be evaluated, and presented AllSomeNone BJCP5414 BJHP6311

Stepwise Regression Almost never appropriate –Small differences in samples can lead to large differences in results 1 paper discusses differences between two stepwise regressions –Df are wrong (hence F, and p are also wrong) Use of stepwise regression: –BJCP: 1 –BJHP: 2 (one not described as stepwise)

A Collection of Smaller Issues

Distributional Assumptions Very few tests assume normal distribution of the variables –When sample sizes are at least moderate, normal distribution unimportant Kolmogorov-Smirnov test examines significant difference from normality –Not important difference from normality (Field?) 2 papers (BJCP) used the KS test –Non-parametric tests

Other Miscellany Mann-Whitney test described as comparing medians (it doesn’t necessarily) Principal components analysis described as exploratory factor analysis (it’s not) Expected values of chi-square test violated Arithmetical errors in chi-square test Correlation used as measure of agreement –We all know that it isn’t Inappropriate dichotomisation of continuous variables –Never necessary

Hall of Shame

Conclusions

Summary Picture isn’t rosy Errors are not limited to psychology –Garcia-Berthou and Alcaraz (2004) found errors in Nature and the British Medical Journal There are a lot of areas for improvement

Solutions? Short Term More statistical refereeing? –More guidelines for reviewers –More reviewers with expertise in statistics –BJCP and BJEP have statistical reviewers Rapid response? –Could be set up with the electronic journals –Work in other fields

Solutions? Long Term Statistical / methodological training? –Undergraduate? Postgraduate? CPD? Work more closely with statisticians? –Common in other fields – MSc in Medical Statistics is possible, MSc in Psychological Statistics is not

Final Thought Aaagggghhhhh! We just did a piece of qualitative research?