1. Study Quality and Publication Bias in Experimental Stroke

2. Presenter Disclosure Information
Malcolm Macleod PhD FRCP
Study Quality and Publication Bias in Experimental Stroke
FINANCIAL DISCLOSURE:
None
UNLABELED/UNAPPROVED USES DISCLOSURE:
None
CONFLICT OF INTEREST DISCLOSURE:
President, EuroHYP

3. 1026 interventions in experimental stroke
Tested in experiments
O’Collins et al, 2006

4. 1026 interventions in experimental stroke
603
Tested in focal ischaemia
O’Collins et al, 2006

5. 1026 interventions in experimental stroke
883
374
Effective in focal ischaemia
O’Collins et al, 2006

6. 1026 interventions in experimental stroke
883
550 97 18
Tested in clinical trial
O’Collins et al, 2006

7. 1026 interventions in experimental stroke
883 1 3 97 17
550
Effective in clinical trial
O’Collins et al, 2006

8. Get screenshot of paperM
10-60 M

9. What is translational medicine?
From: Institute of Translational Medicine, University of Texas Medical Branch

10. Animal data in stroke There are huge amounts of often confusing data
Systematic review can help to make sense of it
If you select extreme bits of the evidence you can “prove” either harm or substantial benefit
However, if you have a precise and highly significant overall effect, then it is probably real
Hypothermia: a systematic search identified 277 experiments in 3353 animals
Better
Worse
Van der Worp et al, 2007

12. Potential sources of bias in animal studies
Internal validity
External validity
Publication bias
Are the models we use good models?
Co-morbidities
Problem
Solution
Selection Bias
Randomisation
Performance Bias
Allocation Concealment
Detection Bias
Blinded outcome assessment
Attrition bias
Reporting drop-outs/ ITT analysis
Are animal experiments falsely positive?
Are clinical trials falsely negative?
Do animal studies not model human disease with sufficient fidelity to be useful?

13. Blinded outcome assessment
Internal Validity Randomisation and blinding in studies of hypothermia in experimental stroke
Blinded outcome assessment
Yes No
Efficacy è
47%
39%
Randomisation
37%
Infarct Volume
101 publications
222 experiments
3256 animals
Improved outcome by 43.5% ( )
Are animal experiments falsely positive?
Are clinical trials falsely negative?
Do animal studies not model human disease with sufficient fidelity to be useful?
Van der Worp et al, 2007

14. Internal Validity NXY-059
Infarct Volume
11 publications
29 experiments
408 animals
Improved outcome by 44% (35-53%)
Macleod et al, 2008

15. The File Drawer problem Publication bias
worse
better

16. External Validity Publication Bias for FK506
All outcomes
29 publications
109 experiments
1596 animals
Improved outcome by 31% (27-35%)
Precision
Worse
Better
Macleod et al, 2005

17. Publication bias – G-CSF
England T et al 2009

18. Publication bias in experimental stroke
Only 11/525 publications (2.2%) reported no significant treatment effects
Trim and Fill suggested ~16% (214/1573) of experiments remain unpublished
Best estimate of magnitude of problem
Observed efficacy 31.3% ( )
Adjusted efficacy 23.8% ( )

19. How much efficacy is left?
Publication bias
Randomisation
Co-morbidity bias

20. There are many other sources of bias (Sacket, 1979)
Tidying up bias
Repeated peaks bias
Significance bias
Bogus control bias
Hot stuff bias
Selective reporting bias
Excluding inconvenient outliers
Add a couple
Statistical v biological significance
Bogus control bias - dropping out of treatment group on grounds of severity

21. Current performance against key quality items
Randomisation
Blinded Outcome Assessment
Sample Size calculation
Stroke
36%
29% 3%

22. How does stroke compare?
Randomisation
Blinded Outcome Assessment
Sample Size calculation
Stroke
36%
29% 3%
MND
31%
20%
<1% AD
15%
25% 0% PD
12%
EAE 8%
Glioma
14%

23. Internal validity in PD models
Blinded outcome assessment
Composite quality
Rooke et al, submitted

24. Estimates of affinity at cannabinoid receptors
McPartland et al 2007

25. Systematic
Review
And
Meta-analysis
Animal
Studies
Clinical Trial

26. Conclusions We could, and should, do a lot better
Other diseases could do much much better
Like any other process, the processes of translational medicine can be studied, systematically and methodically