Preventing introduction of bias at the bench: from randomizing to experimental results meta-analysis Malcolm Macleod Centre for Clinical Brain Sciences,
Presentation on theme: "Improving the internal validity of experiments in focal ischaemia"— Presentation transcript:
1 Improving the internal validity of experiments in focal ischaemia Malcolm MacleodCentre for Clinical Brain Sciences, University of Edinburgh
2 1026 interventions in experimental stroke O’Collins et al Ann Neurol 2006
3 1026 interventions in experimental stroke 603O’Collins et al Ann Neurol 2006Tested in focal ischaemia
4 1026 interventions in experimental stroke 883374O’Collins et al Ann Neurol 2006Effective in focal ischaemia
5 1026 interventions in experimental stroke 8835509718O’Collins et al Ann Neurol 2006Tested in clinical trial
6 1026 interventions in experimental stroke 883139717550O’Collins et al Ann Neurol 2006Effective in clinical trial
7 What’s my problem?I want to improve the outcome for my patients with strokeTo get that, I want to conduct high quality clinical trials of interventions which have a reasonable chance of actually working in humansBut which of the remaining 929 interventions should I choose?
9 “…you will meet with several observations and experiments which, though communicated for true by candid authors or undistrusted eye-witnesses, or perhaps recommended by your own experience, may, upon further trial, disappoint your expectation, either not at all succeeding, or at least varying much from what you expected”Robert Boyle (1693)Concerning the Unsuccessfulness of Experiments
10 What is a Valid Experiment? One which describes some biological truth in the system being studiedInternal validity: the extent to which an experiment accurately describes what happened in that model systemCan be inferred by extent of reporting of measures to avoid common biases
11 What is a Valid Translational strategy? One which considers all available supporting animal dataOne which considers the likelihood of publication biasOne which tests a drug under circumstances similar to those in which efficacy has been demonstrated in animal models
12 Potential sources of bias in animal studies Internal validityProblemSolutionSelection BiasRandomisationPerformance BiasAllocation ConcealmentDetection BiasBlinded outcome assessmentAttrition biasReporting drop-outs/ ITT analysisFalse positive report biasAdequate sample sizesAfter Crossley et al, 2008, Wacholder, 2004Are animal experiments falsely positive?Are clinical trials falsely negative?Do animal studies not model human disease with sufficient fidelity to be useful?
13 Internal validity Dopamine Agonists in models of PD Ferguson et al, in draft
14 Internal validity Dopamine Agonists in models of PD Ferguson et al, in draft
15 Internal validity Dopamine Agonists in models of PD Ferguson et al, in draft
16 Blinded outcome assessment Internal Validity Randomisation and blinding in studies of hypothermia in experimental strokeRandomisationBlinded outcome assessmentYesNoEfficacy è47%39%47%37%Efficacy èvan der Worp et al Brain 2007Are animal experiments falsely positive?Are clinical trials falsely negative?Do animal studies not model human disease with sufficient fidelity to be useful?YesNo
17 Stem cells in experimental stroke Lees et al, in draft
18 Neurobehavioural score Internal Validity Randomisation, allocation concealment and blinding in studies of Stem cells in experimental strokeInfarct VolumeNeurobehavioural scoreLees et al, in draft
21 Internal Validity False positive reporting bias The positive predictive value of any test result depends onp (α)Power (1-ß)Pre-test probability of a positive resultafter Wacholder, 2004
22 Internal Validity False positive reporting bias The positive predictive value of any test result depends onp (α) (0.05)Power (1-ß) (0.30)Pre-test probability of a positive result (0.50)after Wacholder, 2004Positive predictive value = 0.67i.e. only 2 out of 3 statistically positive studies are truly positive
23 Chances that data from any given animal will be non-contributory assume simple two group experiment seeking 30% reduction in infarct volume, observed SD 40% of control infarct volumeNumber of animalsPower% animals wasted418.6%81.4%832.3%67.7%1656.4%43.6%3285.1%14.9%
25 External Validity Publication Bias for FK506 All outcomes29 publications109 experiments1596 animalsImproved outcome by 31% (27-35%)PrecisionMacleod et al, JCBFM 2005WorseBetter
26 External Validity Hypertension in studies of NXY-059 in experimental stroke Infarct volume:9 publications29 experiments408 animals44% (35-53%) improvementHypertension:7% of animal studies77% of patients in the (neutral) SAINT II studyMacleod et al, Stroke in press
27 External Validity Hypertension in studies of tPA in experimental stroke Infarct Volume:113 publications212 experiments3301 animalsImproved outcome by 24% (20-28)Hypertension:9% of animal studiesSpecifically exclusion criterion in (positive) NINDS studyEfficacy è25%-2%“Normal”ÝBPPerel et al BMJ 2007Comorbidity
28 Quality of Translation tPA and tirilazad Both appear to work in animalstPA works in humans but tirilazad doesn’tTime to treatment: tPA:Animals – median 90 minutesClinical trial – median 90 minutesTime to treatment: tirilazadAnimals – median 10 minutesClinical trial - >3 hrs for >75% of patientsSena et al, Stroke 2007; Perel et al BMJ 2007Are animal experiments falsely positive?Are clinical trials falsely negative?Do animal studies not model human disease with sufficient fidelity to be useful?
29 Chose your patients – tPA: Effect of time to treatment on efficacy Perel et al BMJ 2007; Lancet 200411% of all patients treatment initiated by 90 minutes33% of sub-3hr patients treatment initiated by 90 minutes
30 How much efficacy is left? Reported efficacyPublication biasRandomisationCo-morbidity bias26%20%5%32%
31 Summarising data from animal experiments StudiesSystematicReviewAndMeta-analysisClinical Trialhow powerful is the treatment?what is the quality of evidence?what is the range of evidence?is there evidence of a publication bias?What are the conditions of maximum efficacy?
32 Resources and acknowledgements Chief Scientist Office, ScotlandEmily Sena, Evie Ferguson, Jen Lees, Hanna VesterinenDavid Howells, Bart van der Worp, Uli Dirnagl, Philip Bath