Systematic reviews of animal studies Malcolm Macleod

Why do systematic reviews of animal studies? To summarise existing data To help design clinical trials To understand where evidence is lacking To understand the limitations of animal models

Basic requirements Understanding the model – or knowing someone who does Clear a priori hypotheses A clear search strategy with inclusion and exclusion criteria Information and data management

The importance of hypotheses Observational research Susceptible to identification of statistically significant but biologically meaningless spurious associations If there isn’t a question for which you wish to know the answer, why are you doing it?

Examples “Hypothermia improves outcome in animal models of stroke” “The efficacy of hypothermia in animal models of stroke depends on the degree of cooling” “Evidence for the efficacy of NXY-059 is confounded by poor study quality” “The evidence for efficacy of stroke dugs in animals is confounded by publication bias”

Search strategy What are you looking for … –Controlled studies testing the effect of hypothermia in an animal model of focal cerebral ischaemia brought about by occlusion of a cerebral artery, where outcome was measured as infarct size or neurobehavioural score. Exclusion criteria –hypothermia was accomplished with use of a pharmacological agent that may also have an intrinsic neuroprotective property; –cooling was used to counteract (spontaneous) hyperthermia after MCAO; –brain cooling lasted <10 min, for example to counteract heating in models of photochemically induced cerebral infarction; –data were presented in a way not suitable for use in a meta-analysis (e.g. no information on group size, mean or variance); or if –mortality was the only outcome measure.

Example search strategy Studies of hypothermia in animal models of acute ischaemic stroke were identified from … (i)PubMed, EMBASE and BIOSIS up till December 31, 200> with the search strategy [[[ OR OR OR OR ] AND [ OR ]] OR ] AND [ OR ] (limit: animals) (ii)hand searching of abstracts of scientific meetings of the International Society of Cerebral Blood Flow and Metabolism, the International Stroke Conference [‘Joint (International) Conference on Stroke and Cerebral Circulation’ before 2000] and the European Stroke Conference during the same time period; (iii)reference lists of identified publications; and (iv)requests to authors of identified publications

Source selection Download search results to reference management system (eg RefMan) 2 investigators independently select sources against inclusion/ exclusion criteria May have to retrieve full text Discrepancies resolved by negotiation or in discussion with third investigator

Data extraction Publication meta-data

Data extraction Publication meta-data Outcome data

Data cleaning

Data analysis Standardised mean difference analysis Weighted mean difference analysis Normalised mean difference analysis Fixed Effects Random Effects Meta-regression

difference s.d.e.s. = difference/s.d. “sd units” SMD 0% unlesioned 100% lesioned d1 e.s. = d1/100% “percent improvement in outcome” NMD n1n2 e.s. = n1-n2 “real” units WMD 0 mm mm3

Testing significance Partitioning of heterogeneity: –Observed heterogeneity = within group heterogeneity + between group heterogeneity –Test against chi squared distribution with n-1 degrees of freedom Observational, so set high statistical bar

Data analysis

Demonstration

So, what can it do …? Describe a literature …. Randomisation Blinded Outcome Assessment Sample Size calculation Stroke36%29%3% MND31%20%<1% PD12%15%0% EAE2%11%0% Glioma14%0%

Estimate efficacy … Hypothermia –101 papers –277 experiments –3353 animals

Describe efficacy in subgroups…

Show potential sources of bias … NXY publications 29 experiments 408 animals Improved outcome by 44% (35-53%)

Illustrate publication bias … 991 publications

Provide evidence to change practice … 1.Animals 2.Sample size calculation 3.Inclusion and exclusion criteria 4.Randomization 5.Allocation concealment 6.Reporting of animals excluded from analysis 7.Blinded assessment of outcome 8.Reporting potential conflicts of interest and study funding

Further resources ers.htmhttp:// ers.htm The CAMARADES podcast … – ts/podcast.xmlhttp:// ts/podcast.xml