Association between risk-of-bias assessments and results of randomized trials in Cochrane reviews: the ROBES study Jelena Savović1, Becky Turner2, David Mawdsley1, Hayley Jones1, Rebecca Beynon1, Julian Higgins1, Jonathan Sterne1 1University of Bristol, 2MRC Clinical Trials Unit at UCL 14th May 2018

Background Methodological flaws in a trial (e.g. inadequate allocation concealment) can lead to bias. The true bias affecting any individual trial is unknown. BUT Using collections of meta-analysis data, we can quantify the average bias associated with particular flaws. This helps with interpretation of trial results, especially when flawed trials are included in meta-analyses.

Cochrane risk of bias information Since January 2008, Cochrane authors have used a ‘risk of bias’ tool to assess potential biases in trials within reviews. Judgements of high/unclear/low risk of bias for: Sequence generation Allocation concealment Blinding of participants, personnel, outcome assessors Incomplete outcome data Selective outcome reporting

ROBES database Source of data: 1399 systematic reviews with activated risk-of-bias tables from issue 4, 2011 of the Cochrane Library Data extraction: MAs with binary outcomes & 5+ included trials Fully completed risk-of-bias tool Final data set: 228 meta-analyses, including 2243 trials Outcomes categorised: mortality other objective subjective

Meta-epidemiology model How do study characteristics affect treatment effect estimates in RCTs? Compare treatment effects within each meta-analysis with high/unclear vs. low risk of bias (e.g. for blinding) Calculate the ratio of odds ratios (for high/unclear vs. low) in each meta-analysis Fit a “meta-meta-analysis” model Estimate the average ratio of odds ratios associated with high/unclear, across meta-analyses

Meta-epidemiological studies: meta-meta-analytic method Pooled ORs calculated for trials with and without methodological flaw within each meta-analysis, and difference calculated as Ratio of ORs (ROR): ROR = OR with flaw / OR without flaw ROR ROR (95% CI) ROR Flawed studies show more benefit Flawed studies show less benefit ROR

Meta-epidemiological studies: meta-meta-analytic method Pooled ORs calculated for trials with and without methodological flaw within each meta-analysis, and difference calculated as Ratio of ORs (ROR): ROR = OR with flaw / OR without flaw ROR ROR (95% CI) ROR Flawed studies show more benefit Flawed studies show less benefit The RORs are then meta-analysed to estimate overall ROR across all meta-analyses and the between MA variability in bias ROR

Analyses performed We examined associations between high/unclear (vs. low) risk-of-bias judgements and treatment effect size for: sequence generation allocation concealment blinding incomplete outcome data We estimated: average ratio of odds ratios increase in between-trial heterogeneity SD (kappa) variability in bias between meta-analyses (phi)

Results: Sequence generation Outcome (Contributing meta-analyses, contributing trials) All outcomes (189, 2158) Mortality (34, 363) Other objective (47, 523) Subjective (108, 1272) ROR (95% Cr-I) 0.91 (0.86, 0.98) 0.84 (0.71, 1.01) 0.99 (0.87, 1.16) 0.90 (0.83, 0.98) 0.09 0.13 0.10 0.08 0.14 k (within) f (between) .6 .7 .8 .9 1 1.1 1.2 1.3 0.25 High or unclear risk of bias for random sequence generation (versus low risk) – Univariable analyses Ratio of odds ratios Outcome (Contributing meta-analyses, contributing trials) All outcomes (189, 2158) Mortality (34, 363) Other objective (47, 523) Subjective (108, 1272) ROR (95% Cr-I) 0.95 (0.88, 1.03) 0.92 (0.75, 1.18) 1.06 (0.90, 1.27) 0.94 (0.84, 1.03) 0.08 0.14 0.11 0.20 k (within) f (between) .6 .7 .8 .9 1 1.1 1.2 1.3 0.25 High or unclear risk of bias for random sequence generation (versus low risk) – Multivariable analyses Ratio of odds ratios

Results: Allocation concealment Outcome (Contributing meta-analyses, contributing trials) All outcomes (188, 2121) Mortality (35, 358) Other objective (49, 524) Subjective (104, 1239) ROR (95% Cr-I) 0.92 (0.86, 0.98) 0.84 (0.71, 1.01) 0.96 (0.86, 1.07) 0.91 (0.83, 0.99) 0.05 0.07 0.04 0.08 0.12 0.06 k (within) f (between) .6 .7 .8 .9 1 1.1 1.2 1.3 0.25 High or unclear risk of bias for allocation concealment (versus low risk) – Univariable analyses Ratio of odds ratios Outcome (Contributing meta-analyses, contributing trials) All outcomes (188, 2121) Mortality (35, 358) Other objective (49, 524) Subjective (104, 1239) ROR (95% Cr-I) 0.96 (0.88, 1.03) 0.92 (0.74, 1.13) 0.94 (0.81, 1.08) 0.95 (0.86, 1.07) 0.06 0.11 0.07 0.10 0.15 0.09 0.08 k (within) f (between) .6 .7 .8 .9 1 1.1 1.2 1.3 0.25 High or unclear risk of bias for allocation concealment (versus low risk) – Multivariable analyses Ratio of odds ratios

Results: Blinding Outcome (Contributing meta-analyses, contributing trials) All outcomes (144, 1678) Mortality (31, 327) Other objective (32, 334) Subjective (81, 1017) ROR (95% Cr-I) 0.87 (0.80, 0.93) 0.83 (0.72, 0.97) 0.94 (0.81, 1.10) 0.83 (0.73, 0.93) 0.10 0.06 0.22 0.12 0.19 k (within) f (between) .6 .7 .8 .9 1 1.1 1.2 1.3 0.25 High or unclear risk of bias for blinding (versus low risk) – Univariable analyses Outcome (Contributing meta-analyses, contributing trials) All outcomes (144, 1678) Mortality (31, 327) Other objective (32, 334) Subjective (81, 1017) ROR (95% Cr-I) 0.88 (0.81, 0.94) 0.87 (0.73, 1.03) 0.95 (0.79, 1.12) 0.84 (0.75, 0.95) 0.10 0.09 0.17 0.12 0.19 k (within) f (between) .6 .7 .8 .9 1 1.1 1.2 1.3 0.25 High or unclear risk of bias for blinding (versus low risk) – Multivariable analyses Ratio of odds ratios

Results: Incomplete outcome data Outcome (Contributing meta-analyses, contributing trials) All outcomes (167, 1956) Mortality (29, 303) Other objective (43, 471) Subjective (95, 1182) ROR (95% Cr-I) 0.98 (0.92, 1.05) 0.92 (0.79, 1.08) 1.03 (0.90, 1.19) 0.97 (0.88, 1.07) 0.05 0.08 0.07 0.06 0.10 k (within) f (between) .6 .7 .8 .9 1 1.1 1.2 1.3 0.25 High or unclear risk of bias for incomplete outcome data (versus low risk) – Univariable analyses Outcome (Contributing meta-analyses, contributing trials) All outcomes (167, 1956) Mortality (29, 303) Other objective (43, 471) Subjective (95, 1182) ROR (95% Cr-I) 1.01 (0.94, 1.09) 0.99 (0.82, 1.18) 1.04 (0.90, 1.21) 1.00 (0.90, 1.12) 0.07 0.11 0.10 0.09 k (within) f (between) .6 .7 .8 .9 1 1.1 1.2 1.3 0.25 Ratio of odds ratios High or unclear risk of bias for incomplete outcome data (versus low risk) – Multivariable analyses

How does this compare to previous work? BRANDO study

Conclusions For blinding, high/unclear risk-of-bias judgements associated with 13% exaggeration of treatment effect. For sequence generation, high/unclear judgements associated with 9% exaggeration, but this reduced after adjustment. (Similar for allocation concealment.) For incomplete outcome data, no association found. No evidence that average bias differed across outcome types. Variability of treatment effects was higher in trials that lacked blinding and had subjective outcomes.

Discussion The ROBES study relied on risk-of-bias assessments made by Cochrane review authors. “High risk” and “unclear risk” judgements were considered together in the analyses, for consistency with previous empirical studies of bias. Overall findings similar to the BRANDO study, but BRANDO found evidence of differences in average bias across outcome types.

Reference Savović et al. Association between risk-of-bias assessment and results of randomized trials in Cochrane reviews: the ROBES meta-epidemiological study. American Journal of Epidemiology 2018; 187(5): 1113-1122. Funding Medical Research Council Fellowship (G0701659/1) Medical Research Council grants (MR/K014587/1, MC-U105260558 and MC-UU12023/24) National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care West (NIHR CLAHRC West)