1. Association between risk-of-bias assessments and results of randomized trials in Cochrane reviews: the ROBES study
Jelena Savović1, Becky Turner2, David Mawdsley1, Hayley Jones1, Rebecca Beynon1, Julian Higgins1, Jonathan Sterne1
1University of Bristol, 2MRC Clinical Trials Unit at UCL
14th May 2018

2. Background
Methodological flaws in a trial (e.g. inadequate allocation concealment) can lead to bias.
The true bias affecting any individual trial is unknown.
BUT
Using collections of meta-analysis data, we can quantify the average bias associated with particular flaws.
This helps with interpretation of trial results, especially when flawed trials are included in meta-analyses.

3. Cochrane risk of bias information
Since January 2008, Cochrane authors have used a ‘risk of bias’ tool to assess potential biases in trials within reviews.
Judgements of high/unclear/low risk of bias for:
Sequence generation
Allocation concealment
Blinding of participants, personnel, outcome assessors
Incomplete outcome data
Selective outcome reporting

4. ROBES database Source of data:
1399 systematic reviews with activated risk-of-bias tables from issue 4, 2011 of the Cochrane Library
Data extraction:
MAs with binary outcomes & 5+ included trials
Fully completed risk-of-bias tool
Final data set:
228 meta-analyses, including 2243 trials
Outcomes categorised:
mortality
other objective
subjective

5. Meta-epidemiology model
How do study characteristics affect treatment effect estimates in RCTs?
Compare treatment effects within each meta-analysis with high/unclear vs. low risk of bias (e.g. for blinding)
Calculate the ratio of odds ratios (for high/unclear vs. low) in each meta-analysis
Fit a “meta-meta-analysis” model
Estimate the average ratio of odds ratios associated with high/unclear, across meta-analyses

6. Meta-epidemiological studies: meta-meta-analytic method
Pooled ORs calculated for trials with and without methodological flaw within each meta-analysis, and difference calculated as Ratio of ORs (ROR):
ROR = OR with flaw / OR without flaw
ROR
ROR (95% CI)
ROR
Flawed studies show more benefit Flawed studies show less benefit
ROR

7. Meta-epidemiological studies: meta-meta-analytic method
Pooled ORs calculated for trials with and without methodological flaw within each meta-analysis, and difference calculated as Ratio of ORs (ROR):
ROR = OR with flaw / OR without flaw
ROR
ROR (95% CI)
ROR
Flawed studies show more benefit Flawed studies show less benefit
The RORs are then meta-analysed to estimate overall ROR across all meta-analyses and the between MA variability in bias
ROR

8. Analyses performed
We examined associations between high/unclear (vs. low) risk-of-bias judgements and treatment effect size for:
sequence generation
allocation concealment
blinding
incomplete outcome data
We estimated:
average ratio of odds ratios
increase in between-trial heterogeneity SD (kappa)
variability in bias between meta-analyses (phi)

9. Results: Sequence generation
Outcome (Contributing meta-analyses, contributing trials)
All outcomes (189, 2158)
Mortality (34, 363)
Other objective (47, 523)
Subjective (108, 1272)
ROR (95% Cr-I)
0.91 (0.86, 0.98)
0.84 (0.71, 1.01)
0.99 (0.87, 1.16)
0.90 (0.83, 0.98)
0.09
0.13
0.10
0.08
0.14 k
(within) f
(between) .6 .7 .8 .9 1
1.1
1.2
1.3
0.25
High or unclear risk of bias for random sequence generation (versus low risk) – Univariable analyses
Ratio of odds ratios
Outcome (Contributing meta-analyses, contributing trials)
All outcomes (189, 2158)
Mortality (34, 363)
Other objective (47, 523)
Subjective (108, 1272)
ROR (95% Cr-I)
0.95 (0.88, 1.03)
0.92 (0.75, 1.18)
1.06 (0.90, 1.27)
0.94 (0.84, 1.03)
0.08
0.14
0.11
0.20 k
(within) f
(between) .6 .7 .8 .9 1
1.1
1.2
1.3
0.25
High or unclear risk of bias for random sequence generation (versus low risk) – Multivariable analyses
Ratio of odds ratios

10. Results: Allocation concealment
Outcome (Contributing meta-analyses, contributing trials)
All outcomes (188, 2121)
Mortality (35, 358)
Other objective (49, 524)
Subjective (104, 1239)
ROR (95% Cr-I)
0.92 (0.86, 0.98)
0.84 (0.71, 1.01)
0.96 (0.86, 1.07)
0.91 (0.83, 0.99)
0.05
0.07
0.04
0.08
0.12
0.06 k
(within) f
(between) .6 .7 .8 .9 1
1.1
1.2
1.3
0.25
High or unclear risk of bias for allocation concealment (versus low risk) – Univariable analyses
Ratio of odds ratios
Outcome (Contributing meta-analyses, contributing trials)
All outcomes (188, 2121)
Mortality (35, 358)
Other objective (49, 524)
Subjective (104, 1239)
ROR (95% Cr-I)
0.96 (0.88, 1.03)
0.92 (0.74, 1.13)
0.94 (0.81, 1.08)
0.95 (0.86, 1.07)
0.06
0.11
0.07
0.10
0.15
0.09
0.08 k
(within) f
(between) .6 .7 .8 .9 1
1.1
1.2
1.3
0.25
High or unclear risk of bias for allocation concealment (versus low risk) – Multivariable analyses
Ratio of odds ratios

11. Results: Blinding
Outcome (Contributing meta-analyses, contributing trials)
All outcomes (144, 1678)
Mortality (31, 327)
Other objective (32, 334)
Subjective (81, 1017)
ROR (95% Cr-I)
0.87 (0.80, 0.93)
0.83 (0.72, 0.97)
0.94 (0.81, 1.10)
0.83 (0.73, 0.93)
0.10
0.06
0.22
0.12
0.19 k
(within) f
(between) .6 .7 .8 .9 1
1.1
1.2
1.3
0.25
High or unclear risk of bias for blinding (versus low risk) – Univariable analyses
Outcome (Contributing meta-analyses, contributing trials)
All outcomes (144, 1678)
Mortality (31, 327)
Other objective (32, 334)
Subjective (81, 1017)
ROR (95% Cr-I)
0.88 (0.81, 0.94)
0.87 (0.73, 1.03)
0.95 (0.79, 1.12)
0.84 (0.75, 0.95)
0.10
0.09
0.17
0.12
0.19 k
(within) f
(between) .6 .7 .8 .9 1
1.1
1.2
1.3
0.25
High or unclear risk of bias for blinding (versus low risk) – Multivariable analyses
Ratio of odds ratios

12. Results: Incomplete outcome data
Outcome (Contributing meta-analyses, contributing trials)
All outcomes (167, 1956)
Mortality (29, 303)
Other objective (43, 471)
Subjective (95, 1182)
ROR (95% Cr-I)
0.98 (0.92, 1.05)
0.92 (0.79, 1.08)
1.03 (0.90, 1.19)
0.97 (0.88, 1.07)
0.05
0.08
0.07
0.06
0.10 k
(within) f
(between) .6 .7 .8 .9 1
1.1
1.2
1.3
0.25
High or unclear risk of bias for incomplete outcome data (versus low risk) – Univariable analyses
Outcome (Contributing meta-analyses, contributing trials)
All outcomes (167, 1956)
Mortality (29, 303)
Other objective (43, 471)
Subjective (95, 1182)
ROR (95% Cr-I)
1.01 (0.94, 1.09)
0.99 (0.82, 1.18)
1.04 (0.90, 1.21)
1.00 (0.90, 1.12)
0.07
0.11
0.10
0.09 k
(within) f
(between) .6 .7 .8 .9 1
1.1
1.2
1.3
0.25
Ratio of odds ratios
High or unclear risk of bias for incomplete outcome data (versus low risk) – Multivariable analyses

13. How does this compare to previous work?
BRANDO study

14. Conclusions
For blinding, high/unclear risk-of-bias judgements associated with 13% exaggeration of treatment effect.
For sequence generation, high/unclear judgements associated with 9% exaggeration, but this reduced after adjustment. (Similar for allocation concealment.)
For incomplete outcome data, no association found.
No evidence that average bias differed across outcome types.
Variability of treatment effects was higher in trials that lacked blinding and had subjective outcomes.

15. Discussion
The ROBES study relied on risk-of-bias assessments made by Cochrane review authors.
“High risk” and “unclear risk” judgements were considered together in the analyses, for consistency with previous empirical studies of bias.
Overall findings similar to the BRANDO study, but BRANDO found evidence of differences in average bias across outcome types.

16. Reference
Savović et al. Association between risk-of-bias assessment and results of randomized trials in Cochrane reviews: the ROBES meta-epidemiological study. American Journal of Epidemiology 2018; 187(5):
Funding
Medical Research Council Fellowship (G /1)
Medical Research Council grants (MR/K014587/1, MC-U and MC-UU12023/24)
National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care West (NIHR CLAHRC West)