1. Estimation and Adjustment of Bias in Randomised Evidence Using Mixed Treatment Comparison Meta-analysis Sofia Dias, NJ Welton, AE Ades with Valeria Marinho, Georgia Salanti, Julian Higgins Avon RSS, May 2010 Department of Community Based Medicine

2. 2 Overview Motivation Treatment networks and MTC Adjusting for Bias in Mixed Treatment Comparisons Meta-analysis (MTC) The MTC model Example: Fluoride dataset Probability of bias model Results and Conclusions

3. 3 Mixed Treatment Comparisons Often more than two treatments for a given condition Network of trials comparing different interventions for a condition Direct and indirect evidence available on treatment effects Because of the network structure, there is enough information to estimate and adjust for bias within the network For bias adjustment, there is no need to rely on exchangeability assumption between meta-analyses in different fields

4. 4 Example: The Fluoride Data 6 different interventions for preventing dental caries in children and adolescents 1.No Treatment 2.Placebo 3.Fluoride in Toothpaste 4.Fluoride in Rinse 5.Fluoride in Gel 6.Fluoride in Varnish From 6 Cochrane Reviews* Active Treatments *Marinho et al., 2002; 2003; 2004 (Cochrane Library)

5. 5 Network and Number of trials Pl NT G R V T 69 1 31 13 3 1 4 9 4 6 3 1 4 1 130 trials eight 3-arm trials one 4-arm trial 150 pairwise comparisons

6. 6 Introduction to MTC 1. Six treatments 1,2,3,4,5,6 2. Take treatment 1 (No Treatment) as reference 3. Then the treatment effects d 1k of all other treatments relative to 1 are the basic parameters 4. Given them priors: d 1,2, d 1,3,…, d 1,6 ~ N(0,100 2 )

7. 7 Functional parameters in MTC The remaining contrasts are functional parameters d 2,3 = d 1,3 – d 1,2 d 2,4 = d 1,4 – d 1,2 … d 4,6 = d 1,6 – d 1,4 d 5,6 = d 1,6 – d 1,5 Any information on functional parameters tells us indirectly about basic parameters Either FE or RE model satisfying these conditions CONSISTENCY assumptions 123

8. Notation Data i = 1,…,130 study index k = 1, 2, 3,…,6 treatment index r ik – number of caries occurring in trial i, treatment k, during the trial follow-up period E ik – exposure time in arm k of trial i (in person years)

9. 9 Fluoride: Poisson MTC RE model rate at which events occur in arm k of trial i Exposure time in person years MTC consistency equations Priors i = 1,…,130

10. 10 MTC results: LHR relative to No Treatment Residual deviance is 278.6 (270 data points)

11. 11 Posterior mean of residual deviances for each point

12. Check how evidence is combined in the network Poor fit can indicate inconsistency in the network For each pair, separate direct evidence from indirect evidence implied by the rest of the network* Can see how evidence is combined in the network to give overall MTC estimate Helpful to locate pairs of comparisons where there may be problems 12 *Dias et al., Stats in Med. 2010

13. LHR for Placebo v Toothpaste Bayesian p-value = 0.32 Direct Indirect MTC 13

14. LHR for Placebo v Varnish Direct Indirect MTC Bayesian p-value = 0.04 14

15. LHR for Rinse v Varnish Direct Indirect MTC Bayesian p-value = 0.02 15

16. BIAS MODELS But we have additional information on the risk of bias of all included studies 16

17. 17 Treatments No of studies Allocation concealment Blinding NTPTRGV adequateunclearinadequateDoubleSingle? 1010010 4130310 3030102 1010100 3021021 9054063 4031031 6184676100 2522032203 9063900 3030300 1010100 1001010 4031220 1010010 Total 1301198211031710

18. 18 MTC RE model with bias MTC consistency equations Priors

19. 19 MTC Bias Model Assume non-zero mean bias, b i = b 0, in comparisons of NT or Pl with Active treatments For Active-Active comparisons assume mean bias is zero Expect bias to increase size of treatment effect: b < 0

20. 20 Fluoride: Risk of Bias indicators Allocation concealment Best empirical evidence of bias But… 98/130 studies unclear Only 11/130 studies adequate Some comparisons have no adequately concealed trials Blinding also available to inform risk of bias status Used Any bias as a composite indicator of bias: 54/130 studies at risk of bias.

21. 21 Probability of Bias Model Any study with unclear allocation concealment has a probability p of being at risk of bias Adequately concealed trials are not at risk of bias Inadequately concealed trials are at risk of bias Use only allocation concealment as bias indicator Bias terms identifiable in this rich network

22. 22 Probability of Bias Model

23. 23 Comparing Model Fit ResDev*pDDIC Between trial heterogeneity MTC with no bias adjustment278.6259.3537.90.22(0.19, 0.26) Bias adjustment AnyBias277.6257.9535.50.15(0.12, 0.18) Probability of bias274.6253.0527.60.12(0.10, 0.15) * Compare with 270 data points

24. 24 Posterior mean of residual deviances for each point: MTC and Prob of bias models Study 42: Placebo v Toothpaste (1 of 69 trials) Allocation concealment unclear Study 63: No Treat v Varnish (1 of 4 trials) Allocation concealment unclear and not double blind Study 102: Placebo v Varnish (1 of 3 trials) Allocation concealment unclear

25. 25 Treatment effects relative to No Treatment (LHR) Unadjusted MTC (solid) and Probability of Bias model (dashed)

26. Varnish effects Cochrane Review to assess efficacy of Fluoride Varnish (Marinho et al, 2004) Noted that the small number and poor methodological quality of varnish trials might be overestimating the true effect of this intervention. The results of the bias-adjusted analysis support this hypothesis. 26

27. 27 Which treatment is best? Unadjusted MTCBias-adjusted MTC Probability Best (%) Rank Probability Best (%) Rank No Treatment0606 Placebo0505 Toothpaste3.62.99.32.7 Rinse4.12.853.81.6 Gel3.73.212.42.9 Varnish88.51.224.62.8

28. 28 Results: Probability of Bias Bias posterior mean = -0.19, CrI (-0.36, -0.02) posterior sd = 0.40, CrI (0.29, 0.55) Trials with unclear allocation concealment are at risk of bias with probability p Posterior mean of p = 0.13 Model identified 5 trials (with unclear allocation concealment) as having a high probability of bias

29. Prob of bias for studies with unclear allocation concealment o – unclear allocation concealment + – unclear allocation concealment and single blind – unclear allocation concealment and unclear blinding status 29

30. 30 Other findings Between trial heterogeneity in treatment effects reduced in bias-adjusted model Model with Active-Active bias was also fitted with similar results: Active-Active bias had posterior mean of zero But assumptions on direction of bias… Assumed bias would favour the newest treatment (also the most intensive)

31. 31 Conclusions Bias estimation and adjustment possible within MTC because there is a degree of redundancy in the network Assumption that study specific biases are exchangeable within the network Uses only internal evidence Weaker than required from using external evidence Ideas extend to multiple bias indicators But will need a very rich evidence structure

32. 32 Consequences for Decision Modelling Uses only internal evidence May be more acceptable to patient groups, pharmaceutical industry… Risk of bias indicator chosen based on empirical research Results may change if different bias indicators chosen Again: Assessment of model fit & sensitivity analysis crucial if decisions based on these models are to have credence

33. 33 References Our website: http://bristol.ac.uk/cobm/research/mpeshttp://bristol.ac.uk/cobm/research/mpes Dias S, Welton NJ, Marinho VCC, Salanti G, Higgins JPT and Ades AE (2010) Estimation and adjustment of Bias in randomised evidence using Mixed Treatment Comparison Meta-analysis. Journal of the Royal Statistical Society A, to appear Vol 173 issue 4 (available online). Dias S, Welton NJ, Caldwell DM and Ades AE (2010) Checking consistency in mixed treatment comparison meta-analysis. Statistics in Medicine, 29, 945-955. Schulz KF, Chalmers I, Hayes RJ and Altman DG (1995) Empirical Evidence of Bias. Dimensions of Methodological Quality Associated With Estimates of Treatment Effects in Controlled Trials. JAMA, 273, 408-412.