1. A systematic review of selected journals
1. Introduction
6. Recommendations
4. Results
Non-inferiority trials: inconsistent guidelines, inconsistent reporting?
A systematic review of selected journals
1MRC Clinical Trials Unit at UCL, 2Department of Infectious Disease Epidemiology, London School of Hygiene and Tropical Medicine, 3Department of Medical Statistics, London School of Hygiene and Tropical Medicine
Sunita Rehal1, Tim Morris1, Katherine Fielding2, James Carpenter1,3, Patrick Phillips1
2. Objective
Non-inferiority trials are designed to show a new intervention performs not much worse (defined by the non inferiority margin ∆) when compared to a standard intervention
The CONSORT extension statements (2006 & 2012)1,2; draft FDA (2010)3; EMEA 2000 and 20064,5; ICH E96; ICH E107 and SPIRIT (2010)8 guidelines are currently available guidelines that contain guidance for non-inferiority trials
Some recommendations provided by guidelines are unclear. For example:
All guidelines recommend justifying the choice of the margin on a clinical basis, but some recommend statistical justifications additionally (Draft FDA 20101, ICH E102, EMEA 20063)
The draft FDA guidelines recommend ITT and as-treated analyses but do not define “as-treated”, whereas the CONSORT guidelines state “authors should indicate if conclusions are related to per-protocol (PP), intention-to-treat (ITT) or both and if conclusions are stable between them”.
Presentation of 95% confidence intervals are widely recommended but some state 90% confidence intervals may be appropriate
Guidance on how to handle missing data in non-inferiority trials is sparse
It is unclear what sensitivity analyses should be done. The ICH E96 and SPIRIT8 guidelines recommend using sensitivity analyses to investigate impact of missing data, whereas the EMEA and CONSORT guidelines recommend sensitivity of the trial is shown by obtaining similar results for the ITT and PP analyses
As a result of the confusion within these guidelines, we performed a systematic review to investigate whether these inconsistencies are reflected in real practice
The objectives of this systematic review were to assess:
The quality of reporting of non-inferiority trials in top medical journals
Whether poor conduct and reporting would be associated with demonstrating non-inferiority
3. Methods
Systematic review of reporting in top medical journals with an impact factor >10:
NEJM; Lancet; JAMA; BMJ; Annals of internal medicine; PLOS medicine; JAMA of internal medicine; Archives of Internal Medicine (as of May 2015)
Searched in Ovid (MEDLINE) for:
‘non-inferiority’; ‘non-inferior’; ‘noninferiority’; ‘noninferior’
From January 2010 to May 2015
Quality grading system developed based on:
Justification of margin (yes vs. no/poor); analyses performed (<2 vs. ≥2); consistency between type I error rate and significance level of confidence interval (yes vs. no/unclear)
Table 1: Justification of choice of margin
No. articles
(N=168)
Made no attempt for justification
90 (54%)
Clinical basis. No evidence for consultation with external expert group, and no reference to previous trials of the control arm
32 (19%)
Preservation of treatment effect based on estimates of control arm effect from previous trials
13 (8%)
Expert group external to the authors. No reference to previous trials of the control arm
6 (4%)
The same margin as was used in other similar trials
5 (3%)
10-12% recommended by disease specific FDA guidelines
4 (2%)
General comment that margin was decided according to FDA/regulatory guidance.
Clinical basis and based on previous similar trial. No evidence for consultation with external expert group, and no reference to previous trials of the control arm
3 (2%)
Other
11 (7%)
Table 2: Quality of reporting of trials associated with conclusions of non-inferiority
168 articles were included in the review following eligibility:
90 (54%) made no attempt to justify the choice of the non-inferiority margin (Table 1)
91 (54%) reported two analyses
56 (32%) reported the ITT and PP analyses
65 (39%) reported one analysis
54 (32%) reported ITT analysis
95 (57%) had consistent type I error rate with the significance level of confidence intervals
99 (59%) did not report whether or not any imputation was done, neither was there any acknowledgement that missing data was not anticipated to be problematic
Of the 56 (33%) that used imputation
35/56 (63%) used single imputation methods
11/56 (20%) used multiple imputation
6/56 (11%) used complete case analysis
64 (38%) performed sensitivity analyses
13/64 (20%) used either ITT/PP analyses
Concluded non-inferiority
Grade
Yes (N=132)
No (N=29)
Other (N=7)
Total (168)
Excellent
11 (73%)
2 (13%) 15
Good
55 (86%)
9 (14%)
0 (0%) 64
Fair
48 (80%)
8 (13%)
4 (7%) 60
Poor
18 (62%)
10 (34%)
1 (3%) 29
Articles were graded on whether:
Margin justified
≥2 analyses
Type I error rate consistent with significance level of confidence interval
Excellent if all were fulfilled
Good if two criteria were fulfilled
Fair if one criteria fulfilled
Poor if none fulfilled
χ12=3.76; p=0.05 (Cochran-Armitage test); Excluding trials that concluded ‘other’
5. Conclusions
There were clear violations of available guidelines. This includes the CONSORT statements which are recommended by all journals on their website for authors to use when submitting manuscripts for publication
Poor reporting from this review and inconsistencies within guidelines suggest:
The non-inferiority design is not well understood by those using it
Methods for non-inferiority are yet to be optimised
Justification of the non-inferiority margin was poor
Definitions of analyses conducted were widely variable, especially in PP analyses. Subtle changes in PP definitions may be may be superficially small but could make critical differences to the results of a trial
Over a quarter of articles were not clearly consistent with regards to the type I error rate and corresponding confidence interval due to poor reporting of a one-sided or two-sided hypothesis test or total absence of the type I error rate in the sample size calculation
The majority of articles did not explicitly state whether or not methods to handle missing outcome data would be considered so it is unclear whether the potential issues surrounding missing data are well recognised
Poorly reported trials were less likely to demonstrate non-inferiority; p=0.05 (Cochran-Armitage test)
There is urgency for research into appropriate ways of handling missing data, especially in the PP analysis for non-inferiority trials
Justification of margin should be made mandatory in journals
Supplementary content should be taken advantage of, containing additional detail not included in the main paper
Reference(s) should be made to preserve the treatment effect based on the estimate of the control arm from previous trials
Recommendations
Consideration of missing data and use of sensitivity analyses to test assumptions of missing data for primary analyses
Confidence intervals should be consistent with the type I error rate used in sample size calculations
Definition of analyses should be based on the primary outcome rather than satisfying ambiguous ITT and PP definitions
7. References
Piaggio G, Elbourne DR, Altman DG, Pocock SJ, Evans SJ, Group C. Reporting of noninferiority and equivalence randomized trials: an extension of the CONSORT statement. Jama. 2006;295(10):
Piaggio G, Elbourne DR, Pocock SJ, Evans SJ, Altman DG, Group C. Reporting of noninferiority and equivalence randomized trials: extension of the CONSORT 2010 statement. Jama. 2012;308(24):
Food DA, H.H.S. Draft Guidance for Industry Non-Inferiority Clinical Trials
Committee for Medicinal Products for Human Use (CHMP) CfPMP. Points to Consider on Switching Between Superiority and Non-inferiority London, England2000 [cited 2016 July 26th]. Available from:
Committee for Medicinal Products for Human U, Efficacy Working P, Committee for Release for C. Committee for Medicinal Products for Human Use (CHMP) guideline on the choice of the non-inferiority margin. Statistics in medicine. 2006;25(10):
International conference on harmonisation; guidance on statistical principles for clinical trials; availability--FDA. Notice. Federal register. 1998;63(179):
Food, Drug Administration HHS. International Conference on Harmonisation; choice of control group and related issues in clinical trials; availability. Notice. Federal register. 2001;66(93):
Chan AW, Tetzlaff JM, Gotzsche PC, Altman DG, Mann H, Berlin JA, et al. SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials. Bmj. 2013;346:e7586.