Joel Singer, Programme Head, Methodology and Statistics, CIHR Canadian HIV Trials Network

What is a Non-inferiority Trial? A trial to show that a particular therapeutic agent or policy is no worse than some other standard.

What is a Non-inferiority Trial? Type 1: Both agents have been shown superior to placebo/no treatment Type 2: Only one agent has been shown superior to placebo/no treatment

What is a Non-inferiority Trial? The standard (STD) should have clinical efficacy: (i) that is of substantial magnitude; (ii) is precisely estimated; (iii) estimates that are relevant to the setting in which the trial is being conducted.

ICH Guidelines A suitable active comparator could be a widely used therapy whose efficacy in the relevant indication has been clearly established and

ICH: Constancy Assumption quantified in well-designed and well documented superiority trials and which can be reliably expected to have similar efficacy in the contemplated active control trial.

Example of non-Constancy Suppose vancomycin, has been shown to provide a large improvement, compared to no treatment, in cure rate in patients with various infections.

Example of non-Constancy Suppose a high prevalence of vancomycin-resistant enterococci (VRE) has developed. NI trial: comparing an experimental antibiotic (EXP) with vancomycin is conducted where VRE is prevalent

Example of non-Constancy A biased estimate of the efficacy of EXP would be obtained if the constancy assumption is falsely presumed to hold in the NI study.

What is a Non-inferiority Trial? Because proof of exact equality is impossible, a prestated margin of noninferiority for the treatment effect in a primary patient outcome is defined.

What is a Non-inferiority Trial? Non-inferiority trials are intended to show whether a new treatment has at least as much efficacy as the standard or is worse by an amount less than Δ,often on the premise that it has some other advantage

What is a Non-inferiority Trial? Both participants and outcome measures in a noninferiority or equivalence trial should be similar to those in trial(s) that established the efficacy of the reference treatment.

What is a Non-inferiority Trial? The required sample size is calculated using the confidence interval (CI) approach, considering where the CI for the treatment effect lies with respect to both the margin of noninferiority and a null effect.

What is a Non-inferiority Trial? Sample size depends on the level of confidence chosen, the risk of type II error (or desired power), and Δ.

What is a Non-inferiority Trial? A prestated margin of noninferiority can be specified as a difference in means or proportions or the logarithm of an odds ratio, risk ratio, or hazard ratio.

What is a Non-inferiority Trial? The meaning and clinical importance of measures such as relative risk, odds ratio and relative hazard will be dependent on the base rate of outcome

What is a Non-inferiority Trial? A prestated margin of noninferiority is often chosen as the smallest value that would be a clinically important effect.

What is a Non-inferiority Trial? The required size of non-inferiority trials is therefore usually larger than that for superiority trials, But: The actual calculation is the same as in superiority trials

What is a Non-inferiority Trial? One should avoid features that might dilute true differences between treatments, thereby enhancing the risk of erroneously concluding non- inferiority

What is a Non-inferiority Trial? For superiority trials, intention-to- treat (ITT) analysis (analyzing all patients within their randomized groups, regardless of whether they completed allocated treatment) is recommended

What is a Non-inferiority Trial? In non-inferiority trials, ITT analysis will often increase the risk of falsely claiming non-inferiority (type I error), although not always

What is a Non-inferiority Trial? In non-inferiority and equivalence trials, non-ITT analyses might be desirable as a protection from ITT’s increase of type I error risk (falsely concluding non-inferiority).

What is a Non-inferiority Trial? Interpreting a non-inferiority trial’s results depends on where the CI for the treatment effect lies relative to both the margin of non-inferiority and a null effect. The observed treatment effect is not sufficiently informative.

What is a Non-inferiority Trial? For superiority trials, intention-to- treat (ITT) analysis (analyzing all patients within their randomized groups, regardless of whether they completed allocated treatment) is recommended

Major Methodological Issues for Equivalence/Non-inferiority Trials 1. Difficult to get consensus on difference considered equivalence or non-inferior. 2. In superiority trials, lack of rigor in the conduct of a trial will tend to lead to equivalence between the comparators and will cause a conservative bias.

3. Lack of rigor or anything which causes background noise in an equivalence/noninferiority trial will bias in favour of the hypothesis of interest (no difference).

Important Variables for Defining Equivalence/non-inferiority 1. Outcome. 2. Prevalence of the disease. 3. Other: cost, invasiveness, ease of use.

Trial Design Nucleoside backbone: d4T and 3TC Inclusion criteria:pVL > 5000 copies/mL any CD4 cell count any stage of CDC-classification

Outcome Measures % of patients with treatment failure at week 48: less than 1log10 decline in pVL in first 12 weeks 2 consecutive pVL > 50 copies/mL from week 24 onwards new CDC-C event or death change of allocated treatment

All analyses Intention-to-Treat (unless stated otherwise) all randomised patients pVL data: missing=failure Four pre-defined pairwise comparisons NVP-bd vs EFV (primary) NVP-od vs NVP-bd NVP-od vs NVP+EFV EFV vs NVP+EFV Methods

Given that the 2NN trial was conducted as a non-inferiority trial, how does one go about interpreting the results? The goal of an inferiority trial is to be able to rule out differences greater than the minimally important clinical difference (MICD).

Translated into a statistical statement, this means we can say with confidence (based on the 95% confidence interval) that the true difference is unlikely to exceed the difference deemed clinically important.

Methodological Concerns With the 2NN Trial 1. Choice of definition of outcome. 2. Background noise.

Concerns in 2NN Study re non-inferiority 1. patients who never received treatment (20 in EFV, 10 in NVP-BD). 2. early compliance failures. 3. Question inclusion of changes in d4T or 3TC as evidence of failure.

What Were the Primary Results of the 2NN Trial?

Treatment Success and Failure Success: only significant difference: EFV vs NVP+EFV, p< % of patients

There was no statistical difference for the primary comparison (NVP vs EFV) so one cannot say definitively that one treatment is better than the other.

The 95% confidence interval (1% in favour of nevirapine to 13% in favour of efavirenz) does not allow one to rule out a clinically important difference (at least as defined by the investigators) on the primary efficacy variable.

Conclusions NVP and EFV have comparable potency in suppressing HIV-1 replication NVP-od and NVP-bd show comparable efficacy Co-administration of NVP and EFV results in higher treatment failure due to increased toxicity

Comments Conclusion slide does not address primary outcome (combination of disease progression, virologic failure and change of therapy) Does not address issue of proof of non- inferiority

Some Technical Sample Size issues Typically, one wants the sample size to be large enough so that the bounds of the confidence interval will exclude values outside the non-inferiority range 80%-90% of the time

Some Technical Sample Size issues If the outcome is continuous, and one assumes homogeneity of variance, then the sample size required for non- inferiority under the null hypothesis is the same as the sample size to detect the indicated difference under the alternative hypothesis

Some Technical Sample Size issues If the outcome is dichotomous, then the sample size required for non-inferiority under the null hypothesis is not the same as the sample size to detect the indicated difference under the alternative hypothesis because the variability is a function of p1 and p2

Some Technical Sample Size issues In a non-inferiority trial, the hypotheses are inverted. The null hypothesis is that there is a difference > m, and the alternative is the null hypothesis

Some Technical Sample Size issues Generally, the clinical world is concerned about absolute differences between treatments, but depending on the statistical model being used, the null and alternative may be stated in other terms eg hazard ratios

Some Technical Sample Size issues The choice of what is considered non- inferior in terms of a hazard ratio is typically driven by what one assumes the baseline rates are Eg if the hazard rates are relatively low, a somewhat large hazard rate may translate into a relatively low absolute difference

Summary of 2NN Study

1. Demonstrated a non-significant 6% advantage in the primary outcome in favour of efavirenz over nevirapine bid on intent-to-treat analysis. 2. Confidence interval around obtained difference between efavirenz and nevirapine bid include confidence bounds to 13% in favour of efavirenz. Cannot exclude non-inferiority on primary outcome set out by study criteria.