1. The Application of Tipping Point Analysis in Clinical Trials
Kevin Ding
8/1/2018

2. Tipping points and FDA question
Yan et al. (2009) from FDA: “Tipping points are outcomes that result in a change of study conclusion. Such outcomes can be conveyed to clinical reviewers to determine if they are implausibly unfavorable. The analysis aids clinical reviewers in making judgment regarding treatment effect in the study.”
FDA question and comment on one of our studies:
For primary endpoints, in addition to analyses on observed data only, examine the potential effects of missing data and rescue on your results using tipping point sensitivity analyses. The tipping point analyses should vary assumptions about average values of the primary endpoint among the subsets of patients on the investigative drug and placebo arms who withdrew from treatment prior to the planned endpoint
For continuous endpoints, to avoid the untenable assumption that unobserved data is missing-at-random, provide analyses of covariance at each endpoint
Business Use Only

3. Imputation methods and robustness check
How to deal with missing data is a long standing challenge in statistics.
In statistics, imputation is the process of replacing missing data with substituted values.
How to substitute values? Right now, LOCF is rarely used in regulatory submission.
Multiple imputation (MI) is more often used. Generally three approaches for doing MI:
For monotone missing data: Use sequential regression procedure
For non-monotone missing data:
Markov Chain Monte Carlo (MCMC) simulation methodology
Fully conditional specification (FCS) methods
For Continuous variables, depending on missing pattern, use either sequential regression or MCMC, which assumes that all the variables in the imputation model have a joint multivariate normal distribution.
For categorical variable, although theoretically MCMC can be used, but in SAS only FCS can be used, which allows each variable to be imputed using it’s own conditional distribution instead of one common multivariate distribution. This is especially useful when negative or non-integer values can not be used in subsequent analyses such as imputing a binary outcome variable.This specification may be necessary if your are imputing a variable of a binary outcome for a logistic model or count variable for a poisson model.
After imputation, check robustness of conclusion based on imputed data through a variety of sensitivity/supportive analyses, e.g. pattern mixture model (PMM), tipping point analysis (TPA)
If TPA is used, clinicians assess clinical plausibility of tipping points.
Business Use Only

4. EMA question for our studies
In the both pivotal studies, continuous variables (e.g., ASAS components) were analyzed using a MMRM which is valid under the missing at random (MAR) assumption.
However, in the Guideline on Missing Data in Confirmatory Clinical Trials (EMA/CPMP/EWP/1776/99 Rev. 1) it is required that, when missing data are particularly problematic, in addition to MAR assumption further sensitivity analyses that treat certain types of missing data as MNAR should be provided.
The MAH should conduct at least one sensitivity analysis (for each ranked secondary endpoint) where missing data are handled in a conservative manner in line with the failure imputation for dichotomous variables (e.g., BOCF) and to justify the lack of these sensitivity analyses.
Business Use Only

5. Missing Mechanism & assumptions
No assumption
MCAR
MAR
MNAR
Missing Complete at Random
Missing at Random – ignorability assumption
Missing Not at Random
The missingness is independent of both unobserved and observed data.
the probability of a missing value can depend on some observed quantities but does not depend on any unobserved data.
Missingness that depends on unobserved predictors.
The probability of a missing value does not depend on any other observations in the data set, regardless of whether they are observed or missing
The probability a variable is missing depends only on available information.
The value of the unobserved variable itself predicts missingness.
LOCF (last observation carried forward), BOCF (baseline value carried forward),
CC (Complete-case Analysis) - listwise deletion/Pairwise Deletion - Available Case analysis,
MMRM (mixed model repeated measurement) – REML (restricted maximum likelihood),
PMM (Pattern-mixture modeling):
Reference-based Imputation: Copy Reference/Copy Differences in Reference/Jump to Reference
WOCF (worst observation carried forward),
Single-value Imputation (for example, mean replacement, regression prediction (conditional mean imputation), regression prediction plus error (stochastic regression imputation )
Multiple Imputation (ML)
Delta-adjustment imputation:Tipping Point Approach (TPA)
Imputation based on logical rules: Non-responder imputation
GEE
Weighted GEE
Business Use Only

6. Why Tipping Point Analysis (TPA)?
MAR assumed that the statistical behavior of the unobserved data is the same as it had been observed, such that the unobserved data can be predicted from the observed data.
This may not be true, actually we will never be sure whether this is true or not as the assumption is untestable.
So we can do some sensitivity analysis based on more relaxed assumption on missing mechanism: MNAR to see if missing mechanism deviates from MAR, what result would be.
In broad sense, PMM is such a sensitivity approach, which includes two methods:
reference-based imputations: drug-treated patients after early discontinuation of study medication have outcomes similar to placebo-treated patients
delta-adjustment imputations: drug-treated patients after early discontinuation have outcomes worse than otherwise similar drug treated patients who did not dropout by an amount equal to delta
Reference-based (control-based) imputation is a sensitivity analysis assuming withdrawals have trajectory or distribution of control arm.
Delta-adjustment imputation is a sensitivity analysis assuming trajectory of withdrawals is worse by some 𝛿.
A single delta-adjustment analysis allows testing if a specific departure from MAR overturns the MAR result.
The delta-adjustment method can also be applied repeatedly as a progressive stress test to find how extreme the delta must be to overturn the MAR result, which is TPA
Business Use Only

7. Relationship between PMM and TPA
The pattern-mixture model (PMM) approach to sensitivity analysis models the distribution of a response as the mixture of a distribution of the observed responses and a distribution of the missing responses.
Missing values can then be imputed under a plausible scenario for which the missing data are missing not at random (MNAR).
If this scenario leads to a conclusion different from inference under MAR, then the MAR assumption is questionable.
TPA is a kind of PMM. It is an imputation based on a series of adjustments on imputed values from MAR-based Multiple Imputation (MI), so called delta-adjustment.
Another kind of PMM is Reference-based imputation (RBI) , which assumes that after discontinuation, subjects discontinued from the experimental treatment arm will exhibit an evolution of the disease similar to subjects in the control arm.
TPA does not assume this. The only assumption of TPA is that missingness depends on unobserved predictors (MNAR). It is a much weaker assumption than RBI. So it is better than RBI to test robustness of result due to deviation from MAR.
| Presentation Title | Presenter Name | Date | Subject | Business Use Only

8. Why is TPA popular?
The reason for popularity of this method in recent years:
the missing mechanism assumptions of MCAR or MAR cannot be assessed, while most often used statistical models (MMRM, GEE, ANCOVA) use either MAR or MCAR for missing data, highlighting the need for sensitivity analyses.
TPA is a sensitivity analysis under MNAR assumption, which is weaker than MCAR and MAR.
In practice, we do not know what is the real missing mechanism in data, so the safest assumption is MNAR, however primary analysis is usually based on either MCAR or MAR.
TPA is like a progressive stress-testing to assess how severe departures from MAR or MCAR must be in order to overturn conclusions from the primary analysis.
If implausible departures from MAR is needed in order to change the results from statistically significance (p<=0.05) to insignificance (p>0.05), the results will be said to be robust.
If the tipping point obtained for overturning the primary analysis result is clinically plausible, the conclusion would be questioned by Health Authority.
| Presentation Title | Presenter Name | Date | Subject | Business Use Only

9. Tipping Point Analysis (TPA) for Binary Endpoint: One-time-point analysis
Two kinds of TPA:
marginal approach: the adjustment is applied after imputation of all missing values and the adjustment at one visit does not influence imputed values at other visits: constant departure from MAR over time
conditional approach: the delta-adjustment is applied in a sequential, visit-by-visit manner: departure from MAR increases over time
The first approach is more commonly used in regulatory submission.
TPA for binary endpoint in one-time-point analysis (such as logistic regression) can be a delta-adjustment method based on any imputation method, such as Non- responder imputation (NRI) or some modeling approach, such as Generalized linear mixed model (GLMM).
In practice, it is often not based on Multiple imputation(MI), which is the process of “filling in” missing data points with probable values derived from the existing dataset, instead, an “exhaustive scenario” approach is used (shown next).
It creates a matrix of all possible response patterns for the observed missing data in the two groups, the “Control arm” on one axis and the “Treatment arm” on another axis.
For each possible combination of missingness for two treatment groups, it is possible to categorize the point as to whether the assumptions on the missing data change the statistical conclusions or not. The region that marks where the decision changes, is called the tipping-point boundary.
Business Use Only

10. SAS table of TPA for Binary Endpoint
We can see here at week 16, treatment group has 4 missing values, placebo group has 5 missing values.
The results of all the possible combinations for the missingness in both groups are presented.
The area within the red line overturns significant treatment effect. The larger this area is, the less robust the treatment effect is.
This is called “TPA via exhaustive scenarios”. This approach is better than imputation-based delta adjustment because it is an assumption-free method, so it is easier to be accepted by Health authority.
Business Use Only

11. SAS figure of TPA for Binary Endpoint: dots are insignificant p values
Business Use Only

12. R output of TPA for Binary Endpoint Heat map figure made by R package TippingPoint
The red region is dangerous region where significant treatment effect will be overturned. The green region is safe region where significance of treatment effect can be kept.
Number of successes is number of responders.
R figure provides the same quantitative information as SAS table but a better and nicer presentation (with greyed background colors for increased p values)
Business Use Only

13. TPA for Continuous Endpoint
Two presentation methods:
TPA1 using SAS: Presenting all the combinations of deviation (shift) from imputed values (under MAR/MCAR) for nonrespondents in treatment group and those in control group and comparing the region containing significant p values associated with each combination with the region with insignificant p values.
TPA2 using R package TippingPoint: Presenting all the combinations of average outcome (like mean of endpoint) for nonrespondents (discontinued subjects) in treatment group and that in control group and overlaying the contour of imputed values under MAR/MCAR to see what proportion of imputed values fall into “danger area”.
Graphically both TPA1 and TPA2 contain a reference point corresponding to average outcome for two treatment groups without any shifting.
TPA1 is mainly used as supportive/sensitivity analysis for regulatory submission while TPA2 mainly serves as an internal exploratory analysis to assess plausibility of imputation method.
Adjusting which treatment group?
Typically, only the experimental treatment arm would be delta-adjusted while allowing the control arm to be handled using a MAR based approach. To be safe and cover all possible cases, we can also adjust (by worsening outcome) both experimental treatment arm and placebo arm.
Different from TPA for binary endpoint, TPA for continuous endpoint is mostly based on MI imputed values or results from some integrated approach, such as combining MI and GLMM through proc mcmc in SAS (shown later).
Business Use Only

14. General procedure of TPA implementation for continuous endpoints
Implementing TPA for continuous endpoints includes the following steps with step 2 to step 4 being the standard multiple imputation (MI) steps:
Determine adjustment group and adjustment direction.
The missing data are filled in m times to generate m complete data sets.
The m complete data sets are analyzed by using standard procedures.
The results from the m complete data sets are combined for the inference.
Repeat the steps #2 to generate multiple imputed data sets with a specified shift parameter that adjust the imputed values for observations in experimental treatment group, or both experimental treatment and placebo group, which is determined in step 1).
Repeat the step 3 for the imputed data sets with shift parameter applied.
Repeat the step 4 to obtain the p-value to see if the p-value is still <=0.05.
Repeat the steps 5-7 with more stringent shift parameter applied until the p-value >0.05.
After obtaining TP, clinical team assess the clinical plausibility of TP.
| Presentation Title | Presenter Name | Date | Subject | Business Use Only

15. Implementation of TPA for continuous endpoint (1)
1. Convert the data from vertical structure to horizontal structure (one subject one line with each time point as a variable on the column).
2. Use macro to generate multiple imputed data sets, with a specified sequence of shift parameters that adjust the imputed values for observations in both treatment arms at the analysis time point (here week 24). The core code for imputation of data with arbitrary missing pattern is:
proc mi data=haq2 seed=123 nimpute=300 out=outmi;
class Trt01p TNFRES ;
by trt01p;
fcs reg;
mnar adjust( w24/ shift= &sj1 adjustobs=(Trt01p ='Placebo' ))
adjust( w24/ shift= &sj2 adjustobs=(Trt01p =‘XXXX 150 mg' ));
var tnfres weight w0 w1 w2 w3 w4 w8 w12 w16 w20 w24 ;
run;
The MNAR statement imputes missing values by using the pattern-mixture model approach, assuming the missing data are missing not at random (MNAR).
You can also use a regression predicted mean matching method to impute missing values by changing fcs statement to “ fcs regpmm”. This method is similar to the regression method except that it imputes a value randomly from a set of observed values whose predicted values are closest to the predicted value for the missing value from the simulated regression model(Heitjan and Little 1991; Schenker and Taylor 1996).The predictive mean matching method ensures that imputed values are plausible; it might be more appropriate than the regression method if the normality assumption is violated (Horton and Lipsitz 2001, p. 246).
If the data has monotone missing pattern, then you can change fcs statement above to: “monotone method=reg; ”, which use sequential regression method in MI.
Business Use Only

16. Implementation of TPA for continuous endpoint (2)
3. Convert the data structure of the imputed data back to vertical (one time point one line), apply statistical model of interest (MMRM/GEE) on it and combine results for each shift parameter by using Rubin’s rule:
proc mianalyze data=dif2;
by Shift1 shift2;
modeleffects estimate;
stderr stderr;
ods output parameterestimates=dif_nmar_t;
run;
4. Fine-tune the tipping point by adjusting the range and increments of the shift parameters and present the final result in tabular or graphical form.
The limitation of the code used in step 2 is adjustments for two treatment groups can only be with the same magnitude at one time point. If different magnitudes of adjustments are needed to apply to two treatment arms at the analysis visit, then more complex programming (using manual adjustment rather than proc mi option) is involved, often a macro is created.
Business Use Only

17. Another MI method for continuous endpoint: MCMC
Another method of MI for continuous endpoint is Markov Chain Monte Carlo (MCMC) which assumes that all the variables in the imputation model have a joint multivariate normal distribution (Schafer, 1997) .
if data has monotone missing pattern, then regression method used for imputation does not use Markov chains.
Typical clinical trials often have data with non-monotone missing pattern (both intermittent missing data and dropouts). For data sets with non-monotone missing patterns, you can use either MCMC or FCS method.
MCMC is essentially a Bayesian approach.
You can implement MCMC for MI in SAS by using either
mcmc statement in proc mi
or proc mcmc
Advantage of proc mcmc over mcmc in proc mi: explicitly modeling imputed using the observed information.
One example: ANCOVA of change from baseline of a score at a particular time point with some missing data
If a linear or nonlinear time trend is observed, then we can explicitly model this endpoint on time and treatment groups by taking a Bayesian approach.
Business Use Only

18. Example of proc mcmc for MI
Instead of maximizing the likelihood function, PROC MCMC draws samples (using a variety of sampling algorithms) to approximate the posterior distributions of model parameters.
In a time series figure, score change from baseline is a line(straight line or curve) starting from the origin. We can use a random slope model (without intercept) to fit this line.
A normal prior with large variance is assigned to all regression parameters (beta0-beta4), and a uniform prior is assigned to the standard deviation of score change (sigma) and random effect of the mean (tau).
Gamma is a patient-level random effect. You use a RANDOM statement to specify the random effects, and you use the SUBJECT= option to specify the subject.
We can see that PROC MCMC can perform joint modeling of missing responses and covariates by treating missing values as random variables and estimating their posterior distributions, an approach of incorporate modeling while doing MI.
Business Use Only

19. SAS output of TPA for a continuous score (1)
The values on two axis are delta (shift values).
The yellow highlighted area is the safe area (with all significant p values).
The area on the right of the boundary is the dangerous area, which includes p values corresponding to all combinations of shifts from the imputed values (under MAR) for dropout subjects in treatment and placebo groups that would overturn significant treatment effect.
Business Use Only

20. SAS output of TPA1 for a continuous score (2)
Business Use Only

21. R output of TPA for continuous score
TPA figure made by TippingPoint package in R
The blue convex hull contains 95% of the 30 MIs (imputed values for change from baseline at W24), generated under the MAR
Two pairs of vertical and horizontal blue lines correspond to minimum and maximum values of outcomes observed in each group, and dashed blue lines represent average outcomes (means of changes from baseline for the score).
The green line is tipping points (values that would result in a p-value = 0.05), the red region is dangerous region where significant treatment effect can be overturned.
It illustrates what proportion of imputed values under MAR fall into the “dangerous red area”. If this proportion is visually shown to be high, that is showing MAR assumption is not robust.
It is a tool for a quick judgement of the validity of MAR assumption
Business Use Only

22. Interpretation of result of TPA
The larger is the shift required to see a ‘change in inference”, the more robust are the imputed results to deviations from the MAR assumption.
The robustness of significant treatment effect is determined by whether shift parameters are plausible, which may need clinical input (the increasing shifts may lead to clinically meaningless outcomes).
If the shift (delta that resulted in the change of conclusion) is plausible, then the conclusion from primary analysis may be doubted. That is, from the clinical point of view there is reason to view the primary result with caution.
Examples of clinical evaluation of tipping point plausibility are shown later.
| Presentation Title | Presenter Name | Date | Subject | Business Use Only

23. TPA in regulatory submissions
Tipping Point Analysis (TPA) for continuous endpoints in FDA submissions
Study #
company
drug name
indication
submission date
adjustment treatment arm
reference imputation 1
Forest Laboratories
depression
FETZIMA
9/24/2012
both treatment and placebo
MAR 2
Pharmaxis Pharmaceuticals
Bronchitol (Mannitol Powder)
cystic fibrosis 
2013
both treatment and placebo, only active treatment 3
GSK
Fluticasone
asthma
22-Oct-2013
MCAR 4
Otsuka
Brexipirazole
Schizophrenia
11-Jul-2014
only active treatment 5
J&J
Paliperidone Palmitate
Schizoaffective disorder
2014 6
Pfizer
Tofacitinib
Psoriatic Arthritis
2/22/2017 7
Janssen-Cilag International N.V.
Paliperidone
Schizoaffective Disorder
28 July 2014
Only active treatment
LOCF (MCAR) & MI (MAR)
Business Use Only

24. Example of TPA in FDA submission 1
In FDA’s Statistical Review for NDA Drug Name: FETZIMA (Levomilnacipran) extended-release capsules 20, 40, 80, and 120 mg Indication: Major Depressive Disorder Applicant: Forest Laboratories, Inc.
A “tipping point” analysis was conducted by increasing the shift parameter beyond the maximum value of 8 considered by the sponsor. The mean difference in MADRS change scores between drug and placebo would loose statistical significance at alpha = at a shift parameter of 16 (see Table 16). The value of 16 appears to be rather large and unlikely to be a realistic mean difference at yt+1 between patients that drop-out after the tth visit and patients that continue. The PMM model results are consistent with the primary MMRM model results at the more realistic values of the shift parameter (i.e., 2, 4, …, 14).
The bolding is added by me, which shows clinical evaluation of tipping point plausibility.
Business Use Only

25. Example of TPA in FDA submission 2
In Dry Powder Mannitol (DPM) Pharmaxis Pulmonary and Allergy Drugs Advisory Committee January 30, 2013, tipping point approach was used for stress test to see how robust primary analysis method is robust to the departure of the MAR assumption.
“They explored the tipping point in the ITT population at which DPM would no longer show a significant effect. To do this, the penalty at each missing time point is increased up to the point that statistical significance is lost. They showed what happens when they stress tested the data even more. They increased the size of penalty for each missing visit in the pattern mixture model up until the point where significance is lost. The penalty would need to be more than 450 mLs at each missing time point before the effect estimate is reduced to 55 mLs and is no longer significant. This means that each patient leaving before week six could be penalized by 1,350 mLs. A tipping point requiring such a large volume does not seem plausible. They challenged the robustness even further, again using the same pattern mixture model, but this time identifying a tipping point when only penalizing the DPM arm but not control. Even applying this extreme method, the tipping point needed to reach 150 mLs before significance was lost. Now, this means that even patients withdrawing before week six in the control arm carry no penalty at all, but, similarly, DPM withdrawals being penalized by 450 mLs.”
The second bolding part shows delta adjustment can be made on both arms or only experimental drug arm.
Business Use Only

26. Example of TPA in EMA submission
In EMA’s assessment report for PALIPERIDONE (2015), various sensitivity analysis including tipping point analysis were performed.
The key secondary endpoint, change from double-blind baseline in PSP score, was analyzed using a mixed model repeated measures (MMRM) Analysis of Covariance (ANCOVA) model. Robustness and consistency of findings at the Month 15 end point was assessed through a number of supportive and sensitivity analyses (pattern mixture model, tipping-point analysis, and pattern mixture modeling with multiple imputation).
To evaluate the validity of the MAR assumption, several sensitivity analyses based on missing not at random (MNAR) were performed to assess the robustness and consistency of findings at the Month 15 end point. These analyses were based on Pattern Mixture Models (PMM), a Tipping-Point Analysis and a Pattern Mixture Model with Multiple Imputation approaches.
Business Use Only

27. Conclusion
TPA provides reassurance that missing data are not likely to “affect” the results (i.e., primary analysis conclusions hold) by checking how severe departures from MAR/MCAR
Agencies (FDA, EMA) do expect this kind of investigations, in particular when a trial is pivotal to a submission.
TPA can be based on any imputation method or imputation-modeling integrated approach. But Multiple Imputation (MI) is most often used, it constructs multiple complete data sets by filling each missing datum with plausible values, and then it obtains parameter estimates by averaging over multiple data sets.
The implementation of MI is closely related to missing pattern of data.
Comparing with frequentist sequential regression approach, the Bayesian paradigm (MCMC) offers an alternative model-based solution, in which the missing values are filled with draws samples to approximate the posterior distributions of model parameters.
The implementation of TPA often uses different methods for binary endpoints and continuous endpoints.
Business Use Only

28. References
Liublinska, Viktoriia Sensitivity Analyses in Empirical Studies Plagued with Missing Data. Doctoral dissertation, Harvard University.
Yan X, Lee S, Li N (2009) Missing data handling methods in medical device clinical trials. Journal of Biopharmaceutical Statistics 19: 1085–1098, available at accessed 23 June
Yongqiang Tang (2017) An Efficient Multiple Imputation Algorithm for Control-based and Delta-Adjusted Pattern Mixture Models using SAS, Statistics in Biopharmaceutical Research, 9:1,
Michael O’Kelly, Bohdana Ratitch (2014) Clinical trials with missing data : a guide for practitioners, ISBN: , John Wiley & Sons, Ltd
Tipping point analysis - multiple imputation for stress test under missing not at random (MNAR):
| Presentation Title | Presenter Name | Date | Subject | Business Use Only

30. Contact Hong (Kevin) Ding: Novartis Pharmaceuticals Corporation
One Health Plaza
East Hanover, NJ
USA
Phone    +1 
| Presentation Title | Presenter Name | Date | Subject | Business Use Only