1. Francis KL Chan Department of Medicine & Therapeutics CUHK
Design & Interpretation of Randomized Trials: A Clinician’s Perspective
Francis KL Chan
Department of Medicine & Therapeutics
CUHK

2. Common problems of RCTs
Originality
Hypothesis
Allocation concealment & randomization
Evaluation of baseline data
“Intention-to-treat” analysis
Subgroup analysis

3. Is the study original? Ground breaking research?
Does this work add to the literature in any way?
Bigger, longer?
More rigorous methodology?
Results add to a meta-analysis of previous studies?
Different population (age, sex, ethnic groups)?

4. Hypothesis & End point Many RCTs did not explicitly state their study
hypotheses
“The aim of this study was to compare the efficacy of
a new treatment with the standard treatment…”
Hypothesis 1:
Treatment A is superior to the standard treatment
Hypothesis 2:
Treatment A is equivalent to the standard treatment

5. Sample size estimation
None!
Hypothesis?

6. Failure to detect a difference=
Equivalence?

7. Superiority Trial Test hypothesis (H): µN - µS> 
The new treatment (µN) is superior to the standard treatment (µS) if the difference exceeds by a clinically important amount ().
Test hypothesis (H): µN - µS> 

8. Equivalence trial
The new treatment is equivalent to the standard treatment if the maximal allowable difference does not exceed by a clinically important amount.

9. Equivalence trial - 0 + Equivalent Difference
New agent is not inferior to the standard
Not equivalent
Favors standard treatment
Not equivalent
Favors new treatment
- 
Difference

10. Assume non-inferiority if the lower limit of 95% CI is less than –5%,
N=904 per group!

11. Allocation concealment & randomization
Concealment of allocation
(investigators and patients not knowing the assigned treatment before randomization)
Was treatment assigned by an independent staff?
What was the method of allocation concealment?
contact with central office
blinded packages
sealed (opaque) envelopes

12. Allocation concealment

13. Does P>0.05 indicate comparability of treatment groups?
Comparison of baseline data
Does P>0.05 indicate comparability of treatment groups?
Chan et al. Lancet 1997

14. Baseline data
Effect of azathioprine on the survival of patients with primary biliary cirrhosis
Azathioprine
Placebo
Mean age
54.7
54.9
Serum bilirubin (mol/L)
37.2
30.9
Stage I disease % 14 12
Stage II disease % 44 43
Stage III disease % 15
Stage IV disease % 27 30
Christensen et al. Gastro 1985

15. Baseline data
Effect of azathioprine on the survival of patients with primary biliary cirrhosis
Azathioprine
Placebo
Mean age
54.7
54.9
Serum bilirubin (mol/L)
37.2
30.9
Stage I disease % 14 12
Stage II disease % 44 43
Stage III disease % 15
Stage IV disease % 27 30
Christensen et al. Gastro 1985

16. Adjusted for bilirubin
UnadjustedP=0.10
Adjusted for bilirubin
P=0.01

17. P=0.04
P=0.02
Columbus Investigators. NEJM 1997

18. Comparison of baseline data
Significant imbalance may not affect outcome
Non-significant imbalance may affect outcome
Significance tests for baseline differences are inappropriate.

19. Significance tests for baseline differences
INAPPROPRIATE
Chan et al. Lancet 1997

20. Comparison of baseline data
Significant imbalance may not affect outcome
Non-significant imbalance may affect outcome
Significance tests for baseline differences are inappropriate.
Table of baseline data should focus on factors affecting outcome.

21. 45 baseline factors!

22. Comparison of baseline data
Significant imbalance may not affect outcome
Non-significant imbalance may affect outcome
Significance tests for baseline differences are inappropriate.
Table of baseline data should focus on factors affecting outcome.
Analysis adjusted for baseline factors that are known to strongly influence the outcome (Covariate-adjusted analysis).
Analysis of covariance for a quantitative outcome
Logistic regression for a binary response
Cox’s-proportional hazard model for time-to-event data

23. “Intention-To-Treat” Analysis
“…results were analyzed according to the ITT principle.”
Question:
How were missing outcomes/ protocol violators handled in the so called “ITT” analysis?

24. “Intention-To-Treat” Analysis
Endpt
Savage et al. NEJM 1997

25. Recommendations for ITT Analysis
Minimize missing response on primary outcome
Follow up subjects who withdraw early
Report all deviations and missing response
Investigate & report the effect of missing response

26. Subgroup Analysis
Randomised trial of home-based psychosocial nursing intervention for patients recovering from myocardial infarction. Frasure-Smith et al. Lancet 1997
“…The poor overall outcome for women, and the possible harmful impact of the intervention on women, underlie the need for…”

27. Subgroup Analysis
Effect of antenatal dexamethasone administration on the prevention of respiratory distress syndrome.
Am J Obstet Gynecol 1981;141:
Steroid
Placebo P
Pre-ecclampsia
21.2%
(7/33)
27.3%
(9/33)
0.57
No pre-ecclampsia
7.9%
(21/267)
14.1%
(37/262)
0.021

28. Subgroup Analysis
Effect of antenatal dexamethasone administration on the prevention of respiratory distress syndrome.
Am J Obstet Gynecol 1981;141:
Difference 6.1%
Steroid
Placebo P
Pre-ecclampsia
21.2%
(7/33)
27.3%
(9/33)
0.57
No pre-ecclampsia
7.9%
(21/267)
14.1%
(37/262)
0.021

29. P value depends on effect size & SE
Subgroup Analysis
Effect of antenatal dexamethasone administration on the prevention of respiratory distress syndrome.
Am J Obstet Gynecol 1981;141:
P value depends on effect size & SE
Difference 6.1%
Steroid
Placebo P
Pre-ecclampsia
21.2%
(7/33)
27.3%
(9/33)
0.57
No pre-ecclampsia
7.9%
(21/267)
14.1%
(37/262)
0.021
Difference 6.2%

30. Evaluation of Subgroup Analysis
Tests of interaction (assess whether a treatment effect differs between subgroups) rather than subgroup P values
Diff in Subgroup A – Diff in Subgroup B
SE of the above Diff
Z =

31. Trial of vitamin D supplements in pregnancy to prevent infant hypocalcemia. BMJ 1980;281:11-4.
Interaction Test
Difference = 0.42
Difference = 0.15
No evidence that the effect of Vit D is different between bottle-fed and breast-fed infants
0.42 – 0.15 = 0.27
SE of this Diff = 0.22
Z = Diff / SE = 1.23
P = 0.2

32. General points regarding subgroup analysis
Emphasis should remain on overall comparison
More convincing if confined to a limited number of pre-specified subgroup hypothesis
Rely on interaction tests, not P values
View subgroup findings as exploratory (to be confirmed in subsequent trials)