1. Change from baseline or analysis of covariance
Change from baseline or analysis of covariance?: Lord's Paradox and other matters.
Stephen Senn
Change from baseline or analysis of covariance?: Lord's Paradox and other matters.
In longitudinal studies it is generally agreed that analysis of covariance (ANCOVA) fitting baselines provides treatment estimates with lower variances than does a simple analysis of change scores (SACS). However, it also sometimes claimed that unless the means at baseline of groups being compared are equal, then such ANCOVA adjusted estimates are biased whereas SACS estimates are not. This would seem to imply that for cases where "Lord's paradox" applies, that is to say where SACS and ANCOVA may be expected to give different answers, the former should be preferred to the latter. In this talk I shall disagree with this point of view and show that cases where SACS is unbiased and ANCOVA is not are extremely difficult to construct in practice if a causal interpretation can still be given to the "treatment estimate" but that the reverse is not the case. This suggests that cases where SACS can be a good approach to analysis are few and far between.
(C) Stephen Senn 2004

2. Outline Adjustment in Randomised Clinical Trials Lord’s Paradox
The argument for ANCOVA
Lord’s Paradox
ANCOVA versus simple analysis of change scores (SACS)
Observational studies
The argument against ANCOVA
Resolution?
Why ANCOVA although not perfect may be best after all
(C) Stephen Senn 2004

3. SACS and ANCOVA
A simple randomised clinical trial in which there are two treatment groups and only two measurements per patient: a baseline measurement, X and an outcome measurement, Y.
Popular choices of outcome measure are
1) raw outcomes Y
2) change score d = Y - X
3) covariance adjusted outcomes Y - X. (where  is chosen appropriately)
NB As Laird (Am Stat., 37, , 1983) has shown, covariate adjusted change scores are the same as 3)
(C) Stephen Senn 2004

4. The Estimators Associated with the Measures
If subscript t stands for treatment and c for control we have: 1) 2) 3)
1) and 2) are just special cases of 3). If  is chosen to be the regression of Y on X, then 3) corresponds to analysis of covariance.
(C) Stephen Senn 2004

5. Warning These three measures, measure the same thing
No question of choosing between them on the basis of clinical relevance
Can only choose between them on the basis either of variance, or statistical philosophy
ANCOVA may generally be expected to have the lowest variance
Baseline is irrelevant to the definition of the treatment effect.
(C) Stephen Senn 2004

6. (C) Stephen Senn 2004

7. ANCOVA and Baseline by Treatment Interaction
It is often stated that ANCOVA relies on the parallelism assumption.
This is not true.
If the effect of treatment varies with baseline it varies whether or not ANCOVA is used.
ANCOVA is a first approximation and better than either doing nothing or using change scores.
(C) Stephen Senn 2004

8. Not to use ANCOVA, because you fear parallelism may not apply, is like saying crossing the channel in a rowing boat is dangerous I prefer to swim”.
(C) Stephen Senn 2004

9. Dichotomania Obsessive compulsive disorder
Cochrane Collaboration has a galloping case
Numbers Needed to Treat should have been strangled at birth
Division of patients into sheep and goats
Ignoring existence of geep and shoats
Use of difference from baseline
Sin number one
Destruction of information
Arbitrary division into responders non-responders
Sin number two
Further destruction of information
Unjustified causal interpretation
(C) Stephen Senn 2004

10. A Red Herring
It is sometimes claimed that measurement error invalidates ANCOVA
The reason is that if baseline is measured with error the regression of outcome on baseline is attenuated
However this claim is incorrect
ANCOVA is still valid
The reason is that it is appropriate to correct for an observed imbalance using an observed regression
(C) Stephen Senn 2004

11. Counter-Claims
There is a significant minority of papers arguing against ANCOVA as a means of dealing with bias
E.g. Liang and Zeger (2000), Sankyha, Samuelson (1986), American Statistician
The variance claims are accepted
Claims are made that ANCOVA is biased unless there is balance at baseline
(C) Stephen Senn 2004

12. Justification of the Counter-Claim
(C) Stephen Senn 2004

13. Lord’s Paradox
Lord, F.M. (1967) “ A paradox in the interpretation of group comparisons”, Psychological Bulletin, 68,
“A large university is interested in investigating the effects on the students of the diet provided in the university dining halls….Various types of data are gathered. In particular the weight of each student at the time of his arrival in September and his weight in the following June are recorded”
(C) Stephen Senn 2004

14. Two Statisticians Statistician One
Calculates difference in weight for each hall
Finds non-significant difference in each case
(Also no difference between halls)
Statistician Two
Adjust for initial weight
Finds significant hall effect
Concludes difference between halls
(C) Stephen Senn 2004

15. A Simulated Example
Starting and final weights for two groups of students
Males and females
300 In each group
Analysis illustrated with S Plus
(C) Stephen Senn 2004

16. (C) Stephen Senn 2004

17. Statistician One’s Analysis
Paired t-Test
data: Y.males and X.males
t = 0.662, df = 299, p-value =
data: Y.females and X.females
t = , df = 299, p-value =
Standard Two-Sample t-Test
data: diff.males and diff.females
t = , df = 598, p-value =
(C) Stephen Senn 2004

18. (C) Stephen Senn 2004

19. Statistician Two’s analysis
Call: lm(formula = Y ~ X + sex)
Coefficients:
Value Std. Error t value Pr(>|t|)
(Intercept) X
sex
(C) Stephen Senn 2004

20. What People Usually Conclude
Where baseline values are not expected to be equal between groups ANCOVA can mislead
Therefore even though SACS will have a higher variance it should be preferred under such circumstances since it is obviously unbiased
(C) Stephen Senn 2004

21. A Counter Counter-Example
Suppose we design a bizarre clinical trial
Only person with diastolic blood pressure at baseline equal to 95mmHg or 105mmHg may enter
In the first stratum they are randomised 3 to 1 and in the second 1 to 3
Situation as follows
(C) Stephen Senn 2004

22. A Stupid Trial Numbers of Patients by dbp and Treatment
Total
Baseline
diastolic
blood pressure
95mm Hg
300
100
400
105 mmHg
800
(C) Stephen Senn 2004

23. Approach to Analysis Stratify by baseline dbp
Produce treatment estimate for each stratum
Overall estimate is average of the two
Stratification deals with the imbalance
(C) Stephen Senn 2004

24. An Equivalent Approach
Create dummy variable stratum
S = -1 if baseline dbp, X = 95mmHg
S = 1 if baseline dbp, X =105 mmHg
Regress dbp at outcome, Y, on treatment indicator T and on stratum indicator S
Estimate will be same as by stratification
If you want variance estimate to be exactly the same you need to include interaction also
(C) Stephen Senn 2004

25. An Equivalent Equivalent Approach
Regress Y on T and X rather than on T and S
This is called ANCOVA!
Note that S= (X-100)/5
Hence this approach is equivalent to the previous one, which is equivalent to stratification, which is unbiased
On the other hand SACS is biased
Hence we have produced a counter-example
(C) Stephen Senn 2004

26. Conclusion
Contrary to what is often claimed there are cases where ANCOVA is unbiased but SACS is biased.
No simple statement of the form “ANCOVA is more efficient but SACS is unbiased” is possible.
In fact it is very difficult to imagine cases where SACS is the preferred analysis
(C) Stephen Senn 2004

27. Lord’s Paradox Revisited
Statistician one assumes that in the absence of any differential treatment effect the two groups despite different baselines would show equivalent changes
Statistician two assumes that in the absence of any differential treatment effect the change of the groups as a whole is the same as the change within groups
Both of these causal assumptions are untestable
(C) Stephen Senn 2004

28. However It is easy to design trials for which
ANCOVA is unbiased
SACS is biased
A causal interpretation can be given
It is rather difficult to design trials for which
SACS is unbiased
ANCOVA is biased
A causal interpretation can be given
(C) Stephen Senn 2004

29. The Necessary Condition for ANCOVA to be Unbiased
Or in everyday language that the bias in the raw comparison at outcome should be  times the bias at baseline where  is the individual regression effect
(C) Stephen Senn 2004

30. Cut-off Designs
Trochim and Capelleri have suggested that in many clinical trials randomisation will be unethical because some patients by the nature of their illness may be unwilling to assume the risks associated with an experimental treatment. They propose a class of designs called “cut-off” designs in which some patients are assigned to treatment in a deterministic manner on the basis of baseline values.
The position, for example, might be as given in the diagram below.
(C) Stephen Senn 2004

31. A Cut-off Design in Hypertension
Randomise to
standard or
experimental
Experimental
treatment only
Standard
treatment only
Mild hypertension
Moderate hypertension
Severe hypertension
(C) Stephen Senn 2004

32. Cut off Designs
Provided that the relationship between baseline and outcome is linear ANCOVA is valid
Cut off designs are thus a wide class of design for which ANCOVA is unbiased
SACS will be biased
Thus we have more counterexamples to the claims of Liang and Zeger
(C) Stephen Senn 2004

33. A Challenge Can you design a trial for which SACS is unbiased
ANCOVA is biased
A causal interpretation can be given?
(C) Stephen Senn 2004

34. Some Schemes That won’t Work
Select patient according to true baseline values
Not possible in practice since not known
Still won’t work since correlation of true values is not 1
Select patients according to average of values at baseline and outcome
You need a crystal ball
Select according to some other value
ANCOVA will be biased but so will SACS
Select on binary covariate
Either this is permanent (e.g. sex), in which case causal inference doubtful
Or it varies over time in which case there will be a regression
(C) Stephen Senn 2004

35. Conclusion
In RCTs ANCOVA is the appropriate way to use baseline information
SACS, responder analysis, NNTs all wasteful
A hallmark of second rate analysis
In observational studies things are more complex
ANCOVA may not be perfect but it may be the best you can do
(C) Stephen Senn 2004

36. Here there be tygers!
(C) Stephen Senn 2004