Presentation on theme: "External Validity of Trials. Background External or ecological validity refers to whether the results of the trial can be generalised to the general clinical."— Presentation transcript:
External Validity of Trials
Background External or ecological validity refers to whether the results of the trial can be generalised to the general clinical population. Randomisation per se does NOT make a study externally valid.
Validity Issues Are the ‘correct’ patients being included in the trials? Are the ‘correct’ practitioners taking part in the study? Are the ‘correct’ facilities being used?
Trial Problems Many if not most trials are undertaken in circumstances that are far removed from ‘usual’ clinical practice. Usually clinicians running trials are content experts yet the results are to be applied by non-experts.
Osteoporosis Trials Most of the large osteoporosis trials with fracture as an outcome recruited participants from ‘bone’ clinics. Patients under care of clinical experts but results are expected to be applied by clinicians who are not experts.
Participants Usually participants who go into trials are different from those who do not. These ‘storm troopers’ of patients usually comply with instructions better than average. Combined with different clinicians and patients results can be different from the ‘real’ world.
Trial participants v non % (n) withTook part in trial n = 1 409 Did not take part in trial, n = 1 814 1 risk factor42.4 (597 ) 62.9 (1 141).000 2 risk factors40.9 (576)29.4 (533).000 3 risk factors13 (183)6.9 (126).000 4 risk factors3.8 (53)0.8 (14).000
Crude risk of fracture Type of fracture Took part in trial n = 1 409 Did not take part in trial, n = 1 814 P value Overall6.5%10.4%.000 Hip1.4%1.8%.405 Wrist2.5%3.8%.044
Risk of fracture (adjusted) Type of fracture OR (adjusted) 95% CI lower upper P value Overall0.420.31 0.55.000 Hip0.500.28 0.90.021 Wrist0.520.34 0.79.002
Alendronate Alendronate is a bisphosphonate for osteoporosis treatment. Large trials conducted in expert clinical centres showed it reduced fractures with few or no side-effects. Things were different when it was licensed and ‘real’ clinicians used the drug.
Alendronate problems To take a bisphosphonate one needs to take it on an empty stomach with a large glass of water and remain upright for at least 2 hours. Otherwise the drug can get affect the oesophagus and cause ulceration. In real life this is what happened and led to a warning. Trial data gave to evidence this was a problem.
The Wrong Solution Often once the phase III trial has been finished companies finance a ‘real’ world non-randomised study. This is usually nothing more than a marketing exercise and produces worthless data on ‘real’ world experience of the problem.
A correct solution Involves some form of randomisation to eliminate confounding. For example one might undertake a cluster trial of giving the drug non- specialist physicians to use of their patients BEFORE the drug is widely available. This will produce more robust data.
Alternatively We should try and design our trials to be as pragmatic as possible so that the results are widely generalisable. This means recruiting non-expert clinicians and their patients. Using a pragmatic design rather than an explanatory design.
Back Pain Trials An early RCT of chiropractic manipulation for back pain was criticised by physiotherapists that the chiropractic was undertaken in private practice. May have been a ‘Bach’ effect.
MRC back pain trial To address the issue of nice premises the MRC back pain trial included an arm where patients were randomised to be treated on private premises or NHS premises. Increased the costs of the trial hugely.
Surgical Trials Often surgical trials have poor external validity because the new technique is usually developed in a teaching hospital among ‘expert’ surgeons (e.g. laparoscopic methods). It is unlikely the results of surgery undertaken in ‘top’ hospitals would be the same as a ‘bog-standard’ DGH.
Big and Simple Try and make the trial large, as it allows for various PRE-SPECIFIED sub-group analyses (specialists vs generalists). Simple which means one can get as many in as possible.
Summary Most trials will NEVER be able to recruit exactly the same types of participants as those who receive treatment in clinical practice. PRAGMATIC trials have the strongest external validity and usually one of these should be done before implementation.