Presentation is loading. Please wait.

Presentation is loading. Please wait.

Randomized Controlled Trials: Allocation Concealment

Similar presentations


Presentation on theme: "Randomized Controlled Trials: Allocation Concealment"— Presentation transcript:

1 Randomized Controlled Trials: Allocation Concealment
Kenneth F. Schulz, PhD

2 Fixation on Sequence Generation
Traditionally, many medical researchers mistakenly consider the sequence generation process as “randomization” They frequently ignore allocation concealment Without adequate allocation concealment, proper random sequences can be subverted For example, a properly generated sequence posted on a bulletin board

3 Allocation Concealment
Prevents selection bias by concealing the allocation sequence from those assigning participants to intervention groups Until the moment of assignment

4 Mechanisms for Selection Bias in RCTs
If those responsible for entering participants know, or can detect, the upcoming treatment allocations: Decide entrance based on that knowledge Channel participants with a better prognosis to the experimental group and those with a poorer prognosis to the control group, or vice versa In any case, they introduce bias into the treatment comparison

5 Do Not Confuse Allocation Concealment with Blinding
Allocation concealment seeks to prevent selection bias, protects assignment sequence before and until allocation, and can always be successfully implemented In contrast, blinding seeks to prevent ascertainment bias, protects sequence after allocation, and cannot always be successfully implemented

6 Importance of Allocation Concealment
Unclearly concealed and inadequately concealed trials, compared to adequately concealed trials, exaggerated the estimates of an intervention’s effectiveness by 30% to 40%, on average Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273:

7 Similar Results on Allocation Concealment in a Separate Study
Replicated our methods in different subject areas (digestive & circulatory diseases and mental health) Examined 11 meta-analyses Inadequately concealed trials exaggerated estimates of effectiveness by 37% [Moher D, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 1998; 352: ]

8 Allocation Concealment: Summary
Adequate allocation concealment emerges from our analyses as crucial to reducing bias Without it, the whole point of randomisation vanishes and bias may distort results Our results support Mosteller and his colleagues “When the randomization leaks, the trial’s guarantee of lack of bias runs down the drain”

9 Dickinson K, et al., BMJ 2000;320:1308-1311.
Method of allocation of treatment in 208 controlled trials in head injury Dickinson K, et al., BMJ 2000;320: Method of allocation No. of trials Adequate Centralised randomisation by telephone 1 Numbered/coded identical containers administered sequentially 11 Randomisation scheme controlled by pharmacy 8 Sequentially numbered, sealed, opaque envelopes 2 Not adequate Other 18 Date of birth Day of week 3 Alternation Not stated 161

10 Actual allocation method with the method just specified as “random”
Difficult (Peto and I. Chalmers experience) MA of OCs David G. knew the PI who put him in contact with the statistician Described a “great” method Post-it notes! Seemed indignant We asked for clarification Perfect also for stratification—different colors!!

11 Personal Accounts of Deciphering Assignment Sequences
Conducted over 20 epidemiological workshops for medical residents and medical junior faculty Each included participants Asked how many of the participants had deciphered, or had witnessed someone else decipher, an assignment sequence

12 Personal Accounts of Deciphering Assignment Sequences (cont.)
With assurance of anonymity, more than half related at least one instance of deciphering Not half of all trials No accurate denominator Nevertheless, not a rare occurrence

13 Decipherings Run the Gamut From Simple to Intricate
Simple the most frequent and usually take advantage of inadequate allocation concealment schemes: Posting sequence on a bulletin board Opening unsealed envelopes Translucent envelopes to a light bulb Opening many envelopes that were not sequentially numbered

14 Intricate Decipherings Less Frequent
Needed to circumvent more adequate allocation schemes With sequentially numbered drug containers Based on appearance of tablets in unsealed containers Appearance of the label With central allocation, obtaining the next few allocations at once Envelopes to a “hot light” in radiology

15 Attempted to Decipher a Numbered Container Scheme
Gave up Found attending physician rifling the P.I.’s files for the assignment sequence Horrified? No, impressed with his brilliance and proceeded to help Should be kept in a locked location

16 Ignorance Is Not Bliss “At the annual meeting of the Society for Clinical Trials, held in Houston, … Schulz reported data showing that the experts’ fears about ignorance are justified. Schulz examined 250 reports of clinical trials and came up with an intriguing finding…” [Rachel Nowak. Problems in clinical trials go far beyond misconduct. Science 1994; 264: 1539.]

17 Ignorance Is Not Bliss “Most physicians are not trained in basic scientific principles, let alone clinical trials” Stephen George, Duke University, Chair of the Statistics Committee for NCI, NIH Cancer Clinical Cooperative Groups “I fantasize that in the future (training) will become a requirement for running clinical trials” John Gallin, Director of NIH’s Clinical Center

18 Rigorous Trials Annoy Humans
Investigators Certain Ps to benefit May want the results of study to reveal the “truth” Proper trial procedures attempt to impede human inclinations Hence, aspects of properly conducted RCTs annoy investigators

19 Without Methodological Rigour
The challenge of deciphering may frequently become too great a temptation to resist Even without intent to bias “The only way to get rid of a temptation is to yield to it.” Oscar Wilde Deciphering may just reflect human inquisitiveness and ingenuity Does not necessarily indicate scientific malevolence

20 Without Methodological Rigour
Or deciphering may reflect deliberate acts to alter findings Whatever the motivations, innocent or naïve or deliberate, those actions undermine the validity of the trial Investigators must devote diligent, persistent attention to randomization

21 Minimal Standards: Common Allocation Concealment Approaches
Sequentially numbered, opaque, sealed envelopes (SNOSE) Pharmacy control Numbered or coded containers Central randomization Realistically, these standards should be exceeded

22 Envelopes for Allocation Concealment
More susceptible to manipulation through human ingenuity Less than ideal method of allocation concealment If used, investigators must diligently develop and monitor the process

23 Envelopes (Cont.) SNOSE
Ensure that the envelopes are opened sequentially only after the P’s name and other details are written on the appropriate envelope

24 Envelopes (Cont’d) Pressure-sensitive or carbon paper inside
Supports enrolment Encourages orderly opening Promotes proper assignment Creates valuable audit trail Cardboard or aluminum foil inside envelope

25 “Pharmacy Controlled”
Compliance with randomization/allocation concealment methods questionable Aware of gross distortions Pharmacy ran out of one drug . . . Alternate assignment Should not assume pharmacists knowledge Ensure they, and all research partners, follow proper trial procedures

26 “Numbered Containers”
Excellent – prevents foreknowledge Must take proper precautions Appearance – exactly the same Weight – exactly the same Sound when shaken – exactly the same Lettering – exactly the same Odor – exactly the same Sealed Difference between concealment and blinding

27 “Central Randomization”
Frequently telephone Fax Internet “Computer” Assignments only provided after participants irrevocably enrolled Must check procedures for leaks Monitor process for adherence

28 RCTs: Anathema to the Human Spirit
Must acknowledge the vagaries of human nature Must establish methodological safeguards that thwart attempts to contaminate trials with bias

29 Subversions of Sequences
Conscious or subconscious? Direct or compensatory? Doesn’t matter Biases the trial

30 Participants Randomize Placebo New Oral Drug Outcome Outcome
40% Non-compliance 60% Compliance 25% Non-compliance 75% Compliance Group representing the policy of no treatment Group representing the policy of oral treatment Outcome Outcome

31 RCT Compared the Effectiveness of Clofibrate in Preventing Cardiac Deaths in Men Who Had Survived a Myocardial Infarction Clofibrate Placebo 5 Year mortality 20.2% 20.9% (p = .55) Eliminating deviates from clofibrate (80% adherence) 15.0% 20.9% (p < .05) Eliminating deviates from both groups 15.0% 15.1% Authors state that: One can justify almost any conclusion, dependent upon the analysis chosen Manipulating deviates leads to severe bias Can you ever do so?

32 Exclusions After Randomization
Can introduce bias and should be carefully scrutinized All randomized patients should be analyzed, and analyzed as part of the group to which they were initially assigned ITT (Intention-to-treat)

33 Exclusions of LFU Damage Internal Validity
Without outcomes from those lost to follow-up, investigators have little choice but to exclude them from the analysis Any losses damage internal validity However, differential rates of loss among comparison groups cause major damage Investigators must minimize their losses to follow-up

34 Retention of Trial Participants
Minimizing losses exudes difficulties Eliminating losses may be impossible But investigators too frequently profess insurmountable difficulties Most should work harder to obtain higher follow-up rates Investigators must commit adequate attention and resources to develop and implement procedures to minimize losses

35 Innovative Twists that Cultivate High Follow-up Rates
Establish many conveniently placed FU facilities Too often investigators expect participants to visit a single, inconvenient location Shortening data collection form caters to the participants’ wishes Long sessions lead them to vote with their feet Be considerate Foster follow-up by not overburdening Also may engender higher quality data on the main items

36 END

37 Meta-analysis of Polyglycolic
Meta-analysis of polyglycolic acid versus catgut for perineal repair on short term pain 1 adequately concealed trial … OR = .89 4 unclearly concealed trials … OR = .44 Protective effect of polyglycotic acid was 51% lower in the unclearly concealed trial (ROR = .49; 95% CI of )

38 Meta-analysis of Corticosteriods
Meta-analysis of corticosteroids vs. none, after preterm rupture of membranes, on respiratory distress syndrome (RDS) 3 adequately concealed trials … OR = 0.72 4 unclearly concealed trials … OR = 0.53 Protective effect of corticosteroids was exaggerated by 27% in the unclearly concealed trials ROR = 0.73; 95% CI of

39 Randomization Process (P 138B)
Proper approach to: 4 OB/GYN Journals 4 General Medical Journals Method of generation of random numbers 32% 49% Randomization concealment 23% 26% Both 9% 15% In the 4 OB/GYN Journals, found the 2 treatment group sizes to be much too similar Expected Observed 5 ( 5%) 48 (50%) 8

40 Separation of Executer and Generator
Person(s) who prepare the randomization scheme should not determine eligibility, administer treatment, or assess outcome Neglected critical element Obviously important—regardless of methodological quality, access to sequence and thus opportunity to introduce bias Faults in this element may be the crack though which much of the bias seeps into controlled trials

41 “Not quite as random as I pretended”
[Martyn C. Not quite as random as I pretended. Lancet 1996; 347:70.] “Bias in the way treatments are assigned can be a more powerful determinant of the outcome of the trial than the treatments that are being investigated.” “Trialists … often find it hard to prevent their hopes that a new treatment might prove to be beneficial from evolving into a belief that it really is so.”

42 Insert published article
RCT in Norway Examined the effect of Chlorhexidine compared to placebo on postpartum infection and Strep B infections in neonates Reported as a randomized controlled trial in the published article Insert published article

43 RCT in Norway Midwives knew on which shelf the Chlorhexidine and placebo were kept in the cabinet and just proceeded to administer whatever they deemed appropriate Even if they forgot the shelf, they could smell the difference No monitoring of the allocation process

44 Letter from a Physical Therapist
A “friend of mine” conducted an RCT comparing 2 training regimens following surgical repair of the knee for anterior cruciate ligament damage “randomization method used was alternating assignment” Referred to one patient as “difficult” and asked the friend to which group she was assigned Friend responded “that she was in the group that required slightly less contact and that her inclusion was by his choice”

45 “Not so blind, after all”
Sometimes subverting can work against a treatment’s apparent effectiveness “Medical staff … may try to help their sickest patients by sneaking them into the treatment group instead of the control group.” “The drug would then have to be significantly better than conventional treatment just to appear equal in efficacy” [Wallich P. Not so blind after all. Scientific American; May 1996.]

46 RCTs: Anathema to the Human Spirit
Scientific need to obtain unbiased data from an inherently biased source – human beings Many trialists are unaware of the rationale behind RCTs And the need for trial rigor Some may intellectually grasp the need, but once immersed in a trial, have many contradictory interests (e.g. episiotomy trial in Canada)

47 Anathema: A Psychologist’s View
”Fundamental conflict” between scientific and human imperatives … “Many probably think they already know what treatment is best.” Geoffrey Scobie, psychologist at the University of Glasgow; Doctors cheat because they know what’s best New Scientist, 16 December 95, p. 10 Of course, what we think we ‘know’ is frequently wrong

48 Inadequately Concealed
Reporting of Exclusions, Double-blinding, and Schedule Generation Related to the Level of Allocation Concealment for 250 Trials Authors Reported Adequately Concealed (n=79) Unclearly Concealed (n=150) Inadequately Concealed (n=21) No Exclusions (Apparent) 53% 67% 76% Double-blinding 73% 39% 14% Adequate generation of schedule 29% 15% 0%


Download ppt "Randomized Controlled Trials: Allocation Concealment"

Similar presentations


Ads by Google