Presentation is loading. Please wait.

Presentation is loading. Please wait.

Randomized Controlled Trials: Allocation Concealment Kenneth F. Schulz, PhD.

Similar presentations


Presentation on theme: "Randomized Controlled Trials: Allocation Concealment Kenneth F. Schulz, PhD."— Presentation transcript:

1 Randomized Controlled Trials: Allocation Concealment Kenneth F. Schulz, PhD

2 Fixation on Sequence Generation Traditionally, many medical researchers mistakenly consider the sequence generation process as randomization Traditionally, many medical researchers mistakenly consider the sequence generation process as randomization They frequently ignore allocation concealment They frequently ignore allocation concealment Without adequate allocation concealment, proper random sequences can be subverted Without adequate allocation concealment, proper random sequences can be subverted For example, a properly generated sequence posted on a bulletin board For example, a properly generated sequence posted on a bulletin board

3 Allocation Concealment Prevents selection bias by concealing the allocation sequence from those assigning participants to intervention groups Prevents selection bias by concealing the allocation sequence from those assigning participants to intervention groups Until the moment of assignment Until the moment of assignment

4 Mechanisms for Selection Bias in RCTs If those responsible for entering participants know, or can detect, the upcoming treatment allocations: If those responsible for entering participants know, or can detect, the upcoming treatment allocations: Decide entrance based on that knowledge Decide entrance based on that knowledge Channel participants with a better prognosis to the experimental group and those with a poorer prognosis to the control group, or vice versa Channel participants with a better prognosis to the experimental group and those with a poorer prognosis to the control group, or vice versa In any case, they introduce bias into the treatment comparison In any case, they introduce bias into the treatment comparison

5 Do Not Confuse Allocation Concealment with Blinding Allocation concealment seeks to prevent selection bias, protects assignment sequence before and until allocation, and can always be successfully implemented Allocation concealment seeks to prevent selection bias, protects assignment sequence before and until allocation, and can always be successfully implemented In contrast, blinding seeks to prevent ascertainment bias, protects sequence after allocation, and cannot always be successfully implemented In contrast, blinding seeks to prevent ascertainment bias, protects sequence after allocation, and cannot always be successfully implemented

6 Importance of Allocation Concealment Unclearly concealed and inadequately concealed trials, compared to adequately concealed trials, exaggerated the estimates of an interventions effectiveness by 30% to 40%, on average Unclearly concealed and inadequately concealed trials, compared to adequately concealed trials, exaggerated the estimates of an interventions effectiveness by 30% to 40%, on average Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273:

7 Similar Results on Allocation Concealment in a Separate Study Replicated our methods in different subject areas (digestive & circulatory diseases and mental health) Replicated our methods in different subject areas (digestive & circulatory diseases and mental health) Examined 11 meta-analyses Examined 11 meta-analyses Inadequately concealed trials exaggerated estimates of effectiveness by 37% Inadequately concealed trials exaggerated estimates of effectiveness by 37% [Moher D, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta- analyses? Lancet 1998; 352: ]

8 Allocation Concealment: Summary Adequate allocation concealment emerges from our analyses as crucial to reducing bias Adequate allocation concealment emerges from our analyses as crucial to reducing bias Without it, the whole point of randomisation vanishes and bias may distort results Without it, the whole point of randomisation vanishes and bias may distort results Our results support Mosteller and his colleagues Our results support Mosteller and his colleagues When the randomization leaks, the trials guarantee of lack of bias runs down the drain

9 Method of allocation of treatment in 208 controlled trials in head injury Dickinson K, et al., BMJ 2000;320: Method of allocation No. of trials Adequate Centralised randomisation by telephone1 Numbered/coded identical containers administered sequentially 11 Randomisation scheme controlled by pharmacy8 Sequentially numbered, sealed, opaque envelopes2 Not adequate Other 18 Date of birth1 Day of week3 Alternation3 Not stated161

10 Actual allocation method with the method just specified as random Difficult (Peto and I. Chalmers experience) Difficult (Peto and I. Chalmers experience) MA of OCs MA of OCs David G. knew the PI who put him in contact with the statistician David G. knew the PI who put him in contact with the statistician Described a great method... Post-it notes! Described a great method... Post-it notes! Seemed indignant... We asked for clarification Seemed indignant... We asked for clarification Perfect also for stratificationdifferent colors!! Perfect also for stratificationdifferent colors!!

11 Personal Accounts of Deciphering Assignment Sequences Conducted over 20 epidemiological workshops for medical residents and medical junior faculty Conducted over 20 epidemiological workshops for medical residents and medical junior faculty Each included participants Each included participants Asked how many of the participants had deciphered, or had witnessed someone else decipher, an assignment sequence Asked how many of the participants had deciphered, or had witnessed someone else decipher, an assignment sequence

12 With assurance of anonymity, more than half related at least one instance of deciphering With assurance of anonymity, more than half related at least one instance of deciphering Not half of all trials Not half of all trials No accurate denominator No accurate denominator Nevertheless, not a rare occurrence Nevertheless, not a rare occurrence Personal Accounts of Deciphering Assignment Sequences (cont.)

13 Decipherings Run the Gamut From Simple to Intricate Simple the most frequent and usually take advantage of inadequate allocation concealment schemes: Simple the most frequent and usually take advantage of inadequate allocation concealment schemes: Posting sequence on a bulletin board Posting sequence on a bulletin board Opening unsealed envelopes Opening unsealed envelopes Translucent envelopes to a light bulb Translucent envelopes to a light bulb Opening many envelopes that were not sequentially numbered Opening many envelopes that were not sequentially numbered

14 Needed to circumvent more adequate allocation schemes Needed to circumvent more adequate allocation schemes With sequentially numbered drug containers With sequentially numbered drug containers Based on appearance of tablets in unsealed containers Based on appearance of tablets in unsealed containers Appearance of the label Appearance of the label With central allocation, obtaining the next few allocations at once With central allocation, obtaining the next few allocations at once Envelopes to a hot light in radiology Envelopes to a hot light in radiology Intricate Decipherings Less Frequent

15 Attempted to Decipher a Numbered Container Scheme Gave up Gave up Found attending physician rifling the P.I.s files for the assignment sequence Found attending physician rifling the P.I.s files for the assignment sequence Horrified? No, impressed with his brilliance and proceeded to help Horrified? No, impressed with his brilliance and proceeded to help Should be kept in a locked location Should be kept in a locked location

16 Ignorance Is Not Bliss At the annual meeting of the Society for Clinical Trials, held in Houston, … Schulz reported data showing that the experts fears about ignorance are justified. Schulz examined 250 reports of clinical trials and came up with an intriguing finding… [Rachel Nowak. Problems in clinical trials go far beyond misconduct. Science 1994; 264: 1539.]

17 Ignorance Is Not Bliss Most physicians are not trained in basic scientific principles, let alone clinical trials I fantasize that in the future (training) will become a requirement for running clinical trials John Gallin, Director of NIHs Clinical Center Stephen George, Duke University, Chair of the Statistics Committee for NCI, NIH Cancer Clinical Cooperative Groups

18 Rigorous Trials Annoy Humans Investigators Investigators Certain Ps to benefit Certain Ps to benefit May want the results of study to reveal the truth May want the results of study to reveal the truth Proper trial procedures attempt to impede human inclinations Proper trial procedures attempt to impede human inclinations Hence, aspects of properly conducted RCTs annoy investigators Hence, aspects of properly conducted RCTs annoy investigators

19 Without Methodological Rigour The challenge of deciphering may frequently become too great a temptation to resist The challenge of deciphering may frequently become too great a temptation to resist Even without intent to bias Even without intent to bias The only way to get rid of a temptation is to yield to it. Oscar Wilde The only way to get rid of a temptation is to yield to it. Oscar Wilde Deciphering may just reflect human inquisitiveness and ingenuity Deciphering may just reflect human inquisitiveness and ingenuity Does not necessarily indicate scientific malevolence Does not necessarily indicate scientific malevolence

20 Without Methodological Rigour Or deciphering may reflect deliberate acts to alter findings Or deciphering may reflect deliberate acts to alter findings Whatever the motivations, innocent or naïve or deliberate, those actions undermine the validity of the trial Whatever the motivations, innocent or naïve or deliberate, those actions undermine the validity of the trial Investigators must devote diligent, persistent attention to randomization Investigators must devote diligent, persistent attention to randomization

21 Minimal Standards: Common Allocation Concealment Approaches Sequentially numbered, opaque, sealed envelopes (SNOSE) Sequentially numbered, opaque, sealed envelopes (SNOSE) Pharmacy control Pharmacy control Numbered or coded containers Numbered or coded containers Central randomization Central randomization Realistically, these standards should be exceeded Realistically, these standards should be exceeded

22 Envelopes for Allocation Concealment More susceptible to manipulation through human ingenuity More susceptible to manipulation through human ingenuity Less than ideal method of allocation concealment Less than ideal method of allocation concealment If used, investigators must diligently develop and monitor the process If used, investigators must diligently develop and monitor the process

23 Envelopes (Cont.) only after the Ps name and other details are written on the appropriate envelope only after the Ps name and other details are written on the appropriate envelope SNOSE SNOSE Ensure that the envelopes are opened sequentially Ensure that the envelopes are opened sequentially

24 Envelopes (Contd) Pressure-sensitive or carbon paper inside Pressure-sensitive or carbon paper inside Supports enrolment Supports enrolment Encourages orderly opening Encourages orderly opening Promotes proper assignment Promotes proper assignment Creates valuable audit trail Creates valuable audit trail Cardboard or aluminum foil inside envelope Cardboard or aluminum foil inside envelope

25 Pharmacy Controlled Compliance with randomization/allocation concealment methods questionable Compliance with randomization/allocation concealment methods questionable Aware of gross distortions Aware of gross distortions Pharmacy ran out of one drug... Pharmacy ran out of one drug... Alternate assignment Alternate assignment Should not assume pharmacists knowledge Should not assume pharmacists knowledge Ensure they, and all research partners, follow proper trial procedures Ensure they, and all research partners, follow proper trial procedures

26 Numbered Containers Excellent – prevents foreknowledge Excellent – prevents foreknowledge Must take proper precautions Must take proper precautions Appearance – exactly the same Appearance – exactly the same Weight – exactly the same Weight – exactly the same Sound when shaken – exactly the same Sound when shaken – exactly the same Lettering – exactly the same Lettering – exactly the same Odor – exactly the same Odor – exactly the same Sealed Sealed Difference between concealment and blinding Difference between concealment and blinding

27 Central Randomization Frequently telephone Frequently telephone Fax Fax Internet Internet Computer Computer Assignments only provided after participants irrevocably enrolled Assignments only provided after participants irrevocably enrolled Must check procedures for leaks Must check procedures for leaks Monitor process for adherence Monitor process for adherence

28 RCTs: Anathema to the Human Spirit Must acknowledge the vagaries of human nature Must acknowledge the vagaries of human nature Must establish methodological safeguards that thwart attempts to contaminate trials with bias Must establish methodological safeguards that thwart attempts to contaminate trials with bias

29 Subversions of Sequences Conscious or subconscious? Conscious or subconscious? Direct or compensatory? Direct or compensatory? Doesnt matter Doesnt matter Biases the trial Biases the trial

30 Participants Outcome Outcome Outcome Outcome Placebo 40% Non- compliance 60% Compliance Group representing the policy of no treatment 25% Non- compliance 75% Compliance Group representing the policy of oral treatment New Oral Drug Randomize

31 RCT Compared the Effectiveness of Clofibrate in Preventing Cardiac Deaths in Men Who Had Survived a Myocardial Infarction ClofibratePlacebo 5 Year mortality 20.2%20.9% (p =.55) Authors state that: One can justify almost any conclusion, dependent upon the analysis chosenOne can justify almost any conclusion, dependent upon the analysis chosen Manipulating deviates leads to severe biasManipulating deviates leads to severe bias Can you ever do so?Can you ever do so? Eliminating deviates from clofibrate (80% adherence) 15.0%20.9% (p <.05) Eliminating deviates from both groups 15.0%15.1%

32 Exclusions After Randomization Can introduce bias and should be carefully scrutinized Can introduce bias and should be carefully scrutinized All randomized patients should be analyzed, and analyzed as part of the group to which they were initially assigned All randomized patients should be analyzed, and analyzed as part of the group to which they were initially assigned ITT (Intention-to-treat) ITT (Intention-to-treat)

33 Exclusions of LFU Damage Internal Validity Without outcomes from those lost to follow-up, investigators have little choice but to exclude them from the analysis Without outcomes from those lost to follow-up, investigators have little choice but to exclude them from the analysis Any losses damage internal validity Any losses damage internal validity However, differential rates of loss among comparison groups cause major damage However, differential rates of loss among comparison groups cause major damage Investigators must minimize their losses to follow-up Investigators must minimize their losses to follow-up

34 Retention of Trial Participants Minimizing losses exudes difficulties Minimizing losses exudes difficulties Eliminating losses may be impossible Eliminating losses may be impossible But investigators too frequently profess insurmountable difficulties But investigators too frequently profess insurmountable difficulties Most should work harder to obtain higher follow-up rates Most should work harder to obtain higher follow-up rates Investigators must commit adequate attention and resources to develop and implement procedures to minimize losses Investigators must commit adequate attention and resources to develop and implement procedures to minimize losses

35 Innovative Twists that Cultivate High Follow-up Rates Establish many conveniently placed FU facilities Establish many conveniently placed FU facilities Too often investigators expect participants to Too often investigators expect participants to visit a single, inconvenient location Shortening data collection form caters to the participants wishes Shortening data collection form caters to the participants wishes Long sessions lead them to vote with their feet Long sessions lead them to vote with their feet Be considerate Be considerate Foster follow-up by not overburdening Foster follow-up by not overburdening Also may engender higher quality data on the Also may engender higher quality data on the main items

36 END

37 Meta-analysis of polyglycolic acid versus catgut for perineal repair on short term pain 1 adequately concealed trial … OR =.89 4 unclearly concealed trials … OR =.44 Protective effect of polyglycotic acid was 51% lower in the unclearly concealed trial (ROR =.49; 95% CI of ) Meta-analysis of Polyglycolic

38 Meta-analysis of corticosteroids vs. none, after preterm rupture of membranes, on respiratory distress syndrome (RDS) 3 adequately concealed trials … OR = unclearly concealed trials … OR = 0.53 Protective effect of corticosteroids was exaggerated by 27% in the unclearly concealed trials ROR = 0.73; 95% CI of Meta-analysis of Corticosteriods

39 Randomization Process (P 138B) In the 4 OB/GYN Journals, found the 2 treatment group sizes to be much too similar Expected Expected Observed Observed 5 ( 5%) 5 ( 5%)0 48 (50%) 48 (50%)8 Proper approach to: 4 OB/GYN Journals 4 General Medical Journals Method of generation of random numbers 32%49% Randomization concealment 23%26% Both 9%15%

40 Separation of Executer and Generator Person(s) who prepare the randomization scheme should not determine eligibility, administer treatment, or assess outcome Person(s) who prepare the randomization scheme should not determine eligibility, administer treatment, or assess outcome Neglected critical element Neglected critical element Obviously importantregardless of methodological quality, access to sequence and thus opportunity to introduce bias Obviously importantregardless of methodological quality, access to sequence and thus opportunity to introduce bias Faults in this element may be the crack though which much of the bias seeps into controlled trials Faults in this element may be the crack though which much of the bias seeps into controlled trials

41 Not quite as random as I pretended [Martyn C. Not quite as random as I pretended. Lancet 1996; 347:70.] Bias in the way treatments are assigned can be a more powerful determinant of the outcome of the trial than the treatments that are being investigated. Trialists … often find it hard to prevent their hopes that a new treatment might prove to be beneficial from evolving into a belief that it really is so. Trialists … often find it hard to prevent their hopes that a new treatment might prove to be beneficial from evolving into a belief that it really is so.

42 RCT in Norway Examined the effect of Chlorhexidine compared to placebo on postpartum infection and Strep B infections in neonates Examined the effect of Chlorhexidine compared to placebo on postpartum infection and Strep B infections in neonates Reported as a randomized controlled trial in the published article Reported as a randomized controlled trial in the published article Insert published article

43 RCT in Norway Midwives knew on which shelf the Chlorhexidine and placebo were kept in the cabinet and just proceeded to administer whatever they deemed appropriate Midwives knew on which shelf the Chlorhexidine and placebo were kept in the cabinet and just proceeded to administer whatever they deemed appropriate Even if they forgot the shelf, they could smell the difference Even if they forgot the shelf, they could smell the difference No monitoring of the allocation process No monitoring of the allocation process

44 Letter from a Physical Therapist A friend of mine conducted an RCT comparing 2 training regimens following surgical repair of the knee for anterior cruciate ligament damage A friend of mine conducted an RCT comparing 2 training regimens following surgical repair of the knee for anterior cruciate ligament damage randomization method used was alternating assignment randomization method used was alternating assignment Referred to one patient as difficult and asked the friend to which group she was assigned Referred to one patient as difficult and asked the friend to which group she was assigned Friend responded that she was in the group that required slightly less contact and that her inclusion was by his choice Friend responded that she was in the group that required slightly less contact and that her inclusion was by his choice

45 Not so blind, after all Sometimes subverting can work against a treatments apparent effectiveness [Wallich P. Not so blind after all. Scientific American; May 1996.] Medical staff … may try to help their sickest patients by sneaking them into the treatment group instead of the control group. The drug would then have to be significantly better than conventional treatment just to appear equal in efficacy

46 RCTs: Anathema to the Human Spirit Scientific need to obtain unbiased data from an inherently biased source – human beings Scientific need to obtain unbiased data from an inherently biased source – human beings Many trialists are unaware of the rationale behind RCTs Many trialists are unaware of the rationale behind RCTs And the need for trial rigor And the need for trial rigor Some may intellectually grasp the need, but once immersed in a trial, have many contradictory interests (e.g. episiotomy trial in Canada) Some may intellectually grasp the need, but once immersed in a trial, have many contradictory interests (e.g. episiotomy trial in Canada)

47 Anathema: A Psychologists View Fundamental conflict between scientific and human imperatives … Many probably think they already know what treatment is best. Fundamental conflict between scientific and human imperatives … Many probably think they already know what treatment is best. Geoffrey Scobie, psychologist at the University of Glasgow; Doctors cheat because they know whats best Geoffrey Scobie, psychologist at the University of Glasgow; Doctors cheat because they know whats best New Scientist, 16 December 95, p. 10 Of course, what we think we know is frequently wrong Of course, what we think we know is frequently wrong

48 Reporting of Exclusions, Double-blinding, and Schedule Generation Related to the Level of Allocation Concealment for 250 Trials AuthorsReported Adequately Concealed (n=79) Unclearly Concealed (n=150) (n=150) Inadequately Concealed (n=21) (n=21) No Exclusions (Apparent)53%67%76% Double-blinding73%39%14% Adequate generation of schedule 29%15%0%


Download ppt "Randomized Controlled Trials: Allocation Concealment Kenneth F. Schulz, PhD."

Similar presentations


Ads by Google