Presentation on theme: "…when the conditions of an experiment are once established and worked up according to the mind's preconceived idea, an induced or premeditated observation."— Presentation transcript:
…when the conditions of an experiment are once established and worked up according to the mind's preconceived idea, an induced or premeditated observation will, as we said, result.
If men discuss and experiment … to prove a preconceived idea …, they no longer have freedom of mind, and they no longer search for truth.
An experiment done to prove something is different from an experiment done to test something.
A Favorite Example (10/2003): Patients with uncomplicated DVT/PE Randomize between Unfractionated heparin (standard) New drug (easier to use) Compare rates of DVT, PE, sudden death Open label, >2,000 patients/subjects Non-inferiority design
The results: Similar outcomes in DVT/PE (slight NS advantage to new drug) Similar outcomes re: mortality Patients getting new drug tended to leave the hospital earlier (sub-q rather than i.v.) Conclusion: Non-inferior in efficacy; dosing convenience make it an attractive drug
Whats Wrong? New drug was compared against old approach of known efficacy Everybody got treated (no placebo gremlins) Trial design was standard; sample size was appropriate; few drop-outs Monitoring was appropriate (Incl. DSMB) Reputable researchers, sites
The trial was ethical and well- done... but … The standard therapy was no longer usual practice … drugs similar to the test drug had already largely replaced unfractionated heparin, at least in academic centers The trial outcome can be restated as This drug is probably no worse than what we used to do ten years ago A strong finding in favor of licensing the drug, but very little help in deciding how to use it wisely
A Non-so-benign Example: Dan Markingson and his mother, Mary Weiss Dans death in an industry-sponsored clinical trial raises questions about how we do them and monitor them,
Dans medical history: Bright kid; got a B.A. in English from the University of Michigan … sought employment in California. Over the following three years, developed paranoia, changed his name, adopted cult beliefs. Indicated to his mother that there was an apocalyptic event approaching, that he might be instructed to kill and would do so.
The story gets complex: Several maternal interventions fell short Hospitalized on a hold, given antipsychotic meds; commitment proceedings begun All examiners agreed that he lacked full decisional capacity and was a danger to himself and others Result: Stay of commitment
Stay of commitment: The court finds in favor of commitment, but stays the order on conditions Typically: As long as you comply with treatment and do OK with it, you wont actually be committed and will still have control over your own life Conceptually like a revokable parole No guardian is named
The treatment chosen for Dan: CAFÉ: Clinical Trial Comparing Atypicals in First Episode Randomly assign among the three most common atypical antipsychotic agents for patients having their first psychotic episode Seroquel (quetiapine), Zyprexa (olanzapine), and Risperdal (risperidone.) Study designed, funded by AstraZeneca
Digression: Atypical antipsychotics All three were licensed and approved; Very successful drugs commercially, having largely supplanted earlier antipsychotic drugs on the strength of claims that they had fewer autonomic side effects and a lower incidence of tardive dyskinesia; One of these would very probably have been chosen if not for study entry
How would the study alter Dans Rx? Double-blind; Limits on other agents; discontinuing or crossing over would require study withdrawal; Treatment in other respects would be pretty much the same as getting one of the drugs non-randomly out of study.
Concerns/Oddnesses: Should he be his own decisional authority on study matters? The trial forbade entry of patients who were suicide risks, but not of patients who were homicide risks(!) Dan had threatened to kill others, but not himself 45 CFR§ (b); 21 CFR§ (b)
The story continues: Dans mother objected to his study enrollment She was deemed not to have status in the matter (because Dans commitment had been stayed, she was not named guardian) Dan was randomized to receive Seroquel, and was discharged to a halfway house
The story continues: Dan seemed improved at his psychiatrist visits, but they were infrequent Social workers and Dans mother recorded concerns that he was still delusional and still at risk to harm himself or others Dans mother asked that he be removed from the study and tried on a different drug - again deemed to have no status
The story continues: At the end of the six-month stay, his psychiatrist asked that the stay be extended (rather than effecting or vacating the commitment) -- might be at risk if he stops treatment Mother warned in writing that she feared he would kill himself or someone else; study participation continued Dan killed himself six months into study participation
Postscript: (Civil legal ugliness I wont get into…) FDA review: Not clear that Dan died because of study participation; no action taken Prosecutorial review: Not clear that Dan died because of study participation; no criminal liability found; no action taken
Questions: Would his drug regimen have been changed if he were not in a study? Would his protections have been better if he had in fact been committed and had a guardian? Was study participation in fact a contributor to his suicide (or was his schizophrenia beyond help)?
Postscript ii: Several faculty members asked for an outside review, arguing that nothing worth prosecuting isnt a high enough standard…
Third example: Popular anti-inflammatory drug about to have its patent expire It has an aspirin-like effect on platelets, so might prevent certain types of clots If so, it may make an intravenous form of the drug useful New formulation new patent $$$$ Lets try it!
The experiment worked: Intravenous drug prevented experimental clots in veins that had been chemically, thermally or abrasionally damaged Worked better than having the animals on warfarin for a week in advance of the injury Worked almost as well as heparin (the standard i.v. anticoagulant) The sponsor decided not to try the study in people
The experiment worked TOO WELL: The drug levels that prevented clots could be achieved with the oral form already on the market No new form no new patent no big bucks (Schillinge, actually – this was in Vienna before the Euro came in) When we now add this drug to a failing anticoagulant regimen, were doing so on the strength of unpublished bunny data
The three examples show several of the problems one might encounter in industry-sponsored research. As Claude Bernard warned 150 years ago, going into research with a desired outcome in mind distorts the entire scientific process: What questions you ask How you ask them How you select subjects How you analyze results How (even whether) you report results
The three examples also suggest that the incentive structure might be important: If your desired outcome is support of product development or marketing, it further influences: What questions you ask How you ask them How you select subjects How you analyze results How (even whether) you report results
How big a deal is this? Marketing has become a much larger fraction of drug-company budgets; Firewalls between marketing and research components of drug companies have largely been taken down; This may now be a bigger concern than it was 25 years ago Carl Elliott, M.D.
The questions you ask: Questions that will likely gain evidence in support of your desired outcome (counterproof) Questions that will support the licensure and marketing of your product Questions that will demonstrate a niche in which your product might outperform others
The questions you might not ask: Questions that will critically challenge the hypotheses that underpin your desired outcome (counterproof) Questions that will sensitively detect the limitations of your product (toxicities; drug interactions) Questions that will demonstrate a niche in which your product might be outperformed by others Questions supporting strategies that do not lead to profit
How you ask the question: Choose subjects who are most likely to benefit; least likely to have problems Structure the competitor arms so that they are unlikely to be as good as they could be Choose a sample size and study interval that make it unlikely that you will detect/document any uncommon problems Make the primary endpoint something that advantages your product
Reality check: Its not automatically evil to do a study that evaluates a product under the most favorable circumstances … you have to find out if it CAN work It becomes a problem when those studies are presented as though they are more representative and more definitive It becomes a problem when the more definitive and more generalizable studies never happen, or never get published
Reality check ii: Its not just industry-sponsored research The same sort of intellectual traps and adverse incentive structures are encountered when someone is pursuing an idea on which hes built his career, or when hes pursuing his next major grant
How (or whether) you report results: Shout from the rooftops when you hypothesis (or product) is supported Publish in a high-profile journal Distribute zillions of reprints Put the caveats in a separate publication with a lower profile Publish unfavorable studies obscurely or not at all
Is this a theoretical concern or real? Lets look at the CAFÉ study, the one that Dan was enrolled in: Authors included 3 AstraZenica employees and seven extramural psychiatrists 400 subjects Primary endpoint: Still taking drug at one year? (A combined endpoint for efficacy and tolerance) Non-inferiority design
CAFÉ study Only 119 of the 400 subjects were still taking the drug at one year Publication doesnt tell why those who stopped the drug did so (side effects vs. ineffectiveness) Conclusion: Non-inferior
CAFÉ study With three groups and only 119 outcomes, a huge difference could fail to be detected statistically. For example, if the three groups had 25, 47 and 47 (rather than equal numbers of favorable outcomes), that would not have reached the customary statistical significance threshold of p < 0.05.
CAFÉ study The study was underpowered to detect a difference, but its primary outcome wasno difference. That no difference is then described with phrases like comparable efficacy, obscuring the fact that the study was pretty insensitive. This turned out to be really important, because an unpublished study suggested that the sponsors drug might actually be inferior!
CAFÉ study No comparisons were made to older antipsychotic agents, such as haloperidol With the newer agents superiority being challenged, even a good study comparing the three newcomers might have been asking the wrong basic question. Carl Elliott (and others) asks: Was this study a marketing exercise or a scientific one?
Carl Elliott (reviewing several critiques of the CAFÉ study): If these experts are right, then the study in which Dan Markingson committed suicide was not simply a matter of inadequate informed consent, or financial conflicts of interest, or even failure to monitor a subject's care. The ethical breach was built into the study from the start. It is one thing to ask people to take risks for science, or the common good, or to help other people. It is another thing entirely to ask them to risk their lives for the marketing goals of AstraZeneca.
One Study, or a Pattern? Meta-analyses have been done both for newer antidepressants and for newer antipsychotic medications Invariably, more than 3/4 of the published studies have reported results favoring the product of the sponsoring pharmaceutical company This appears NOT just to be publication bias; the meta-analysts in some studies have looked for – and found – study design bias in a substantial proportion of reports
So Where Are WE?
The balance is tricky: Pharmaceutical and device manufacturers have a need to test their products Proof-of-concept studies (CAN it work?) have a legitimate place in product development – but often are limited in their generalizability Real-world studies (will it work in practice?) may be more practical, but may miss niche utility Studies comparing products may be very useful, but may also be misleading if designed (or promulgated) from a marketing perspective.
The balance is tricky: Concealing study results that do not favor your product is wrong if it puts anybody at risk Potential subjects have a right to know if the risk they are being asked to bear is to answer a marketing question, a curiosity question, or a question with immediate implications for best clinical practice An IRB is charged to make the call whether the benefit of a study is enough to warrant the risks subjects are asked to bear -- whos to help them decide the tough cases?
And just to make sure youre challenged: The Markingson case led to some lawsuits. One finding might surprise you: The drug manufacturers fiduciary responsibility is to its shareholders No legal (as opposed to ethical) responsibility to put the welfare of the subjects ahead of that of the shareholders Thus, no grounds for tort action if regulations complied with
If we could turn back the clock: Direct marketing of drugs and devices to consumers has led to huge marketing juggernauts in many companies; the old firewalls between marketing and science were probably a good thing. We cant turn back the clock, but perhaps a regulatory structure could be devised to make it harder for marketing considerations to distort the science.
What we can do now: We can be more diligent in trying to identify studies that are likely to be market driven, and make certain that their real nature is frankly disclosed to subjects. We can support proposals that would require product-comparison studies to be independently designed and conducted, rather than being the property of someone with a stake in the outcome.
Its not just the sponsors: Comprehensive Cancer Centers are scored on the proportion of patients entered on study protocols – that creates a pressure to have a study (ANY study) for every patient My own division has come under pressure to implement more Phase-I studies, because of their favorable impact on hospital and clinic revenue streams.
Claude Bernard saw it coming But I ll bet he didn t foresee the enormity of the financial traps that are there to augment the intellectual ones. His warnings are just as timely now as in 1865 … but their challenge has grown.