Presentation on theme: "Evaluating Evidence in Medicine: What Can Go Wrong? Skeptic’s Toolbox 2012 Harriet Hall, MD The SkepDoc."— Presentation transcript:
Evaluating Evidence in Medicine: What Can Go Wrong? Skeptic’s Toolbox 2012 Harriet Hall, MD The SkepDoc
Overview What constitutes evidence in medicine? What can go wrong in clinical studies? Why even “evidence-based medicine” is flawed.
Is This Evidence?
Is This Evidence? MRI Study of Salmon A salmon was shown photographs of humans in social situations. It was asked to think about what emotion the individual in the photo must have been experiencing. The salmon couldn’t talk, but: On the fMRI scan, areas in the salmon’s brain lit up, indicating increased blood flow, indicating that the salmon was thinking.
Is This Evidence That: Salmon can see pictures? Salmon know what human emotions are? Salmon can identify emotions from pictures? Salmon can respond to requests of what to think about?
What’s Wrong With This Picture? The Salmon Was Dead and Gutted!
Statistical Artifact Each fMRI scan measures 50,000 voxels (3-D pixels) and each study involves thousands of scans. If you mine the data, you can find practically anything you want. Brain scans are the new phrenology – A blunt instrument – Scans are pooled to establish normal average – Often don’t mean what people think they mean
Would You Accept This Evidence? I tried it. I got better. It worked for me. Lots of people tried it and got better. We gave it to a lot of people in a study and they improved. We compared it to a no-treatment group or a usual-treatment group and it worked better. We compared it to a placebo and it worked better. The weight of evidence from a large body of studies shows that it works better than placebo
Is This Evidence? I tried it. I got better. It worked for me. – Anecdote. Plural of anecdote is not data. – Post hoc ergo propter hoc fallacy – Does Echinacea prevent colds? – Removing glucosamine didn’t remove effects We gave it to a lot of people in a study and they improved. – Uncontrolled study. Maybe they would have improved without treatment. – Cold got better in a week with treatment, lasted 7 days without treatment.
Is This Evidence? Our study compared it to a no-treatment group or a usual-treatment group and it worked better. – Hawthorne effect: Doing something is better than doing nothing. Our study compared it to a placebo and it worked better. – Was the study blinded? – Double blind, placebo-controlled randomized study is the Gold Standard. BUT: What if we do a Gold Standard study on something totally implausible and it works better than a placebo?
There’s A Lot of Evidence: A Fire Hose of Information 21 million papers are listed in PubMed: – 700,000 more each year – One a minute PubMed lists 23,000 journals, and there are many more not listed You can find a study to support any belief.
Never Believe One Study Early positive studies often superseded by better, negative studies (HRT). Ioannidis: Most published research findings are wrong.
Ioannidis The smaller the study, the less likely the research findings are to be true The smaller the effect, the less likely the research findings are to be true. The greater the financial and other interests, the less likely the research findings are to be true The hotter a scientific field, (with more research teams involved), the less likely the research findings are to be true.
Evaluating a Study Ask a lot of questions I’ll cover some
Skeptics Question Everything
What Kind of Study? Case report Case series Case-control Cohort Epidemiologic RCT Placebo-controlled Blinded (single or double)
Who’s Paying? Studies sponsored by pharmaceutical companies more likely to be positive – Subtle bias – Unpublished negative information Studies by researchers with financial conflicts of interest (consulting fees, honoraria from pharmaceutical company) more likely to be positive 91% vs. 67%
Big Pharma Distortion Turner looked at all antidepressant studies registered with FDA – Published studies: 94% positive – Unpublished studies: 51% positive Evidence that antidepressants don’t work? No.
Turner vs. Kirsch Kirsch said <.5 means ineffective – Effect size from journals:.41 – True effect size:.31 – Therefore antidepressants are not effective Turner said glass not empty, 1/3 full Patients’ responses not all-or-none; partial responses can be meaningful Antidepressants DO work, just not as well as originally thought. Kirsch supports psychotherapy, but its effect size is much less than.5.
Scam Product Testing In-house: by non-academics on company’s payroll – Worthless. Tweaked to get desired results Independent testing companies: guns for hire Minuscule effects touted as significant Effects found, but not specific to product – Amino acids may improve muscle strength Effects may not apply to average people (i.e. taping injuries)
Are the Researchers Biased? Homeopathy studies done by homeopaths Chiropractic studies done by chiropractors Surgical studies done by surgeons Studies published in specialty journals for a biased audience
Who Are the Subjects? Self selection bias: who volunteers? – Believers? – Professional subjects? Select group not typical of the general population. – Men only? No children? Limited age group? – Subjects with concurrent diseases not accepted – Subjects taking other medications not accepted.
Were Negative Studies Suppressed? File drawer effect – Negative studies not submitted for publication. – What if 4/5 studies were negative but only the positive one published? Publication bias – Journals don’t like to publish negative studies. – Journals don’t like to publish replications that debunk original results. (Bem, Wiseman)
Did Workers Mislead Author? Technicians and subordinates know what the researcher hopes to find. – May try to please the boss, consciously or unconsciously – May circumvent blinding procedures – Can record 4.5 as 4 or 5. – Faking to make job easier (homeopathy prep)
Did Workers Mislead Author? Benveniste homeopathy study Counting basophil degranulation under the microscope is somewhat subjective Only one technician got positive results
What Are the Odds? 9 out of ten drugs in Phase I clinical trials fail. 50% of drugs that reach Phase III trials fail. A far higher percentage of promising drugs never make it to clinical trials; they fail in animal and in vitro trials.
Do the Data Justify the Conclusion? Teaching exercise: 1.Read the data section first 2.Draw your own conclusions 3.Read the paper’s conclusions 4.Scratch your head
Do the Data Justify the Conclusion? Conclusion: low cholesterol kills children. The higher the cholesterol, the better for health.
Do the Data Justify the Conclusion? Sample of opportunity: data not collected systematically Too few points to show correlation Correlation doesn’t prove causation Other explanations: – Hygiene – Poverty – Disease – Starvation – Genetic factors – Less access to medical care Better explanation: undernourished children have abnormally low cholesterol levels
Do the Data Justify the Conclusion? Conclusion: by the year % of children will be autistic
What Aren’t They Telling Us? Selection methods Randomization methods Identity of placebo Whether people were fooled by placebo Proper blinding procedures? Other factors – Glassware not thoroughly washed? – Contaminants in lab? – Mouse XMRV virus contaminated cell cultures in CFS study – Did they really do what they said they did?
How Many Dropouts? - 10 total patients: 7 neg. 3 pos. = 30% pos. - 6 drop out because it’s not working - 30% success rate now looks like 75%
Where Was the Study Done? Percent of Acupuncture Trials with Positive Results Canada, Australia, New Zealand 30% US 53% Scandinavia 55% UK 60% Rest of Europe 78% Asia 98% Brazil, Israel, Nigeria 100%
What was the sample size? 1/3 of the chickens got better 1/3 of the chickens stayed the same What about the other third?
Were There Errors in Statistics? Wrong statistical test used Errors in calculation
What About Noncompliance? Did all subjects take their pills? Did they take them on time?
Noncompliance HIV Prophylaxis study in Africa – 95% said they usually or always took meds on time – Pill count data: 88% – Tests showed adequate plasma levels of drug: %
Tooth Fairy Science Are they trying to study something that doesn’t exist?
Emily Rosa and the Emperor's New Clothes
Inaccurate Measuring Methods? Questionnaires rely on unreliable memories and patient honesty. – “30% less pain” – “I eat like a bird” – “Only one drink”
Using a Bogus Test? Measuring the Components of ASEA A mixture of 16 chemically recombined products of salt and water with completely new chemical properties. They used a fluorescent indicator as a probe for unspecified “highly reactive oxygen species”
How Many Endpoints Were There? Multiple endpoints: some will show false correlations just by chance Statistical corrections applied? Inappropriate data mining? The heart prayer study – 6 positive out of 26 factors studied – Inconsistent pattern
Were Goalposts Moved? AIDS prayer study: endpoint death Not enough subjects died: AIDS drugs kept them alive They went back and looked at a lot of other factors and found some apparent successes (i.e., fewer doctor visits) but no change in objective tests like CD4 count. Only 40 patients. Study wasn’t designed to test non-death outcomes.
Statistical Significance ≠ Clinical Significance Did the drug lower the BP by 1% or 30%? Was the endpoint a lab value or a clinical benefit? B vitamin supplements lower homocysteine but don’t lower risk of heart disease PSA screening finds cancers; doesn’t improve survival Are the results POEMS – Patient Oriented Evidence that Matters?
Was There Fraud? Dipak Das, resveratrol researcher – Review board found him guilty of 145 counts of fabrication or falsification of data – 12 of his papers retracted so far
“I was blinded by work and my drive for achievement” Hwang Woo-suk, stem cell researcher in South Korea, claimed to have cloned human embryonic stem cells – Fabricated crucial data – Embezzlement and bioethics law violations – Prison sentence (suspended) – 2 papers in Science retracted. – Fired from his job
Columbia Prayer Study Prayer doubled success of in vitro fertilization – Seriously flawed study – Convoluted design with 3 levels of overlapping prayer groups – No controls for prayers outside study – Investigated for lack of informed consent Authors – Lobo, lead author, only learned of study 6-12 months after it was completed. Denied any involvement other than editorial help. – Cha severed his relationship with Columbia, refused to comment – Wirth: Paranormal researcher with no medical degree Con man who went to federal prison for fraud and conspiracy Bruce Flamm debunked it in Skeptical Inquirer Retracted by journal, but only years later Still being cited as a valid study
How Were the Data Reported? NNT and NNH – Lipitor for primary prevention of heart attacks: 19% Reduction NNT , NNH 200. Absolute risk vs. relative risk – Cellphones increase the risk of acoustic neuroma. Relative risk 200%. – Baseline risk is 1:100,000 – 200% of 1 is 2 – Absolute risk 1 more in 100,000, or %
What Are the Confidence Intervals? Confidence interval of 95%:
Where Was the Study Published? Acupuncture studies in acupuncture journals? Homeopathy studies in homeopathy journals? Acupuncture or homeopathy study published in a major medical journal?
Were the Results Misinterpreted? True acupuncture = sham acupuncture Both better than placebo pill Acupuncturist’s interpretation: – Sham acupuncture must work too Better interpretation: both are impressive placebos.
Were Recommendations Justified? X didn’t work, but it didn’t cause harm, and since we have no other effective treatment for Y, we should continue to use X. New drug X didn’t work better than the placebo, but we didn’t see any side effects, and since we have no other effective treatment for Y, X should be approved for marketing.
Do we really know what the study showed? Peer critiques, letters to editor Media distortions – Presenting preliminary evidence as definitive – Misinterpreting results of study
Glucosamine/Chondroitin Study Overall: not effective Subgroup analysis (10 subgroups) – Pos. in patients with moderate to severe arthritis – Neg. in patients with mild to moderate arthritis Reported in the media as both + and – Authors said study not powered to show effectiveness in subgroup
What does p value mean? It’s significant at the p=0.05 level, so it must be true. p=0.05 – Means 5 in 100 chance that a positive result is false – Doesn’t mean a 95% chance that a positive result is true. Says nothing about the meaning of a positive result
4 possible outcomes
P Value = Specificity: Probability that an ineffective treatment will give a false positive result
Sensitivity: The probability that an effective treatment will show a positive result
Positive Predictive Value: if the study is positive, how likely is it to be true? PPV = TP / TP + FP TPTP FP 0.05
Prior Probability 50% If researchers don’t consider prior probability, they are automatically assigning a PP of 50%. PPV = 80/80+5 = 94%. 6% chance that positive results are wrong. 0.05
PPV with Prior Probability 5% TP: 80% of 5% = 4% FP: 5% of 95% = 4.75% PPV = TP/ TP + FP = 4%/8.75% Chance that positive Result is true: 46% Slightly less than coin toss TPTP FP
PPV with Prior Probability 5% and 1% PP 5% TP: 80% of 5% = 4% FP: 5% of 95% = 4.75% PPV = TP/ TP + FP = 4%/8.75% Chance that positive result is true: 46% Slightly less than coin toss PP 1% TP: 80% of 1% = 0.8% FP: 5% of 99% = 4.95% PPV= 0.8%/ 0.8% % = 0.8/5.75 = 0.14% Chance that positive result is true: 14% Chance that positive result Is false: 86% The Plausibility Problem, by David Weinberg, on SBM FP
Statistical Significance The p=0.05 cutoff is arbitrary Statistical significance doesn’t mean clinical significance Be wary of studies that say outcome was positive but admit it was not statistically significant. – “X worked better than Y, but the results didn’t reach significance.”
Power of a study A power of.8 in medical research is considered a very powerful study and would require a large number of patients. If the power is less than that, the Positive Predictive Value drops even more.
When CAN You Believe Research?Bausell ’ s Quick Checklist Randomized with a credible control group At least 50 subjects per group Dropout rate 25% or less Published in a high-quality, prestigious, peer- reviewed journal
When CAN You Believe Research? Confirmed by other studies Consistent with other knowledge Prior probability
Systematic Reviews and Meta-analyses A flawed method to sort out conflicting research results. A few high quality studies trump the conclusions of a meta-analysis. The results of a meta analysis usually fail to predict the results of future good clinical trials.
Does It Make Sense? Energy medicine proponents claim to have measured a 2 milligauss magnetic field emanating from practitioners’ hands Reproducible measurements by other scientists fall in the range of milligauss. The magnetic field of the earth is 500 milligauss. Refrigerator magnet: 50,000 milligauss. Even if the 2 milligauss measurement were accurate, it would be many orders of magnitude below the cell’s noise level and billions of times less than the energy received by your eye when viewing the brightest star.
Evidence-Based Medicine Isn’t Enough EBM is working hard, but it got something wrong.
What’s missing from the EBM pyramid? Carl Sagan
Don’t forget prior plausibility “Extraordinary claims require extraordinary evidence.” If basic science says a treatment is implausible, we must set the bar for clinical evidence higher.
EBM Founders’ Assumptions The rigorous clinical trial is the final arbiter of any claim that has already demonstrated promise by all other criteria—basic science, animal studies, legitimate case series, small controlled trials, “expert opinion,” etc. Claims lacking in promise were not even part of the discussion.
Plausibility Spectrum for CAM Homeopathy: close to zero. Acupuncture: intermediate – Underlying Oriental concepts have low plausibility – But it’s plausible that inserting needles in the skin could cause physiological effects Herbal medicine: high plausibility because plants produce drugs.
EBM Accepts Tooth Fairy Science Typical fairy tales: – Reiki (faith healing that substitutes Eastern mysticism for Christian beliefs) – Homeopathy (essentially a form of sympathetic magic) – Therapeutic touch (A misnomer for smoothing out wrinkles in a mythical human energy field without actually touching the patient) Trying to apply the tools of science to these therapeutic modalities based on fantasy just produces a lot of confusing noise
Tooth Fairy Science: Studying Therapeutic Touch Therapeutic Touch is said to manipulate the alleged “human energy field.” Controlled TT studies have been done for: –Pain –Bone marrow transplant –Recovery from cardiac surgery Positive results due to the effects of suggestion and attention, not to a mythical energy field.
Tooth Fairy Science: Therapeutic Touch 2008 study: Randomized trial of healing touch to speed recovery from coronary artery bypass surgery: – Decrease in anxiety and length of stay – No significant differences for other endpoints Lab study: therapeutic touch affects DNA synthesis and mineralization of human osteoblasts in culture. Cochrane review: positive
Another EBM Pitfall: Pragmatic Studies Clinical trials select patients to minimize possible confounders: Subjects tend to be healthier and on fewer medications than the average patient. Pragmatic studies look at the outcome of treatments in real-world settings. Clotbuster drugs worked well for strokes in clinical trials, but with more extensive use in ERs they caused more strokes from bleeding complications.
Pragmatic Trials May Not Be Appropriate for CAM Treatments Intended to evaluate practical real-world use of treatments that have already been proven to work in clinical trials. Pragmatic trials can’t provide objective evidence that a treatment has effects beyond placebo. CAM proponents favor pragmatic studies because – They don’t control for placebo effects. – They can bypass good science. – They can make CAM look better than it really is.
Pragmatic Trial Acupuncture vs. usual care for low back pain. Acupuncture wins. Have you proved acupuncture is really more effective? No, this is Cinderella Science
Cinderella Revised Cinderella in rags and ashes Ugly Stepsister who has had a complete makeover Before After
Pragmatic Trial of Acupuncture for Low Back Pain Usual Care Acupuncture Needles alone Acupuncture
Evaluating Evidence in Medicine: What Can Go Wrong? Placebo effect Therapeutic effect of consultation (suggestion, expectation) Unassisted natural healing (natural course of disease) Unrecognized treatments (the spaghetti sauce factor) Regression toward the mean Other concurrent conventional treatment Cessation of unpleasant treatment Lifestyle changes Publishing standards (p=0.05) Publication bias
What Can Go Right? Large well designed studies Prior plausibility Strongly positive results Consensus of experts Coherent body of evidence
Evaluating Evidence in Medicine: What Can Go Wrong?
Dr. Jay Gordon Pediatrician to the Stars “My very strong impression is that children with the fewest vaccines, or no vaccines at all, get sick less frequently and are healthier in general. I truly believe they also develop less autism.”
Take-Home Points There are various kinds of evidence, some more credible than others. Most published clinical studies are wrong. Even “evidence-based medicine” can get it wrong. Statistical significance ≠ clinical significance. Prior probability is important. Clinical studies don’t “prove” – they only change the probabilities.