1. How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data Daniele Fanelli

2. “The million dollar question”…. Data sourceFrequency (%) of misconduct Government-confirmed cases in the US 0.001 – 0.01(Marshall 2000, Steneck 2006) Retractions due to misconduct, in PubMed 0.02 - 0.2(Claxton 2005) Manipulated images submitted to The Journal of Cell Biology 1(Steneck 2006) Investigators disqualified by FDA data audits (1977-1990) 2(Glick 1992) -Only misconduct that has been discovered, and (presumably) proven to be intentional Ultimately, only scientists know about their own intentions!

3. Never Sometimes Frequently Over the years, many surveys have asked scientists directly… “Since entering medical school have you…?” 0 1-5 >5 “Have you participated in research involving […] during the last 10 years?” “Fabricated data” “Modified research or experimental results to improve the outcome” “Failing to present data that contradict one's own previous research” “Indicate the number of […] members you have observed/experienced exhibiting […] within the last 5 years” “Seriously misleading interpretation of results” Yes No Question Form of misconduct Outcome Results appeared inconclusive and dificult to compare …different things, in different ways…

4. “Tricks” in the analysis How many committed or observed X at least once Only questions on fabrication, falsification, alteration and QRP that distort scientific knowledge. No plagiarism, professional misconduct etc… Mixed questions were excluded No other measure of study quality (it’s controversial) –Included all eligible studies that specified their methods –Entered methodological factors in inverse variance weighted regression ES=Log      ) 1 ( p p W= 2 1 SE = )1(pn p  Question by question: effect size and weight

5. 42 literature databases, 14 journals, 8 grey literature db, 2 internet scientific search engines, and references lists The search… "research misconduct" OR "research integrity" OR "research malpractice" OR "scientific fraud" OR "fabrication, falsification" OR "falsification, fabrication" Potentially relevant studies obtained from electronic search (n=3276) Studies retrieved for examination of full text (n=69) Studies included in review (n=21) Studies excluded for one of the following reasons (n=48): -Did not have any relevant or original data -Sample not exclusively composed of researchers -Misconduct not related to research (e.g. cheating on school projects) -Does not distinguish fabrication and falsification from other forms of misconduct not relevant to this review -Presents data only in format not usable in this review Studies excluded because were not surveys on research misconduct (n=3207) Studies included in meta- analysis (n=18) Studies excluded from meta-analysis because did not meet quality criteria (n=3)

6. Characteristics of studies Conducted between 1986-2005 USA (15), UK (3), multinational (2), and Australia (1) Medical/clinical (8), biomedical (6), multidisciplinary (6), economists (1) In total 85 questions: –about fabrication, falsification, alteration, modification (meta-analysis) –Questionable research practices (systematic review only) (Full data set available soon in PLoS ONE)

7. Scientists who admit fabrication, falsification, or alteration of results b= -0.14±0.05 P=0.006 1.97% (N=7, 95%CI: 0.86-4.45) If only asked “fabrication, falsification” 1.06% (N=4, 95%CI: 0.31-3.51) Scientists who know a colleague who fabricated, falsified, or altered results 14.12% (N=12, 95% CI: 9.91-19.72) If only asked “fabrication, falsification” 12.34% (N=11, 95%CI: 8.43-17.71)

8. Questionable Research Practices (e.g. “failing to publish data that contradicts one’s previous research” “dropping data points based on a gut feeling”)

9. Inverse variance-weighted regression What influences admission rates? Asking about self vs colleagues: b±SEP -4.53±0.81 <0.0001 Handed-out surveys vs mailed: 1.17±0.40.0032 Using “fabrication” or “falsification” vs “alteration” or “modification” - + - -1.02±0.390.0086 82% of variance explained (N=15) USA / other Researcher / other Biomedical / other Social Sc. / other Medical / other Controlling for these factors, tested for differences between: b=0.85±0.28 P= 0.0022 n.s. Year

10. Martinson 2005 is outstanding, as conservative! Leave-one-out sensitivity analysis Scientists who admit fabrication, falsification, or alteration of results Scientists who know a colleague who fabricated, falsified, or altered results

11. “Repairing misconduct” ID N cases Action taken % Tangney, 1987 78Took some action to verify their suspicions of fraud or to remedy the situation 46 Rankin, 199731 (incl. Plag.) In alleged cases of scientific misconduct a disciplinary action was taken by the dean 32.4 Some authority was involved in a disciplinary action20.5 Ranstam, 2000 49I interfered to prevent it from happening28.6 I reported it to a relevant person or organization22.4 Kattenbraker, 2007 33Confronted individual55.5 Reported to supervisor36.4 Reported to Institutional Review Board12.1 Discussed with colleagues36.4 Titus, 2008115 (incl. Plag.) The suspected misconduct was reported by the survey respondent 24.4 The suspected misconduct was reported by someone else33.3 Around half of recalled cases had no action whatsoever taken against them

12. Summary of key findings Data fabrication, falsification and alteration was –admitted on average by around 2% (1% - 4%) –directly observed by 14% (10% - 20%) Questionable Research Practices were –admitted on average by up to 34% –directly observed by up to 72% Overall admission rates (self-/non-self) were higher in –Non-self reports, questions not using “fabrication” or “falsification”, handed out questionnaires –Medical/clinical and related research

13. Unlike surveys for other criminal behaviour, Scientists always lose by admitting misconduct Unclear -Risk of multiple reporting -“Muhammad Ali” effect -Unaware of all cases -Unwilling to damage their field Self-reports Non-self-reports Conservative Regression analysis: -Medical research not robust to all sensitivity -Differences in methodology masked most effects -Non-significant effects not necessarily non-significant How Reliable Are These Numbers?

14. Conclusions On average, 2% of scientists admitted misconduct, and 34% QRP –Actual frequencies probably higher –Probably vary depending on field and many other factors, which meta-analysis currently cannot detect Future surveys might benefit by –Focusing on correlates of misconduct –Common methodologies (tip of the iceberg) (this too)