Presentation is loading. Please wait.

Presentation is loading. Please wait.

Research Integrity, The Importance of Data Acquisition and Management Ralph H. Hruban, M.D. Monday, February 13, 2011.

Similar presentations


Presentation on theme: "Research Integrity, The Importance of Data Acquisition and Management Ralph H. Hruban, M.D. Monday, February 13, 2011."— Presentation transcript:

1 Research Integrity, The Importance of Data Acquisition and Management Ralph H. Hruban, M.D. Monday, February 13, 2011

2 Conflict of Interest I receive royalty payments from Myriad Genetics for the PalB2 invention

3 I think what happened is that you are betting on football, and what’s after football is basketball, and then the NCAA tournament. The next thing that follows is betting on baseball… I wish I could take it all back. Pete Rose

4 Aristotle “We become just by performing just actions, temperate by performing temperate actions, brave by performing brave actions” Nicomachean Ethics

5 Henry L. Mencken “Science, at bottom, is really anti-intellectual. It always distrusts pure reason, and demands the production of objective fact.”

6 “The [pirate] code is more what you’d call ‘guidelines’ than actual rules” Screenrant.com Barbossa, Pirates of the Caribbean

7 The PHS regulation (42 C.F.R. 93) defines research misconduct as fabrication, falsification, or plagiarism in proposing, performing, or reviewing research, or in reporting research results. (a) Fabrication is making up data or results and recording or reporting them. (b) Falsification is manipulating research materials, equipment, or processes, or changing or omitting data or results such that the research is not accurately represented in the research record. (c) Plagiarism is the appropriation of another person's ideas, processes, results, or words without giving appropriate credit. (d) Research misconduct does not include honest error or differences of opinion. PHS 42 C.F.R. 93

8 Data Integrity 1.How common is data fraud? 2.Fraud harms patients, the institution and the investigator 3.Examples of inappropriately published data 4.How can you prevent fraud in your own lab?

9 Fraud is More Common Than You Think

10 ORI is pleased to have high school students, Michael Moorin and Tyler Smith, present at the Quest for Research Excellence 2011 Conference. Moorin and Smith made headlines in the news media, such as The Washington Post, when they found over 60% of high school students reported that they had falsified or fabricated the data in their science fair projects misconduct/ Fraud in High School

11 Web Sites and Yet another young scientist starting postgrad Desires their CV to be better, a tad. Such a wonderful gel! I must publish in Cell! The controls I can fix on my iPad.

12 Retractionwatch.wordpress.com /

13 How Common is Misconduct? Meta-analysis of surveys of scientific misconduct 2% of scientists admitted to have fabricated, falsified or modified data or results at least once Fanelli PLoS One 2009; 4:e5738

14

15 Many More Were Aware of Misconduct by Others!

16 Nature 478, (2011)

17

18 Data Integrity 1.How common is data fraud? 2.Fraud harms patients, the institution and the investigator 3.Examples of inappropriately published data 4.How can you prevent fraud in your own lab?

19 Fraud Harms Patients Analyzed 180 retracted articles that involved human subjects or “freshly derived human material,” along with 851 published studies citing that research The retracted papers were cited over 5,000 times According to Steen, 6,573 patients received treatment in studies eventually retracted because of fraud. One study alone, published in 2001, included 2,161 women being treated for postpartum bleeding The downstream studies included more than 400,000 subjects, with 70,501 receiving treatment R. Grant Steen, Journal of Medical Ethics, 2011

20 Deception at Duke Scott Pelley reports on a Duke University oncologist whose supervisor says he manipulated the data in his study of a breakthrough cancer therapy

21 Fraud Harms the Institution and the Investigator “But the research at Duke turned out to be wrong. Its gene-based tests proved worthless, and the research behind them was discredited. Ms. Jacobs died a few months after treatment, and her husband and other patients’ relatives have retained lawyers.” Gina Kolata, on Anil Potti, New York times, July 7, 2011 Anil Potti, MD

22 Potti Scandal The defendants named in the suits are: Duke University Duke University Health System, Inc. Private Diagnostics Clinic PLLC Joseph Nevins, PhD Anil Potti, MD Michael Cuff, MD Sally Kornbluth, MD John M. Harrelson, MD Cancer Diagnostics, Inc.

23 Research Misconduct Harms Patients, the Investigator and the Institution

24 Data Integrity 1.How common is data fraud? 2.Fraud harms patients, the institution and the investigator 3.Examples of inappropriately published data 4.How can you prevent fraud in your own lab?

25

26 Examples Fabricated data Falsified data Selective reporting of data Image manipulation

27 Example 1: Trial of 3 Drugs- Actual Results

28 Trial of 3 Drugs- Results Reported

29 Data Fabrication Fabrication is making up data or results and recording or reporting them

30 Jon Sudbø- Fabrication Medical researcher at the Radium Hospital, Oslo, Norway 2005 article in the Lancet suggested that Ibuprofen reduces oral cancer in smokers The Lancet, 366 (9494): 1359–1366;

31 Jon Sudbø- Fabrication Suspicion aroused because the data were supposedly from a cancer patient database which had not yet opened Of the 908 subjects in the Lancet study 250 had the same date of birth Sudbø later acknowledged that he used fictional data in at least two more papers, published in the New England Journal of Medicine and Journal of Clinical Oncology

32 Jon Sudbø- Fabrication Independent commission investigated and also criticized the co-authors of Sudbø's papers Dr. Atle Klovning, a leading European authority, said that Sudbø's co-authors had probably not lived up to their responsibilities according to the rules of authorship You think they would have noticed the database wasn’t open yet!

33 International Committee of Medical Journal Editors An “author” is generally considered to be someone who has made substantive intellectual contributions to a published study…. An author must take responsibility for at least one component of the work, should be able to identify who is responsible for each other component, and should ideally be confident in their co-authors’ ability and integrity. When a large, multicenter group has conducted the work, the group should identify the individuals who accept direct responsibility for the manuscript

34 Example 2: Trial of 3 Drugs- Actual Results

35 Trial of 3 Drugs- Results Reported

36 Data Falsification Falsification is manipulating research materials, equipment, or processes, or changing or omitting data or results such that the research is not accurately represented in the research record

37 Suggested that microarray data from cell lines (NCI 60) could be used to define drug response signatures, and these signatures could in turn be used to guide therapy

38 Nature Medicine, 2006 The results of this study had therapeutic implications

39 Forensic bioinformatics Keith Baggerly Kevin Coombes Time.com Woodward and Bernstein of Bioinformatics

40 matics2010_baggerly_irrh/

41 Baggerly and Coombes Investigate PottiBaggerly Reported Genes

42 Potti’s paper suffers from a “frameshift mutation” (Off by one error for all of the genes caused by an extra column) Potti et al submit erratum with updated gene lists NOT the end of the saga……. Potti Baggerly

43 Sensitive and Resistant Switched for some Drugs

44 Letter to the Editor (Nature Medicine) – one page letter, 149 pages of Supplementary Data November 2007 Keith Baggerly and Coombes

45

46 Journal of Clinical Oncology, 2007 For cisplatin, U133A arrays were used for training. ERCC1, ERCC4 and DNA repair genes are identified as “important”

47 The four that couldn’t be matched were the genes that were touted to be functionally important Four Genes Didn’t Match

48 Based directly on the Potti and Nevins publications, despite concerns raised by Baggerly and Combes, Duke Initiates three Clinical trials in 2007 Adjuvant Cisplatin With Either Genomic-Guided Vinorelbine or Pemetrexed for Early Stage Non-Small-Cell Lung Cancer (TOP0703) Study Using a Genomic Predictor of Platinum Resistance to Guide Therapy in Stage IIIB/IV Non-Small Cell Lung Cancer (TOP0602) Phase II Study Evaluating The Safety And Response To Neoadjuvant Dasatinib In Early Stage Non-Small Cell Lung Cancer (TOP0706)

49 O, what a tangled web we weave; When first we practice to deceive! Sir Walter Scott

50 July 16, 2010

51 Retraction watch November 2010

52

53

54 Improving Validation Practices in “Omics” Research Routine replication, public data and protocol availability, funding incentives, reproducibility rewards or penalties, and targeted repeatability checks Ioannidis, et al., Science December : Vol. 334:

55 Example 3: Trial of 3 Drugs-Actual Results Results Statistically Significant

56 Trial of 3 Drugs-Reported Results (Results still Significant)

57 It is Still Data Falsification Falsification is manipulating research materials, equipment, or processes, or changing or omitting data or results such that the research is not accurately represented in the research record (No qualifier here that falsification is ok so long as the results were originally statistically significant)

58 Example 4: Trial of 3 Drugs-Actual Results: Results Statistically Significant

59 The PI Tells the Post-Doc “These two data points seems off. I would expect there to be a greater difference”

60 Trial of 3 Drugs-Reported Results (Results still Significant)

61 It is found out later that the Post-Doc Changed the data in question Does the P.I. have any responsibility for what happened?

62 Dipak K. Das NY Times, January 11, 2012 and retractionwatch.wordpress.com The University of Connecticut report alleges Dr. Das “defunded” the work of a student in his lab because she did not produce results that he wanted The investigation of Dr. Das’s work began in January 2009, two weeks after the university received an anonymous allegation about research irregularities in his laboratory

63 Allegations of misconduct often come from a whistle blower inside the group, such as a postdoc or graduate student who does not agree with the PI's tendencies of glossing over data or blatant misconduct

64 What should we do when we “suspect” another PI is falsifying data? What if the other PI is a competitor? We All Have a Responsibility to Maintain Integrity

65 Example 5: 5-Month Trial of 3 Drugs-Actual Results of a 5-Month Design

66 Trial of 3 Drugs-Results Reported

67 Could be Falsification Falsification is manipulating research materials, equipment, or processes, or changing or omitting data or results such that the research is not accurately represented in the research record

68 OK Only if Clearly Documented in the Paper “…not accurately represented in the research record”

69 Example 6: Results (4 Day Expt., but Technician ran the Experiment too long)

70 Trial of 3 Drugs-Results Reported

71 Probably OK if the study design was shorter, but it ought to get you thinking!

72 The most exciting phrase to hear in science, the one that heralds new discoveries, is not ‘Eureka,’ but ‘That’s funny…’ Isaac Asimov

73 Example 7: Trial of 3 Drugs- Results First Run

74 Results Second Run

75 Results Third Run

76 Reported (The Third Run)

77 Likely Falsification Falsification is manipulating research materials, equipment, or processes, or changing or omitting data or results such that the research is not accurately represented in the research record

78 If there were genuine reasons the first two runs didn’t work you ought to document why, fix them and repeat the study a 4 th time! “…not accurately represented in the research record”

79 Example 8: Mouse Model Genetically engineered mouse model suggests that protein X promotes metastases Scientist shares an antibody to human protein X with the collaborating pathologist studying human disease The antibody doesn’t label any human metastases, in fact, it is only expressed in non- metastatic lesions

80 Mouse Model Manuscript published reads “Protein X Promotes Metastases” Is this selective reporting of data?

81 Mouse Model Manuscript published reads “Protein X Promotes Metastases in a Mouse Model” Is this selective reporting of data?

82 Mouse Model What was suggested in the paper? Was the discussion always focused on mouse models or did it stray into suggesting that protein X is important in humans?

83 Example 9: A New Drug to Cure Depression The P.I. develops a new drug to treat depression It works on 100 of 103 patients The investigators go back and review the charts on the three patients on whom the drug didn’t work and on re-review it is clear that the 3 patients have manic depressive illness The drug is reported to be effective in 100% of patients with depression

84 Dangers of Re-Review of Selected Data …such that the research is not accurately represented in the research record

85 Example 10: PowerPoint Presentation Within Hopkins Falsified data are presented at a meeting within the Hopkins community. The data are not published. Is this research misconduct?

86 Yes, it is research misconduct even if the data are not published

87 Image Manipulation Falsification is manipulating research materials, equipment, or processes, or changing or omitting data or results such that the research is not accurately represented in the research record

88 M. Rossner and K. Yamada, JCB, 2004 It‘s so easy to add and subtract with Photoshop

89 Rubber Stamp to “Clean” the Background M. Rossner and K. Yamada, JCB, 2004

90 “Your X-ray showed a broken rib, but we fixed it in Photoshop”

91 Image Manipulations M. Rossner and K. Yamada, JCB, 2004

92 Science 2005Woo-Suk Hwang

93

94 Time, Dec. 15, 2005 Woo-Suk Hwang

95 Reusing Images- Potti Augustine et al., 2009, Clin Can Res, 15:502-10, Fig 4A. Temozolomide, NCI-60. Hsu et al., 2007, J Clin Oncol, 25:4350-7, Fig 1A. Cisplatin, Gyorffy cell lines.

96 M. Rossner and K. Yamada, JCB, 2004 Altering Images If you misrepresent your data, you are deceiving your colleagues, who expect and assume basic scientific honesty— that is, that each image you present is an accurate representation of what you actually observed. In addition, an image usually carries information beyond the specific point being made.

97 Altering Images M. Rossner and K. Yamada, JCB, 2004 Data must be reported directly, not through a filter based on what you think they “should” illustrate to your audience. For every adjustment that you make to a digital image, it is important to ask yourself, “Is the image that results from this adjustment still an accurate representation of the original data?” If the answer to this question is “no,” your actions may be construed as misconduct.

98 Image Manipulation Falsification is manipulating research materials, equipment, or processes, or changing or omitting data or results such that the research is not accurately represented in the research record

99 Data Integrity 1.Fraud in the history of science 2.How common is data fraud? 3.Fraud harms patients, the institution and the investigator 4.Examples of inappropriately published data 5.How can you prevent fraud in your own lab?

100

101 Prevention!! Establish a culture of honesty above all in your lab Inform and educate Screen- Periodically ask to see lab books Detect problems by working closely with primary data

102 Establish a Culture of Honesty “We need this difference to be significant or I won’t get my grant” “These data points don’t fit the results I expected” These small things can add up and can quickly become the norm

103 Establish a Culture of Honesty vs. From day one; “All that matters to me is that the results you present are 100% honest”

104 Inform and Educate Dedicate some journal clubs or lab group meetings to educating those under you on the importance of academic integrity Encourage members of your lab to attend lectures such as this one!

105 Even so we have to screen for problems!

106 Picture of a PowerPoint presentation We are not going to detect fraud if we only look at PowerPoint presentations of finished results

107 We need to carefully review and question primary data

108 Henry L. Mencken “Conscience is the inner voice that warns us somebody may be looking”

109 If it is Too Good to be True Blind the samples and ask the person to rerun the experiment Have someone else in the lab rerun the experiment

110 Tools for detecting misconduct Anti-plagiarism software (eTBLAST, CrossCheck, Turnitin) Screening images (PhotoShop)- Pioneered by J Cell Biology. See M. Rossner and K. Yamada, JCB 2004; 166: found 1% unacceptable manipulation Data Review (digit preference) Liz Wager, Council of Scientific Editors

111 Conclusions Preventing damage would save careers from ruin Everyone has a responsibility to promote a culture in which research misconduct does not happen Harold C. Sox, Annals of Internal Medicine

112 If you are the first or last author on a paper You are responsible: 1.For making sure all of the other authors have read and approved the manuscript 2.For everything in the manuscript- make sure the images included are correct, that the text isn’t copied from somewhere else, that the data weren’t manipulated, that you have appropriate IRB protocols, and that the protocols were followed

113 We need to be aware that at a place like Johns Hopkins people may feel enormous pressures Take Home Message #1

114 Science should be our “touchstone” Take Home Message #2

115 The currency of science is the peer- reviewed and peer-accepted manuscript that is backed by a gold standard of scientific integrity and scrupulous honesty. Anything that tarnishes this gold standard threatens to devalue the worth of scientific currency. Ultimately, society itself suffers because scientific advancement prepares the way for social progress Curt Civin Editor-in-Chief, Stem Cells “Cloned Photomicrographs, not cloned cells”

116 Panel Discussion Ralph Hruban, M.D. Bob Bollinger, M.D., M.P.H. Curt Civin, M.D. Anirban Maitra, M.B.B.S. Sheila Garrity, J.D., M.P.H., M.B.A. – Moderator


Download ppt "Research Integrity, The Importance of Data Acquisition and Management Ralph H. Hruban, M.D. Monday, February 13, 2011."

Similar presentations


Ads by Google