Presentation is loading. Please wait.

Presentation is loading. Please wait.

Integrity and human factors in behavioral research Moving beyond Questionable Research Practices Jelte M. Wicherts 1.

Similar presentations


Presentation on theme: "Integrity and human factors in behavioral research Moving beyond Questionable Research Practices Jelte M. Wicherts 1."— Presentation transcript:

1 Integrity and human factors in behavioral research Moving beyond Questionable Research Practices Jelte M. Wicherts 1

2 Integrity in black and white 2 Uninterested in prestige Critical of own results Reliable and rigorous Open and honest Interested in quality Seeks “truth” Good Interested in prestige Critical of results of others Unreliable and sloppy Secretive and dishonest Interested in quantity Seeks support for own theories Dr. Evil ?

3 3 Integrity in 50 shades of grey

4 A former professor: “I was getting better and better in using techniques to improve poor results. […] What I did was not as white as snow, but it was not pitch-dark either. It was grey and it was common. How else could all the others get all those beautiful results? […] After years of balancing on the cliff, the grey became darker black, and finally I fell all the way down.” Source: D. Stapel, 2012, p. 143-144; my translation “Ik werd er ook steeds bedrevener in gebruik te maken van technieken die matige resultaten konden oppompen. […] Wat ik deed, was niet keurig wit, maar ook niet pikzwart. Het was grijs en het was usance. Hoe kwamen al die anderen anders aan al die schitterende resultaten? […] Na jaren van voor en op het randje balanceren werd het grijs steeds donkerder zwart, en uiteindelijk donderde ik naar beneden.“

5 5 Blackboard in room with PhD students

6 The “bump” below p=.05 6 Source: Masicampo, E. J. & Lalande, D. R. (2012). A peculiar prevalence of p values just below.05. Quarterly Journal of Experimental Psychology, 65, 2271-2279. Based on 3,627 p values from 2008 issues of Journal of Experimental Psychology: General, Journal of Personality and Social Psychology, and Psychological Science.

7 Social neuroscience Vul et al. documented correlations between evoked blood oxygenation level dependent response and behavioral measures of individual differences in 55 fMRI studies that appear inflated. 7 Source: Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009). Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspectives on Psychological Science, 4, 274-290. p =.05 for N=18

8 Sociology and Political Science 8 Gerber, A. S., & Malhotra, N. (2008). Publication bias in empirical sociological research - Do arbitrary significance levels distort published results? Sociological Methods & Research, 37, 3-30. Gerber, A. S., & Malhotra, N. (2008). Do statistical reporting standards affect what is published? Publication bias in two leading political science journals. Quarterly Journal of Political Science, 3, 313-326.

9 9 Success rates (too high?) Source: Fanelli, D. (2010). “Positive” results increase down the hierarchy of the sciences. PloS one, 5(4), e10068. With med. power <.50

10 Commonality: Scientists openly share findings with colleagues. Secrecy: Scientists protect their newest findings to ensure priority in publishing [..] Secrecy: Scientists protect their newest findings to ensure priority in publishing [..] Universalism: Scientists evaluate research only on its merit, i.e., according to accepted standards of the field. Particularism: Scientists assess new knowledge […] based on reputation […] of the individual or research group. Particularism: Scientists assess new knowledge […] based on reputation […] of the individual or research group. Source: Anderson, M.S., Martinson, B. C., & De Vries, R. (2007). Journal of Empirical Research on Human Research Ethics, 2 (4), 3-14 10 Survey among 3,247 US scientists, asking: 1)Whether they subscribed to norms of “good science” 2)Whether they behaved according to these norms 3)Whether their typical colleague behaved according to these norms Norms vs. Counternorms

11 Governance: Scientists are responsible for the direction and control of science through governance, self-regulation and peer review. Administration: Scientists rely on administrators to direct the scientific enterprise through management decisions. Quality: Scientists judge each others’ contributions to science primarily on the basis of quality. Quantity: Scientists assess each others’ work primarily on the basis of numbers of publications and grants. Source: Anderson, M.S., Martinson, B. C., & De Vries, R. (2007). Journal of Empirical Research on Human Research Ethics, 2 (4), 3-14 11 Disinterestedness: Scientists are motivated by the desire for knowledge and discovery. Self-Interestedness: Scientists compete with others in the same field for funding and recognition of their achievements. Organized Skepticism: Scientists consider all new evidence, hypotheses, theories, and innovations, even those that challenge or contradict their own work. Organized Dogmatism: Scientists invest their careers in promoting their own most important findings, theories, or innovation.

12 Do regard their colleagues highly? Source: Anderson, M.S., Martinson, B. C., & De Vries, R. (2007). Journal of Empirical Research on Human Research Ethics, 2 (4), 3-14 norm>counternorm norm=counternorm norm { "@context": "http://schema.org", "@type": "ImageObject", "contentUrl": "http://images.slideplayer.com/8/2130528/slides/slide_12.jpg", "name": "Do regard their colleagues highly.Source: Anderson, M.S., Martinson, B.", "description": "C., & De Vries, R. (2007). Journal of Empirical Research on Human Research Ethics, 2 (4), 3-14 norm>counternorm norm=counternorm norm

13 Do researchers share? “Forty-seven percent of [1240 surveyed] geneticists [from 100 US universities] who asked other faculty for additional information, data, or materials regarding published research reported that at least 1 of their requests had been denied in the preceding 3 years.” Of 200 authors who indicated that additional information of their paper in economy & business “was available upon request”… only 44% actually shared the information. 13 Source: Campbell, E. G et al. (2002). Data withholding in academic genetics: Evidence from a national survey. Journal of the American Medical Association, 287, 473-480. Krawczyk, M. & Reuben, E. (2012). Accountability in Research: Policies and Quality Assurance, 19, 175-186.

14 50 journals with highest IFs in science 44 journals had a “data sharing policy” in 2009 From the 351 articles that were subject to data sharing rules, data from 143 articles (40.7%) were published None of the datasets of 149 articles that were not subject to sharing policy were published Bron: Alsheikh-Ali, A. A., Qureshi, W., Al-Mallah, M. H., & Ioannidis, J. P. A. (2011). Public availability of published research data in high-impact journals. PLoS One, 6, e24357. doi: 10.1371/journal.pone.0024357 14 Do researchers follow journal policy?

15 15 Do researchers share data upon request? In 2005, we requested the raw data from 141 papers published in four APA journals for use in a study of the effects of outliers on the outcome of data analyses. Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006). The poor availability of psychological research data for reanalysis. American Psychologist, 61, 726-728.

16 16 Responses 1.This is an ongoing project, our IRB cannot allow it 2.I have no time to do this…I’m up for tenure 3.My research assistant/postdoc/student left 4.I recently moved, I have a new computer! 5.“I am afraid your request is not possible” 6.This will take me some time, I’ll get back to you 7.I’ll send you the data within a few days

17 Bakker, M. & Wicherts, J. M. (2011). (Mis)reporting of statistical results in psychology journals. Behavior Research Methods, 43, 666-678. Results: 128 papers (50%) contained at least one error 39 papers (15%) contained at least one error related to p =.05 Conclusion: Errors predominantly led to “better” results Results: 128 papers (50%) contained at least one error 39 papers (15%) contained at least one error related to p =.05 Conclusion: Errors predominantly led to “better” results 17 Method: a representative sample of 257 papers Recomputed 4720 p-values from NHST and checked for consistency Method: a representative sample of 257 papers Recomputed 4720 p-values from NHST and checked for consistency p =.06 Are statistical results checked by (co- )authors and reviewers?

18 Reporting errors in papers from which data were or were not shared 18 DATA SHARED (N=21) Wicherts, J. M., Bakker, M., & Molenaar, D. (2011). Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PLoS ONE, 6, e 26828. DATA NOT SHARED (N=28)

19 Gross reporting errors (around p=.05) 19 DATA NOT SHARED (N=28) DATA SHARED (N=21) Wicherts, J. M., Bakker, M., & Molenaar, D. (2011). Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PLoS ONE, 6, e 26828.

20 Errors and data sharing Haphazard data documentation plays a role in reluctance to share and occurrence of errors. Poor data documentation also suggest that authors hardly share data with co-authors. 20

21 A former professor: “So when it finally happened, it was really, really easy. I was always alone. No one ever checked me. They trusted me. I did everything myself. […] Every psychologist has a tool box full of statistical and methodological procedures for when things don’t work out well. I used these procedures to erase ugly findings and to shine up mediocre results.” Dus toen het eenmaal zover was, was het heel, heel, heel makkelijk. Ik was altijd alleen. Niemand die me ooit controleerde. Ze vertrouwden me. Ik deed alles zelf. […] Elke psycholoog heeft een gereedschapskist vol met statistische en methodologische procedures voor als het even tegenzit. Ik gebruikte deze procedures om lelijke bevindingen weg te gummen en matige resultaten op te poetsen. Source: D. Stapel, 2012, p. 164-165. 21

22 22 Shalvi et al., 2011, Organizational Behavior and Human Decision Processes

23 23 Effect! ? P>.05 Add 10 cases P>.05 P<.05 Planned analysis P<.05 Write paper P>.05 Call this a “failed study” Perform new study Redo analysis with adapted dependent var. P>.05 ? Remove outliers (Z > |2|) Grey zone map Bakker, M., van Dijk, A, & Wicherts, J. M. (2012). The Rules of the Game Called Psychological Science. Perspectives on Psychological Science, 7, 543-554. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology. Psychological Science, 22, 1359 –1366.

24 24 Wicherts, J. M., Bakker, M., & Molenaar, D. (2011). PLoS ONE, 6, e 26828. Data shared? 10 errors! significant non- significant Willingness to share research data is related to the strength of the evidence

25 Do researchers act strategically? Suppose you have resources for N=100. What do you do when you expect d =.40? 1)Run one large study with power =.508 2)Run five small studies (of N=20 each) with power.135 25

26 Size matters 26 Bakker, M., van Dijk, A, & Wicherts, J. M. (2012). The Rules of the Game Called Psychological Science. Perspectives on Psychological Science, 7, 543-554.

27 Size matters 27 Bakker, M., van Dijk, A, & Wicherts, J. M. (2012). The Rules of the Game Called Psychological Science. Perspectives on Psychological Science, 7, 543-554.

28 Distribution under H A for N=50 Distribution under H 0 for N = 50 Excess of significant results: Χ 2 = 6.21, p =.013 Asymmetric funnel plot: Z = 2.24, p =.025 Excess of significant results: Χ 2 = 6.21, p =.013 Asymmetric funnel plot: Z = 2.24, p =.025 Habituation to stimuli and later IQ

29 29 Bakker, M., van Dijk, A, & Wicherts, J. M. (2012). The Rules of the Game Called Psychological Science. Perspectives on Psychological Science, 7, 543-554.

30 Are researchers only human! 30 This one SHOULD really be higher! If not my reviewers will kill my paper And I can forget about getting tenure … And I cannot buy the house I wanted

31 Tragedy of the commons 31

32 To sum up Are researchers ambitious & successful? Do they regard their colleagues highly? Do researchers share information and data? Are statistical results checked by (co-)authors and reviewers and are data shared with co-authors? Do researchers act strategically? Do they feel justified to do so? Do they publish all of their hard work? Do they debate choices during data analyses? 32 ✔ ✗ ✗ ✔ ✗ ✔ ✗ ✗

33 Some solutions Let your co-authors (or colleagues) replicate your analyses (the co-pilot model) Openness concerning analytic choices Team up & replicate Publish the data Pre-registration of studies & protocols Power-up main study Ban the term “failed study” 33 Bakker, M., van Dijk, A, & Wicherts, J. M. (2012). The Rules of the Game Called Psychological Science. Perspectives on Psychological Science, 7, 543-554. Wicherts, J. M. (2011). Psychology must learn a lesson from fraud case. Nature, 480, 7. Wicherts, J. M. & Bakker, M. (2012). Publish (your data) or (let the data) perish! Why not publish your data too? Intelligence, 40, 73-76.


Download ppt "Integrity and human factors in behavioral research Moving beyond Questionable Research Practices Jelte M. Wicherts 1."

Similar presentations


Ads by Google