1. P-Hacking: A Practical Guide
@Neuro_Skeptic

3. The Nine Circles of Dante’s Inferno
First Circle: Limbo
Second Circle: Lust
Third Circle: Gluttony
Fourth Circle: Greed
Fifth Circle: Anger
Sixth Circle: Heresy
Seventh Circle: Violence
Eighth Circle: Fraud
Ninth Circle: Treachery

5. The Nine Circles of Scientific Hell
First Circle: Limbo
Second Circle: Overselling
Third Circle: Post-Hoc Storytelling
Fourth Circle: P-Value Fishing
Fifth Circle: Creative Use of Outliers
Sixth Circle: Plagiarism
Seventh Circle: Non-Publication of Data
Eighth Circle: Partial Publication of Data
Ninth Circle: Inventing Data

6. P-Fishing Fourth Circle: P-Value Fishing
“Those who tried every statistical test in the book until they got a p value less than 0.05 find themselves here, an enormous lake of murky water. Sinners sit on boats and must fish for their food. Fortunately, they have a huge selection of different fishing rods and nets (brand-names include Bayes, Student, Spearman and many more). Unfortunately, only one in 20 fish are edible, so they are constantly hungry.”

7. P-Fishing Also known as… Questionable Research Practices (QRPs))
Torturing the data
P-Hacking
Outcome reporting bias
Undisclosed flexibility
Researcher Degrees of Freedom
…and more.

15. P-Hacking Works!
Collect some data

16. P-Hacking Works! Collect some data
Try many statistical tests on the same data

17. P-Hacking Works! Collect some data
Try many statistical tests on the same data
Or try many variants of the same data (e.g. removing ‘outliers’.)

18. P-Hacking Works! Collect some data
Try many statistical tests on the same data
Or try many variants of the same data (e.g. removing ‘outliers’.)
Or try looking at different variables within the dataset

19. P-Hacking Works! Collect some data
Try many statistical tests on the same data
Or try many variants of the same data (e.g. removing ‘outliers’.)
Or try looking at different variables within the dataset
Report the analyses that give the most favourable results (usually the lowest p- values).

20. Why Hacking Is So Useful
There are many choices (‘researcher degrees of freedom’) in data analysis.
For example, in a simple task-based fMRI data analysis, Joshua Carp found 7000 combinations of parameters (very conservative).
Carp, J. (2012).On the plurality of (methodological) worlds: estimating the analytic flexibility of fMRI experiments
Frontiers in Neuroscience

21. How Not To Get Caught
There are many tools, but perhaps the most intuitive: the p curve
Try it now!
Simonsohn, U. Nelson, L. D. Simmons, J. P. (2013). P-curve: a key to the file-drawer.
Journal of Exp. Psychol General

22. How It Can Be Stopped
Smulders YM (2013). A two-step manuscript submission process can reduce publication bias.
Journal of Clinical Epidemiology
Chambers CD (2013). Registered Reports: a new publishing initiative at Cortex
Cortex

27. Happy hacking! @Neuro_Skeptic neuroskeptic@googlemail.com