What is bias? “[L]et us define bias as the combination of various design, data, analysis, and presentation factors that tend to produce research findings when they should not be produced.” John Ioannidis (2005)
What is (the) scientific method? Guidelines, methods, procedures and evaluation criteria to reduce bias – reduce subjectivity, promote objectivity Other issues but bias is central
Blinding Withholding information to reduce bias, e.g., – blind reviewing – blind treatments to participants (placebo) – blind treatments to experimenters (double blind) – blind analysis
Blinding Hierarchy Open Single blind Double blind Complete blind (including the analyst)
Bias harms randomisation Random implies selection cannot be predicted in advance beyond the natural odds. Sample selection to estimate population parameters Assignment of experimental units / participants to treatments Choosing data sets
Cats and survival odds A study shows the odds slightly improve the further a cat falls. Used data supplied by vets in NY. Why? What's the problem?
Do Finches help novices learn to code? Analysis by cohort What's the problem? Solutions?
Predicting software defects Same old … But inference into all software? Some data sets may favour some learners
Random doesn't mean unusual things can't happen! Consider stratified randomisation (reduce confounding) Type I and II errors can, and do, occur! Measurement error (random or biased?) Meta-analysis
Meta-analysis and bias Publication bias – file drawer problem – selective reporting of results – refereeing bias against 'negative' results
Problems with blinding Impractical Unethical Sometimes can address by a crossover design