Dealing with bias Boyce Sigweni Martin Shepperd NB no cats were harmed in the making of this presentation!

What is bias? “[L]et us define bias as the combination of various design, data, analysis, and presentation factors that tend to produce research findings when they should not be produced.” John Ioannidis (2005)

What is (the) scientific method? Guidelines, methods, procedures and evaluation criteria to reduce bias – reduce subjectivity, promote objectivity Other issues but bias is central

Blinding Withholding information to reduce bias, e.g., – blind reviewing – blind treatments to participants (placebo) – blind treatments to experimenters (double blind) – blind analysis

Blinding Hierarchy Open Single blind Double blind Complete blind (including the analyst)

Some work on blind analysis Over to Boyce …

Bias harms randomisation Random implies selection cannot be predicted in advance beyond the natural odds. Sample selection to estimate population parameters Assignment of experimental units / participants to treatments Choosing data sets

Cats and survival odds A study shows the odds slightly improve the further a cat falls. Used data supplied by vets in NY. Why? What's the problem?

Do Finches help novices learn to code? Analysis by cohort What's the problem? Solutions?

Predicting software defects Same old … But inference into all software? Some data sets may favour some learners

Random doesn't mean unusual things can't happen! Consider stratified randomisation (reduce confounding) Type I and II errors can, and do, occur! Measurement error (random or biased?) Meta-analysis

Meta-analysis and bias Publication bias – file drawer problem – selective reporting of results – refereeing bias against 'negative' results

Problems with blinding Impractical Unethical Sometimes can address by a crossover design