1. Dealing with bias Boyce Sigweni Martin Shepperd NB no cats were harmed in the making of this presentation!

3. What is bias? “[L]et us define bias as the combination of various design, data, analysis, and presentation factors that tend to produce research findings when they should not be produced.” John Ioannidis (2005)

4. What is (the) scientific method? Guidelines, methods, procedures and evaluation criteria to reduce bias – reduce subjectivity, promote objectivity Other issues but bias is central

5. Blinding Withholding information to reduce bias, e.g., – blind reviewing – blind treatments to participants (placebo) – blind treatments to experimenters (double blind) – blind analysis

6. Blinding Hierarchy Open Single blind Double blind Complete blind (including the analyst)

7. Some work on blind analysis Over to Boyce …

8. Bias harms randomisation Random implies selection cannot be predicted in advance beyond the natural odds. Sample selection to estimate population parameters Assignment of experimental units / participants to treatments Choosing data sets

9. Cats and survival odds A study shows the odds slightly improve the further a cat falls. Used data supplied by vets in NY. Why? What's the problem?

10. Do Finches help novices learn to code? Analysis by cohort What's the problem? Solutions?

11. Predicting software defects Same old … But inference into all software? Some data sets may favour some learners

12. Random doesn't mean unusual things can't happen! Consider stratified randomisation (reduce confounding) Type I and II errors can, and do, occur! Measurement error (random or biased?) Meta-analysis

13. Meta-analysis and bias Publication bias – file drawer problem – selective reporting of results – refereeing bias against 'negative' results

14. Problems with blinding Impractical Unethical Sometimes can address by a crossover design