Publication Bias in Medical Informatics evaluation research: Is it an issue or not? Mag. (FH) Christof Machan, M.Sc. Univ-Prof. Elske Ammenwerth Dr. Thomas Bodner

Overview Background Detecting publication bias - 2 attempts –study 1: A simplistic approach –study 2: The funnel plot Discussion Conclusions / Outlook

Publication bias - 2 definitions Background Occurs when research that is readily available differs in its results from all the research that has been done in the area. [Rothstein et al 2005] Publication bias is the tendency on the parts of investigators, reviewers and editors to submit or accept manuscripts for publication based on the direction or strength of the study findings. [Dickersin 1990]

Publication bias in context Publication bias and other related biases can be summarised as statistically significant, 'positive' results being: more likely to be published (publication bias) more likely to be published rapidly (time lag bias) more likely to be published in English (language bias) more likely to be published more than once (multiple publication bias) more likely to be cited by others (citation bias) [The Cochrane Collaboration] Background

Why think about publication bias evidence-based practice systematic review as best evidence –should include as many publications on the question invesigated as possible –high danger of publication bias leading to wrong conclusions Strong evidence for publication bias in other fields (social sciences, biomedicine) What about evaluation literature in medical informatics? Background

Results 72 publications reported clear results (positive or negative) –60 positive –12 negative Study 1: Detecting publication bias - a simplistic approach Positive / negative papers positivenegative Detecting PB Assumption If there is a remarkable difference between publications reporting positive and publications reporting negative results in a random sample publication bias is a possible reason. (see also Dickersin 1990) Methods random sample of 86 MI evaluation publications out of evalDB Classify / Count / Compare

graphical depiction of publication bias easy to understand scatter plot displaying study quality (e.g. sample size, standard error,...) and effect size recommended by Cochrane Source: effect size n n Detecting PB Study 2: Detecting publication bias - the funnel plot - basics

Methods study design: meta-analysis Search in Medline and Embase for controlled trials on CPOE and medication errors / ADEs Hand Search in several journals intervention: CPOE system effect size: relative risk for medication errors / ADEs between intervention and control group 26 studies included assessment of publication bias by funnel plot Study 2: Effects of CPOE on medication errors / ADEs Detecting PB Assumption Missing publications of low study quality reporting on negative and / or non-significant results may be an indication for publication bias.

Study 2: Effects of CPOE on medication errors / ADEs Detecting PB

Discussion Study 1: 60 out of 72 studies positive seems a high number limitations: –assumption that ratio of positive / negative studies equals 50:50 not valid –other biases (e.g. langauge bias) –chance –... Discussion Study 1: plot indicates sign for publication bias limitations: –other selection biases (language, citation, etc.) –poor methodological quality of smaller studies –true heterogenity –chance

Conclusions / Outlook Publication bias is an issue existence hard to proof easy to jump to wrong conclusions awareness of authors important quality of publications must be sufficient Evidence-based medical informatics Conclusions / Outlook