Publication of Evaluation Studies: Challenges & Guidelines for authors Elske Ammenwerth UMIT - University for Health Sciences, Medical Informatics and Technology, Hall, Austria

Organisers EFMI Working Group „Assessment of Hospital Information Systems“ ( IMIA Working Group „Technology Assessment and Quality Development in Health Informatics“

Motivation Increasing reports on problems with IT in health care Insufficient integration into clinical workflow Insufficient integration between IT systems Usability problems Danger of errors caused or increased by IT Low user acceptance, user boycott Project failures Loss of money, loss of reputation Bad Health Informatics can kill: Examples where insufficient IT system can hinder processes and harm patients Examples at:

Need for Evaluation: Declaration of Innsbruck Evaluation contributes to better IT systems in health care Evaluation support continuous monitoring and review of IT Evaluation is an ethical imperative To detect problems as early as possible To learn from problems and errors To steadily improve IT systems To contribute to better health care Ammenwerth E, Brender J, Nykänen P, Prokosch H-U, Rigby M, Talmon J, et al. Visions and strategies to improve evaluation of health information systems Reflections and lessons based on the HIS-EVAL workshop in Innsbruck. Int J Med Inf Jun 30;73(6):

Evidence-Based Health Informatics IT systems in health care largely affect quality and efficiency of health care All decisions with regard to IT systems should be grounded on best available evidence! Example: Hospital considers introducing a CPOE system What is the available evidence? What will be the benefit? Which side effects may occur? How to best introduce the CPOE system? What are the costs?

EBHI is based on publications Evidence-Based Health Informatics aggregates available evidence Systematic review: Descriptive collection of evidence Example: „11 from 15 studies on CPOE show a significant reduction of medication errors“ Meta-Analysis: Mathematical aggregation of available evidence

Need for publication Evidence-Based Health Informatics is only possible sufficient high-quality evidence is published! IT evaluation studies should be published Publications should be of high quality (complete, clear, …) Publications should be available and searchable

Challenges Evidence-based health informatics is based on publications: How much evaluation studies are un-published? Many publications are of low quality: How to write a „good“ evaluation paper?

Programm of workshop 1.Introduction: Evidence-based health informatics and publication 2.Publication bias in health informatics: How many studies are published? Why are studies not published? 3.STARE-HI - Guidelines for authors: What information should be contained in a publication of an IT evaluation? 4.Summary and Conclusion

2. Publication Bias a) What is publication bias? b) Survey on publication bias c) How to detect publication bias?

Publication bias Occurs when research that is readily available differs in its results from all the research that has been done in the area. [Rothstein et al 2005] Publication bias is the tendency on the parts of investigators, reviewers and editors to submit or accept manuscripts for publication based on the direction or strength of the study findings. [Dickersin 1990]

Publication bias Studies have a larger chance to be published when they show a significant results, i.e. a positive effect of the evaluated IT system Authors (often involved in the project!) may not want to present unsuccessful implementations. Editors may favour studies showing an interesting (positive) effect.

Publication bias in context Publication bias and other related biases can be summarised as statistically significant, 'positive' results being: more likely to be published (publication bias) more likely to be published rapidly (time lag bias) more likely to be published in English (language bias) more likely to be published more than once (multiple publication bias) more likely to be cited by others (citation bias) [The Cochrane Collaboration]

Publication bias The problem with publication bias: The published evidence is systematically biased towards positive results! Reviews and meta-analysis are based on published evidence and thus will come to biased conclusions!

How many IT evaluation studies are NOT published? What are reasons for not publishing? Is there a bias towards positive findings = publication bias? Questions