Karin Schurink Peter Lucas Marc Bonten Stefan Visscher Incorporating Evaluation into the Design of a Decision-Support System UMC Utrecht Radboud University Nijmegen

Ventilator-Associated Pneumonia  VAP  Pneumonie acquise sous ventilation mécanique  Beademings-gerelateerde Longontsteking

Contents 1.Problem: incorporating evaluation into the design of a Decision-Support System (DSS) 2.Approach: A DSS for Ventilator-Associated Pneumonia (VAP): its underlying Bayesian-network and decision-theoretic model Clinical setting: advising regarding the diagnosis and treatment of VAP in the ICU Design of DSS guided by evaluation considerations

Introduction Ventilator-Associated Pneumonia (VAP)VAP Nosocomial infection on Intensive Care Units (ICU) Gold Standard = infectious-disease specialists Decision-Support System in uncertainty helps ­ diagnosing the patient ­ finding optimal treatment

Previous research in diagnosing and treating disorders in patients Evidence-based clinical guidelines Medical Decision-Support Systems Uncertainty: Bayesian network

General: Bayesian Network; 2 parts 1.qualitative = structure and relations between nodes (vars) 2.quantitative = conditional (estimated) probabilities example: P(pneumonia | body temperature  38.5 °C) A DSS for VAP: its underlying Bayesian-network and decision-theoretic model (1) Bayesian Network for VAP; 2 parts 1.diagnostic = clinical signs/ symptoms, duration of stay, mechanical ventilation 2.therapeutic = most effective combination of antibiotic treatment

Symptoms/ signs: temperature (fever) leukocytosis radiological signs sputum production Risk factors mechanical ventilation hospitalisation colonisation antibiotic use

Decision-theoretic model  Providing utilities for combinations of antibiotics, taking into account side effects financial costs antimicrobial spectrum A DSS for VAP: its underlying Bayesian-network and decision-theoretic model (2) utility : a quantitative measure of the strength of the expert’s preference in decision making

A DSS for VAP: its underlying Bayesian-network and decision-theoretic model (3) Global structure of the Bayesian Network

The problem: incorporating evaluation into the design of a DSS Has the DSS been properly evaluated? Diagnostic performance Usability + effect

Diagnostic performance of the DSS AUC Dataset: rows/ patient days in the ICU 157 VAP days experts vs. DSS

Design for an evaluation study of the DSS for VAP Known systematic effects/ biases in an evaluation study: 1.volunteer effect 2.Hawthorne effect 3.checklist effect

Resulting DSS for VAP (1) Components Clinical database User interface

Added facilities for evaluation 1.all users are asked to use the system 2.the doctor is asked to give his/ her opinion, choice of treatment and motivation for this choice concerning the patient 3.patient’s symptoms are presented: this form or checklist can be edited when necessarypatient’s symptoms are presented 4.at random, a management advice is presented: i. probability of VAP ii. optimal treatment for possible colonisation of the respiratory tract 5.only when an advice is presented, the doctor is asked again whether a VAP is suspected Resulting DSS for VAP (2)

all users (volunteer/ Hawthorne) (checklist)checklist

Screenshot

Conclusions & future work Medical Decision-Support Systems are meant to assist clinicians in the difficult process of medical management Known biases in evaluation studies were discussed More attention should be given to evaluation issues: only then it is possible to perform a reliable evaluation of a DSS In future we intent to perform the described evaluation study  did the clinician revise his/ her judgement, taking into account the system’s advice?  did the system’s advice influence the clinician’s diagnosis?

Contact: Stefan Visscher