1. eHealth Partners Finland Finnish Agency for Technology and Innovation Tekes grants no. 40140/06 and 70030/06 www.uku.fi/ehp Evaluation studies support better design and implementation of health information systems P Nykänen, S Wallden, A Seppälä, J Mäkinen Tampere University, Department of Computer Sciences, eHealth Research Group

2. eHealth Partners Finland www.uku.fi/ehp Nykänen et al.2 Evaluation - definitions, perspectives ● Evaluation is a means to measure and assess the properties of an information system in a given context ○ in planning, development, implementation, installation, operation ● The result of evaluation informs a decision to be made concerning the measured system in the specific context ○ measurements are done and judged or used in a context, there is always a question to be answered Brender J, Handbook of evaluation methods in health informatics. Springer Verlag 2006

3. eHealth Partners Finland www.uku.fi/ehp Nykänen et al.3 Evaluation frameworks ● Many approaches and models exist, multidisciplinary, with focus on various aspects, no single globally accepted methodology available ● Evaluation of health information systems requires consideration and understanding of the domain problem, context, social and organisational environment issues Evaluation is always focussed on the health information system in a given context, on the use and user context EFMI WG HISEVAL Guidelines under preparation: ○ Statements for reporting health IT evaluation studies ○ Good practice guidelines for evaluation of health IT systems

4. eHealth Partners Finland www.uku.fi/ehp Nykänen et al.4 eHP_evaluation study: Effectiveness ●Seamless service chain where patient data can be accessed and transferred over organisational boundaries (with the patient’s consent) was achieved, ○Work flows, mode of action remained as they were in the manual world >> planned organisational changes not achieved ○Two practices: IT-based and manual-based ●Developed IS brought only minor add-on value in patient data management ○IS use on voluntary basis, all actions could be taken manually ○IS use was expensive (cost-model) ○Lack of committment from all stakeholders

5. eHealth Partners Finland www.uku.fi/ehp Nykänen et al.5 eHp evaluation study: Usability 1.What makes usability of a patient information system good? ●The most important usability factor was efficiency, followed by user satisfaction and learnability ●Usability requirements were different for various professionals depending on their profession, task and work context, even inside one user group, different levels of usability were found 2. What is good usability for a patient information system? ●With homogenous patient groups, efficiency can be improved by using default values and templates ●With heterogenous patient groups, efficiency can be improved by using e.g. clear and understandable summaries of patients’ medical history

6. eHealth Partners Finland www.uku.fi/ehp Nykänen et al.6 Research issues in evaluation  IS design / adoption / implementation  IS design / adoption / implementation should result in an organisational change  Evaluation as integral to design / adoption / implementation  Mutual transformation, technology <>organisation  Administrative and management aspects  Analysis of success and failure issues  Socio-technical aspects of health IS  organisational aspects, work flows and tasks of the users  Usability in relation to IS design methods and approaches ○Usability of a health IS is context-dependent User requirements are dependent on the use context, task and user ○ Usability requirements should be considered from the viewpoint of various users and clinical pathways ○Usability concept <>Technology acceptance models focus much on users and technology, not that much on clinical environment, task  Human factors engineering approach  Evaluation methods and approaches  Good practices for studies  Reporting guidelines

7. eHealth Partners Finland www.uku.fi/ehp Nykänen et al.7 EVIDENCE-BASED HEALTH INFORMATICS To build up EVIDENCE-BASED HEALTH INFORMATICS ● IS design & development & evaluation ○ Information systems science - Scientific approaches and methods to be applied ○ Integral parts of the ISD process ● Constructive evaluation – Assess how health information systems serve the business/service objectives now and where are the challenges for improvement during the system lifecycle ○ Evaluation of those qualities and aspects that concern or effect on many users and use situations Human, social and organisational aspects!

8. eHealth Partners Finland www.uku.fi/ehp Nykänen et al.8 Thank you for your attention! Pirkko.Nykanen@uta.fi Tampere University Department of Computer Sciences www.cs.uta.fi Pirkko.Nykanen@uta.fi www.cs.uta.fi