1. Declaration of Innsbruck Jan Talmon - Maastricht University Elske Ammenwerth - UMIT

2. Content Reasons for paying attention to evaluation Introduction Some key observations and recommendations Current activities

3. Reasons for concern Scientific Building an evidence base Managerial Support decisions

4. Introduction 1-2 Interest in evaluation since early 90-ties Mostly decision support Sydpol, COMAC-BME, KAVAS, Eurodiabeta Growing interest in evaluation of Information Systems mid 90-ies ATIM, VATAM Lack of agreed upon evaluation frameworks

5. Introduction 2-2 Working groups of IMIA and EFMI HISEVAL meeting Innsbruck April 2003 3 activities: GEP-HI, STARE-HI, EP-HI Declaration of Innsbruck

6. Declaration - Preamble Evaluation of Health Information Systems is morally imperative Intended to improve functioning of health professionals Has impact on patient care Benefits and potential harms need to be known

7. Declaration - Definitions An HIS is more than the hard- & software It includes also all stakeholders that are affected by the use of the system Evaluation is measuring properties of a HIS Informs a decision to be made Evaluation of HIS has to deal with actors, the artefacts and their interactions.

8. Observations Evaluation generates information Should be relevant for the decision at hand Evaluation supports reflective practice It is part of a professional attitude In line with engineering ethics Evaluation is challenging Each study requires careful design, guided by the decision to be taken. Detailed guidelines are hardly to be given Evaluation is not free It takes time, is costly and require planning from the onset of a project

9. Recommendations 1-3 Evaluation is an ethical imperative Part of code of ethics of IMIA (in context of EPD) Evaluation should be free of pressure There should be no pressure as to what the desired outcome of the study is Evaluation studies should be grounded on theory and rigorous approaches Only validated methods and techniques should be used. No reinvention of the wheel or use of non-validated tools

10. Recommendations 2-3 Reports on evaluation methods and methodical studies are encouraged Not only the study should be reported, but there should be a reflection on the approach taken and the lessons learned Creation of an evidence base Guidelines for good evaluation practice should be developed Guidelines for reporting of studies should be developed CONSORT, STARD

11. Recommendations 3-3 Evaluation should be promoted by centers of excellence Evaluation requires specific skills. Bringing those skills and expertise together will improve the quality of the studies done Evaluation networks should be established The field can only develop through (international) collaboration An open access repository of evaluation resources is needed Not only scientific reports, but also tools, questionnaires, experiences.

12. Recommendations Proposed additions Industries should include evaluation in their offers An accreditation/certification body is required

13. Current activities GEP-HI: Good evaluation practice in Health Informatics Pirkko Nykänen Tampere STARE-HI: Standards for reporting of evaluations in Health Informatics Jan Talmon Maastricht EP-HI: Evaluation Portal for Health Informatics Gerrit Boers Maastricht