1. H.Laerum, Kvalis Evaluation of EMR : Modeling the way physicians work Hallvard Lærum, M.D. Faculty of medicine

2. H.Laerum, Kvalis What is an EMR? ”A patient record, in which all information is stored electronically in such way that it may be found and reused by means of computer tools” Heimly V, Ree AO, and Yang JJ: "Den elektroniske pasientjournalen - Status og aktuelle områder for felles tiltak"; 1998; 9/97; KITH Universal Medical Language System (UMLS): 626 893 concepts 1 358 891 terms 10 000 m of this

3. H.Laerum, Kvalis Why is evaluation difficult? Complex computer systems supporting complex work patterns in complex organizationsComplex computer systems supporting complex work patterns in complex organizations Bias - all sortsBias - all sorts The experts don’t do what they say, and they don’t say what they doThe experts don’t do what they say, and they don’t say what they do Error ?

4. H.Laerum, Kvalis Observation vs. other evaluation methods Questionnaries lieQuestionnaries lie Hard end points are too hardHard end points are too hard An expert can’t completely tell you how he or she “does it”An expert can’t completely tell you how he or she “does it” Brender J. Methodology for constructive assessment of IT-based systems in an organisational context. Int.J.Med.Inf. 1999; 56 :67-86. Heathfield HA, Pitty D, Hanka R. Evaluating information technology in health care: barriers and challenges. BMJ 1998; 316 :1959-61. Friedman CP, Wyatt J. Evaluation methods in medical informatics. New York: Springer, 1997, Chap. 1, ”Challenges of Evaluation”; 6-7.

5. H.Laerum, Kvalis The data Short pilot: 5 physicians, 27 patients, two wards, 7 h observation timeShort pilot: 5 physicians, 27 patients, two wards, 7 h observation time Field notes: 2500 lines of textField notes: 2500 lines of text

6. H.Laerum, Kvalis The model

7. H.Laerum, Kvalis Applications for the model Self-reporting questionnaries ? Time studies Summaries: Tasks: Observation time

8. H.Laerum, Kvalis Applications for the model II Information source studies Computer use studies

9. H.Laerum, Kvalis http://kvalis.ntnu.no