1. Personalized Prediction and Resolution of Clinician Information Needs James J. Cimino, M.D. Departments of Biomedical Informatics and Medicine Columbia University

2. 1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 Covell et al. Information Needs

3. Studying Information Needs Covell DG, Uman GC, Manning PR. Information needs in office practice: are they being met? Ann Intern Med. 1985 Oct;103(4):596-9.

4. Findings of Observational Studies Information needs occur often They are often unresolved Computer-based resources are underused: –Lack of knowledge of existence –Lack of access –Lack of navigational skills –Perceived lack of time

5. Information Needs in Clinical Care ?

6. Clinical Information for Decision Support

7. Health Knowledge for Decision Support

9. Automated Clinical Decision Support Alerting and Reminder System

10. Clinical Decision Support Systems

11. Modes of Decision Support Patient Info Yes Maybe NoClinical Data Decision Support Mode Knowledge Yes No No Knowledge Alerting No Yes YesKnowledge ? Yes Yes Yes Clinical Data or Knowledge User Recognizes Need? Retrieval Uses Clinical Data? Retrieval Uses Knowledge? What is the Information Retrieved?

12. Infobuttons Anticipate Need and Provide Queries i

13. Information Needs of CIS Users Common tasks may have common needs System knows: –Who the user is –Who the patient is –What the user is doing –What information the user is looking at So: We may be able to predict the specific need User is sitting at a computer! So: We may be able to get an answer automatically

14. 1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 Covel et al. Information Needs UMLS Project

15. Unified Medical Language System The purpose of the UMLS is to improve the ability of computer programs to “understand” the biomedical meaning in user inquiries and to use this understanding to retrieve and integrate relevant machine-readable information for users. - Donald A.B. Lindberg 1986/1993

16. 1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 Covel et al. Information Needs UMLS Project First Version of UMLS ICD9  MeSH Medline Button

17. First Attempt: The Medline Button CIS (WebCIS’s predecessor) on mainframe BRS/Colleague (Medline) on same mainframe Get them to talk to each other Search using patient diagnoses and procedures

22. First Attempt: The Medline Button CIS (WebCIS’s predecessor) on mainframe BRS/Colleague (Medline) on same mainframe Get them to talk to each other Search using patient diagnoses and procedures Technical success Practical failure

23. 1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 Covel et al. Information Needs UMLS Project First Version of UMLS ICD9  MeSH Medline Button Mosaic Web-based Generic Queries PubMED WebCIS

26. 1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 G.O. Barnett DXplain Covel et al. Information Needs UMLS Project First Version of UMLS ICD9  MeSH Medline Button Mosaic Web-based Generic Queries PubMEDWeb DXplain DXplain Button WebCIS

30. 1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 G.O. Barnett DXplain Covel et al. Information Needs UMLS Project First Version of UMLS ICD9  MeSH Medline Button Mosaic Infobuttons Web-based Generic Queries PubMEDWeb DXplain DXplain Button WebCIS

36. “Just in Time” Education MRSA

37. “Just in Time” Education Understand Information Needs 1 MRSA

38. “Just in Time” Education Get Information From EMR Understand Information Needs 1 2 MRSA

39. “Just in Time” Education Get Information From EMR Resource Selection Understand Information Needs 1 2 3 MRSA

40. “Just in Time” Education Get Information From EMR Resource Selection Resource Terminology Understand Information Needs 1 24 3 MRSA

41. “Just in Time” Education Get Information From EMR Resource Selection Resource Terminology Understand Information Needs Automated Translation 1 254 3 MRSA

42. “Just in Time” Education Get Information From EMR Resource Selection Resource Terminology Querying Understand Information Needs Automated Translation 1 254 6 3 MRSA

43. “Just in Time” Education Get Information From EMR Resource Selection Resource Terminology Querying Presentation Understand Information Needs Automated Translation 1 254 6 3 7 MRSA

44. Research Issues What are the information needs?

45. Portable Usability Lab User’s Workstation Microphone Video Converter 75 foot cable Converter Controller Cassette Recorder VCR Headphones Video Monitor

47. Morae screen capture

48. Research Issues What are the information needs?

49. Information Resource Use According to Log Files 12% IBs

50. Research Issues Which context information is important? What are the information needs?

51. Context-Specific Resource Use

52. Context-Dependent Information Needs AgeSex TrainingRole DataTask Context ? ! Institution

53. Research Issues What are the information needs? Which context information is important What resources can satisfy needs? How can retrieval be automated? –What context data are used? –How are the data translated? –How are the data transmitted?

54. 1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 G.O. Barnett DXplain Covel et al. Information Needs UMLS Project First Version of UMLS ICD9  MeSH Medline Button Mosaic Infobuttons Web-based Generic Queries Infobutton Manager PubMEDWeb DXplain DXplain Button WebCIS

55. Infobuttons vs. Infobutton Manager Page of Hyperlinks Infobutton Clinical System Resource Infobutton Manager Context Query Knowledge Base s

64. 1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 G.O. Barnett DXplain Covel et al. Information Needs UMLS Project First Version of UMLS ICD9  MeSH Medline Button Mosaic Infobuttons Web-based Generic Queries Infobutton Manager Infobutton Manager Standard PubMEDWeb DXplain DXplain Button WebCIS

66. Infobutton Managers have been Deployed New York Presbyterian Hospital’s WebCIS New York Presbyterian Hospital’s Eclipsys LDS Hospital’s HELP system New York State Psychiatric Institute’s PSYCKES Regenstrief Medical Record System Partners Healthcare System’s Knowledgeliink Vanderbilt University’s PC-POETS

69. Evaluation of the Columbia University’s Infobutton Manager at NYPH Use: who, what, when, where, and why? How usable is it? How useful is it? What impact is it having?

70. Evaluation Modalities System log files Pop-up surveys On-line feedback E-mail surveys

71. Evaluation Log file analysis On-line survey Popup questionnaires E-mail survey Log file analysis: 32 months of data

72. Results: System Log Files March 2004-October 2006: –IM: 3,009 users, 29,541 times

73. Infobutton Manager (IM) Usage

74. Resource Use by Context Laboratory Results Review Inpatient Drug Order Review Microbiology Sensitivity Results Diagnosis List Inpatient Lab Order Entry Inpatient Drug Order Entry Health Resources Infobutton Manager

75. Results: System Log Files March 2004 - October 2006: –IM: 3,009 users, 29,541 times October 2006: – IM: 281 users, 1,022 times (33/day) – HR: 708 users, 5,744 times (185/day) –HR: 3,609 users, 155,718 times

76. Resource Use by Context - October

77. Finding a Resource or Question Users chose questions 48.7% of the time Diagnosis list: 10.5-24.3% Outpatient drug order review:23.0-63.3% Inpatient drug order review: 63.3-78.7% Users chose at least one resource from HR page 92.4% of the time (82.6-93.7%, depending on user type and context)

78. Evaluation Log file analysis On-line survey Popup questionnaires E-mail survey Log file analysis: 32 months of data On-line survey: 108 comments

79. Results: On-line Feedback 108 questions or topics submitted by users –54 (50%) in Laboratory Results Review context 30 were about test interpretation –22 (20%) in Inpatient Drug Order Review 15 were about side effects, dosage, cost, trade names, and interactions

80. Evaluation Log file analysis: 32 months of data On-line survey: 108 comments Popup questionnaires: E-mail survey Popup questionnaires: 195/~3,642 (5.4%)

81. Pop-up Survey Responses Q1: Easy to use Q2: Got answer Q3: Helpful

82. Evaluation Log file analysis: 32 months of data On-line survey: 108 comments Popup questionnaires: E-mail survey Popup questionnaires: 195/~3,642 (5.4%) E-mail survey: 73/1,228 (5.8%)

83. E-mail Survey Responses Q2: Infobuttons are easy to use Q3: Presence of question on the list Q4: Speed of answer Q5: Helpful answer Q6: Positive effect on patient care decisions 1 (strongly positive) to 5 (strongly negative)

84. Results: Pop-Up and E-Mail Surveys Pop-UpE-Mail Easy to use 83% 92% Question on list >50% of time 89% Answered question 69% Useful 77% Helpful >50% of time 90% Positive effect on care 74% Specific cases of improved care 14 (40% of negative responses due to Lab Manual) (Students were the least positive)

85. Summary of Impact Over 29,000 additional accesses to resources (19%) Positive impact at least half the time: 74% 14 respondents identified one or more specific situations in which patient care improved

86. Lessons Learned Use of the IM has been more gradual than desired User fails to select a question half the time –Is the IM failing to anticipate the information need? –Example - questions sugested during lab review: Housestaff did not select a question 57.8% of time Online survey: 4/11 dealt with reference ranges IM almost always provides a link to lab manual –Is the user unable to find questions on the IM page? Solutions: –Education –Add questions –User interface

88. Conclusions 1.Context-specific access to health knowledge resources has been successfully accomplished 2.Impact on patient care decisions has been positive 3.Need to improve user interface for navigation 4.Increased resource use should result in clinicians making better informed patient care decisions

89. Acknowledgments This work is supported by NLM grant R01LM07593 Evlauation plan: –Vimla Patel –Sue Bakken –Leanne Currie –Beth Friedman Programming: Jianhua Li Log files: Rick Gallagher