1. Anticipating and Satisfying Clinician Information Needs: The Infobutton Manager Project James J. Cimino, M.D. Department of (Bio)Medical Informatics Weekly Seminar March 6, 2003

2. Studying Clinician Information Covell DG, Uman GC, Manning PR. Information needs in office practice: are they being met? Ann Intern Med 1985 Oct;103(4):596-9 Other studies: Osheroff, Forsythe, Gorman, Ely… Types of needs –Patient-specific –General knowledge –Institutional Resources –Medical record –Colleagues –Journals –Books

3. Do Clinicians Satisfy their Needs? Needs are underestimated Information needs may not be recognized Resources are outdated or unavailable Computers not used Resolution may be deferred Unsatisfied information needs are a potential source for errors

4. Needs Related to CIS Use? Very small part of the clinician’s day Very important part of the day –New information stimulates needs –Decision-making moments Opportunity to resolve needs (using a computer) Context might predict needs Context might help resolve needs Needs in this setting have not been studied

5. Context-Dependent Information Needs AgeSex TrainingRole DataTask Context ? ! Institution

6. Hypothesis: Information needs are stereotypical Information needs can be predicted based on context Appropriate resource can be identified (automatically or in advance) Resource can be queried automatically Information from context can be used for query “Infobuttons”

7. A Brief History of Infobuttons 1991: “Medline Button” translated ICD9 to MeSH and performed Medline searches 1994: World Wide Web 1994: First CPMC Web-based CIS 1996: WebCIS 1996: Infobuttons link microbiology and medication data to PubMed and Micromedex 1997: Infobuttons for patients (PatCIS) 2002: Handheld, wireless infobuttons (PalmCIS)

8. Infobutton Manager Project Study information needs of CIS users Characterize needs based on context Develop way to anticipate needs Develop ways to resolve needs Deploy anticipation/solution Evaluate

9. Infobutton Manager Team Members Jim Cimino Jianhua Li Sue Bakken Vimla Patel Mark Graham Leanne Currie Mureen Allen Anne-Marie Ramierez Your name here?

10. Methods Determine information needs –WebCIS log files –WebCIS user questionnaires –Observational studies Characterize information needs Build Infobutton Manager (IM) Build infobuttons Populate IM database Integrate IM with WebCIS Evaluation

11. Insert WebCIS Demo Here

12. WebCIS Log File Methods Health Resources page in WebCIS Infobuttons in WebCIS Problems: –Which user selected which resource? –What was the user was doing? Solution: look at previous activity at same IP address

13. WebCIS Log File Records Mar 2 16:16:46 webcis3-i syslog: |WebCIS|ciminoj|156.111.145.61|3131313 |lab^2003-03-02-05.19.00.819650|view Mar 2 16:18:55 webcis3-i syslog: |WebCIS|ciminoj|156.111.145.61|3131313 |healthresource|view Mar 2 16:18:58 webcis3-i syslog: |Resources|(nobody)|156.111.145.61|| |url^http://salk.cpmc.columbia.edu:70/mdxdocs

14. WebCIS Log Files Results Six months of log files 2,607 users (1,034 per month) used a resource 2931 IPs (2008 “156.x.x.x”) User types: Housestaff:51% Attendings:34% Nurses: 5% Other:10%

15. WebCIS Log Files Results 38,763 uses of resources 14,036 outside WebCIS (2339/month) 24,727 within WebCIS (4,104/month; 4/user) 19,913 uses of 24 resources (3,319/month) 4,814 uses of 3 infobuttons (802/month)

16. WebCIS Log Files Results Popular resources:InsideOutside Micromedex:56.5%51.7% Ovid: 7.4% 8.2% NYP Formulary 7.2% 7.1% Harrison’s 5.1% 5.1% PubMed 5.1% 5.1% Health Journals 5.0% 8.2% Medline Plus 3.3% 3.6% ICD9-CM 3.3% 2.9% Infobuttons Pharmacy/Micromedex92.9% - Microbiology/PubMed 6.1% - Pharmacy/PubMed 1.0% -

17. 80.4% Health Resources: 19,913 times in 105 contexts: Lab results:52.1% Radiology results: 9.0% Clinical notes: 7.4% Pathology results: 2.8% Visit list: 2.1% Pharmacy: 2.0% Discharge summary: 1.9% WebCIS Log Files Results 100% Infobuttons: 4,814 times in 2 contexts: Pharmacy:93.9% Lab results (Micro): 6.1%

18. WebCIS Log Files Results Context: LabRadNoteCardPath Vis Phar DSum MDX 53% 9% 3% 3% 3%2% 2% 2% Ovid 49% 9% 6% 2% 3%1% 1%1% Formul. 52% 8% 6% 3% 3%1% 3%2% Harrison 49%10% 5% 3% 4%1% 1%3% PubMed 48%10% 7% 3% 5%1% 3%1% Journals 43% 8% 7% 6% 5%2% 2%1% Med+ 50% 8% 6% 3% 2%4% 2%1% ICD9 37%12%10% 2% 3%2% 1%2% Infobuttons MDX - - - - - - 100% - PubMed 86% - - - - - 14% -

19. WebCIS Log Files Results Context: LabRadNoteCardPath Vis Phar DSum MDX 56%57%61% 59% 49%42% 9% 62% Ovid 7% 7% 8% 5% 8% 4% 1% 6% Formulary 7% 6% 6% 6% 7% 3% 1% 7% Harrisons 5% 6% 4% 5% 6% 3% 1% 7% PubMed 5% 5% 5% 4% 8% 2% 1% 3% Journals 4% 4% 5% 9% 8% 3% 1% 3% Med+ 3% 3% 2% 3% 3%32% 1% 2% ICD9 1% 4% 4% 2% 3% 2%.2% 3% Infobuttons MDX - - - - - -84% - PubMed 3% - - - - - 2% -

20. WebCIS Log Files Results User: Housestaff Attending NurseOther MDX 62%61% 42% 27% Ovid 6% 7% >1% 1% Formulary 6% 5% 3% 5% Harrison 4% 3% 4% 3% PubMed 4% 5% 1% 1% Journals 3% 0% 3% 1% Med+ 2% 2% 3% 2% ICD9 3% 5% 1% 1% Statref 1% 1% 1% 3% PIER 0% 2% 3% 1% Infobuttons MDX 5% 6% 42% 53% PubMed 2% 1% 2% 1%

21. WebCIS Usage Logs Results Take home messages: –Micromedex is a popular resource –Ovid edges out PubMed –Lab tests are a popular context for resources –No correlation between context and resource –Nurses and pharmacists like the Micromeex infobutton for microbiology Still needed: –Context/resource/user type interactions –Concept-oriented contexts Questions: –How much is based on knowledge of resources? –What are the questions in each context?

22. WebCIS User Questionnaires Health resources page outside vs. inside WebCIS Resources used, how often, how useful, anecdotes Infobuttons used, how often, how useful, anecdotes Other resources used

23. WebCIS User Questionnaire Results 1055 users in January 300 randomly selected 250 with valid e-mail addresses surveyed 42 responded 21 attendings 19 housestaff 1 nurse 1 pharmacist

24. WebCIS User Questionnaire Results How often do you access: Never <Month Monthly Weekly Daily Attend 2 1 3 5 9 House 3 11 5 Nurse 1 Pharm 1 Attend 11 8 2 House 14 3 1 1 Nurse 1 Pharm 1 ResourceInfobutResourceInfobut

25. WebCIS User Questionnaire Results Why do you access: Patient Qs Genl Know Other Attend 20 12 3 House 18 10 1 Nurse 1 Pharm 1 1 Attend 10 House 4 Nurse Pharm 1 ResourceInfobutResourceInfobut

26. WebCIS User Questionnaire Results How do you access: None WC-Login WC-Menu Both Other Attend 2 4 7 8 12 House 0 8 8 3 17 Nurse 1 Pharm 1 1 ResourceResource

27. WebCIS User Questionnaire Results What resources and how useful: Very Somewhat Not _ A H A H A H MMDX 17 17 0 2 1 0 Ovid 10 11 1 2 0 0 PubMed 12 7 2 3 0 0 Medline+ 4 4 0 1 1 0 FullText 8 7 2 2 0 0 Harrisons 7 4 3 9 1 0 PIER 2 4 3 1 1 0 Formulary 2 3 2 1 1 0 Health J 6 9 0 1 0 1 Science J 3 4 0 1 1 0 STATref 1 2 6 0 1 1 ICD9 1 2 5 0 0 1

28. WebCIS User Questionnaire Results How useful are infobuttons: Very Somewhat Not _ A H A H A H Micro Org 3 3 5 1 2 0 Micro Sens 7 3 2 3 0 0 Pharmacy 6 1 0 2 0 0

29. WebCIS User Questionnaire Results A A A A A A A A A A A A A A A A A A A PN H H H H H H H H HH H H H H H H H H H A Never <MonthlyMonthlyWeeklyDaily Health Resources 100 80 60 40 20 0 How does self-reporting correlate with log files?

30. WebCIS User Questionnaire Results HH H HH H H H H H H H H H H H H H AA A A AA A A A A A A A A A A A A A A P N A Never <MonthlyMonthlyWeeklyDaily Infobuttons 25 20 15 10 5 0 How does self-reporting correlate with log files?

31. WebCIS User Questionnaire Results Anecdotes: –Avoid errors (drug interactions) –Identify correct doses –Identify pills –Prepare publications –Teaching –Understand abbreviations (sensitivity tests) Other desired resources: –Up To Date –NEJM –MDConsult –Books

32. WebCIS User Questionnaire Results Resources are used and liked Resources are used within WebCIS Resources are used to answer patient-specific questions Use of resources may be overestimated (can’t tell for sure) Infobuttons are used and liked Additional resources requested Questions: –How knowledgeable are users about resources? –What are the context-specific questions

33. Observational Studies Connect portable usability lab to clinical workstation Ask users to think aloud about information needs Videotape screen and audiotape users Transcribe tapes Code transcripts, using tapes Analysis of coding

34. Portable Usability Lab User’s Workstation Microphone Video Converter 75 foot cable Converter Controller Cassette Recorder VCR Headphones Video Monitor

36. Observational Results Sites: –Medical /Surgical –CCU –Ambulatory Care Number of Days: –7 different days –Varying days of the week –Varying times of day Total hours of tape: 15 hrs 29 minutes Total Number of events: 154

37. Observational Results Total Number of events: 154 Medical/Surgical71 (46%) CCU35 (23%) Ambulatory Care48 (31%)

38. Observational Results Total Number by User Type Nurse17 (11%) House Officer103 (67%) Attending/Fellow18 (12%) Student7 (5%)

39. Event Coding Types of Events: –Foreground/Background –Implicit/Explicit –Patient/Institution/Administrative/Domain Types of Resources –Human/Computer/Paper Types of Outcomes –Success/Deferred/Failed

40. Coding Event Type Foreground –Specific information about patient management –Problem, Intervention, Comparison, Outcome Background –Root question: who, what, where,when, why, how? –Related to health disorder or aspect of a disorder

41. Coding Event Type

42. Explicit: Subject expresses an information need Implicit: Subject uses a resource other than WebCIS

43. Coding Event Type

44. Patient –A patient related need Institution –An information need that is related to the infrastructure of the institution Administrative –An institutional information need that is related to a specific patient Domain –A domain related information need that is related to a specific patient

45. Coding Event Type

46. Coding Resources Used Human –The subject sought information from a human interaction in person or via telephone Computer –The subject sought information from a computer interaction either webCIS or other internet resource Paper –The subject sought information from a paper resource – patient chart or text book

47. Coding Event Type

48. Coding Outcomes Success –The subject sought and found the desired information Deferred –The subject did not seek an answer Failed –The subject sought, but did not find desired information

49. Coding Outcomes

50. Coding Events Need to correlate across event types Especially need to look at types of needs that were deferred or that failed Good news: –Information needs exist and can be identified –Needed resources are computer-based Bad news (good news for us): –Many needs failed or deferred –Solutions will require integration of patient data with information resources Questions: –What are the triggers? –What resources can help?

51. Classification of Information Needs Classification Definition Navigation A question about anything in WebCIS (either currently present or could be added at a later date to WebCIS) that the user is unable to access or use. The IB link would take the user to a resource that tells him how to access the needed information. Functionality A response or question which alludes to the need to add an item to WebCIS or give the user access to another resource which answers the need. The IB would access the resource that would solve the problem and present the solution to the user. Laboratory Information needs that are institution or patient specific related to the laboratory. The IB would provide specific information about a laboratory finding. Pharmacy Information needs that are institution or patient specific related to the pharmacy. The IB would provide information about specific medications. Differential Diagnosis Questions related to alternate causes of a symptom or laboratory or radiology finding. IB would provide context-specific differential diagnoses Definition/ Information Questions related to the meaning of words. In addition, general information on a topic. IB would provide a definition or general information about a topic/word. Miscellany Those needs that do not readily translate into patient/institution needs.

52. Methods: Classification Scheme General question: What is the normal value for this calcium result? Generic question: What is the normal value of this laboratory test? Classification: Navigation: because normal laboratory values are available in WebCIS, this IB link would take the user to another page informing the users on how to access the information needed.

53. Methods: Classification Scheme General question: Since the CT scan of the liver is abnormal what are his liver function test results? Generic question: What laboratory tests are associated with this radiology report? Classification: Functionality group since it will require designing an IB that would bring information to the user from another area in WebCIS (or some other resource) while he is at a particular site.

54. Classification Code 48 31.2 27 17.5 19 12.3 31 20.1 9 5.8 15 9.7 5 3.2 154 100.0 Navigation Functionality Laboratory Pharmacy Differential Diag. Definition/info. Miscellany Total Count Percent Observational Studies Results

55. Wide variety of information needs Many needs are stereotypical Many can be answered with simple resources Some complex solutions will be needed Next steps: –Finish the characterizations of needs –Propose ways to solve each need –Build infobuttons (project opportunities!)

56. The Infobutton Manager CGI script using “get” or “post” Context parameters as input Uses context table and infobutton table Returns a list of questions that are links Log files for calls to IM and selecting links Infobutton Manager Manager for maintenance

57. Matching Context to Need ContextContext Context Matching Question Selection Instantiation InfobuttonsInfobuttons Infobutton Manager Context Table Question Table

58. Context Parameters Patient: age, gender User: physician, nurse, student, patient, … Setting: lab reports, medications, … Concept of interest: lab test, drug, …

59. Infobutton Parameters Question type: fixed, fill-in-the-blank Source type: simple query, additional data URL: http://www.guideline.gov/SEARCH/search_results.asp? sSearch_string=<>&bSort_by_date=&query_id=& bDebug=&results=10&view=search_results Question: What are guidelines for evaluation of <>?

60. Infobutton Manager Manager Demo Insert Infobutton Manager Demo Here Insert Infobutton Manager Manager Demo Here

61. Evaluation Plan Baseline observation –Creation of Infobutton Manager tables –Resources used –Unmet information needs Log of access to online resources Introduction of infobuttons –Usability studies Log of access to online resources (infobuttons) Post-intervention observation –Use of infobuttons and other resources –Unmet information needs

62. Discussion Information needs occur and are detectable Information needs are unmet Information needs are predictable based on context Solutions are at hand Solutions are institution independent Integration into clinical systems is easy Will infobuttons reduce unmet information needs? Next project: show that this reduces errors!

63. Acknowledgments National Library of Medicine grant R01-LM07593 Rick Gallagher The team: Jianhua Li, Sue Bakken, Vimla Patel, Mark Graham, Leanne Currie, Mureen Allen, Anne-Marie Ramierez

66. function get_info(mc){ var infowin = window.open("","INFO"); infowin.focus(); document.INFO.info_med.value=mc; document.INFO.submit(); } <form method="POST" name=“INFO” action="wc_infomanage.cgi">

70. Context Parameters Patient: age, gender User: physician, nurse, student, patient, … Setting: lab reports, medications, … Concept of interest: lab test, drug, … Institution: CPMC, RMRS, LDS, …