1. Distributed Knowledge-Based Abstraction, Visualization, and Exploration of Time-Oriented Clinical Data Yuval Shahar, M.D., Ph.D. Medical Informatics Research Center Ben Gurion University, Beer Sheva, Israel & Stanford University, CA, USA

2. The Need for Intelligent Integration of Multiple Time-Oriented Clinical Data Many medical tasks, especially those involving chronic patients, require extraction of clinically meaningful concepts from multiple sources of raw, longitudinal, time-oriented data –Example: “Modify the standard dose of the drug, if during treatment, the patient experiences a second episode of moderate anemia that has persisted for at least two weeks” Examples of clinical tasks: –Diagnosis Searching for “a gradual increase of fasting blood-glucose level” –Therapy Following a treatment plan based on a clinical guideline –Quality assessment Comparing observed treatments with those recommended by a guideline –Research Detection of hidden dependencies over time between clinical parameters

3. The Need for Intelligent Mediation: The Gap Between Raw Clinical Data and Clinically Meaningful Concepts Clinical databases store raw, time-stamped data Care providers and decision-support applications reason about patients in terms of abstract, clinically meaningful concepts, typically over significant time periods A system that automatically answers queries or detects patterns regarding either raw clinical data or concepts derivable from them over time, is crucial for effectively supporting multiple clinical tasks

4. The Temporal-Abstraction Task Input: time-stamped clinical data and relevant events (interventions) Output: interval-based abstractions Identifies past and present trends and states Supports decisions based on temporal patterns, such as: “modify therapy if the patient has a second episode of Grade II bone-marrow toxicity lasting more than 3 weeks” Focuses on interpretation, rather than on forecasting

5. Temporal Abstraction: The Bone-Marrow Transplantation Domain. 0 400 20010050 ²² 1000 2000 ² () ² ² ² 100K 150K () ² ² ² ² ² ² ² ² ² Granu- locyte counts ² ² ² ² Time (days) Platelet counts PAZ protocol M[0]M[1]M[2]M[3]M[1]M[0] BMT Expected CGVHD M[i]= Myelotoxicity (bone-marrow toxicity) of severity Grade i

6. The Bone-Marrow Transplantation Example, Revisited

7. Uses of Temporal Abstractions Therapy planning and monitoring (e.g., to support guideline-based care) Creating high-level summaries of time-oriented medical records Supporting explanation modules for a medical DSS Representing goals of therapy guidelines for quality assurance at runtime and quality assessment retrospectively; guideline intentions regarding both the (care provider) process and (patient) outcomes can be captured as temporal patterns to be achieved or avoided Visualization and exploration of time-oriented clinical data: the KNAVE project

8. The Temporal-Abstraction Ontology Events (interventions) (e.g., insulin therapy) - part-of, is-a relations Parameters (measured raw data and derived concepts) (e.g., hemoglobin values; anemia levels) - abstracted-into, is-a relations Patterns (e.g., crescendo angina; chronic GVHD) - component-of, is-a relations Abstraction goals (user views)(e.g., therapy of diabetes) - is-a relations Interpretation contexts (effect of regular insulin) - subcontext, is-a relations Interpretation contexts are induced by all other entities

9. Temporal-Abstraction Output Types State abstractions (LOW, HIGH) Gradient abstractions (INC, DEC) Rate Abstractions (SLOW, FAST) Pattern Abstractions (CRESCENDO) - Linear patterns - Periodic patterns

10. Temporal-Abstraction Knowledge Types Structural (e.g., part-of, is-a relations) - mainly declarative/relational Classification (e.g., value ranges; patterns) - mainly functional Temporal-semantic (e.g., “concatenable” property) - mainly logical Temporal-dynamic (e.g., interpolation functions) - mainly probabilistic

11. Dynamic Induction of Contexts ss ee sees

12. Induction of Interpretation Contexts

13. The Benefits of Interpretation Contexts Context intervals serve as a frame of reference for interpretation of clinical data; abstractions are meaningful only in a particular clinical context Context intervals focus and limit the computations to only those relevant to a particular context Contexts enable the use of context-specific medical knowledge contexts support maintenance of several concurrent views (or interpretations) of the data Facilitate maintenance of clinical knowledge –The same context-forming entity (e.g., a hepatitis episode) can induce several clinical contexts – The same context (e.g., “a chemotherapy effect”) might be induced by several entities (e.g., multiple medication types)

14. Local and Global Persistence Functions 1 0 I 1 I 2 tt  1  2  th Time Bel(  )

15. Temperature Hemoglobin Level Linear Component Week 2 Week 3Week 1 Anemia Fever Anemia Fever Anemia Fever Linear Component Periodic Pattern Abstraction of Periodic Patterns

16. The RÉSUMÉ System Architecture. Temporal-abstraction mechanisms Temporal fact base Events Contexts Abstracted intervals Primitive data Domain knowledge base Event ontology Parameter ontology Primitive data Events Context ontology External patient database ++ + +

17. Test Domains for the RÉSUMÉ System Medical domains: –Guideline-based care AIDS therapy Oncology –Monitoring of children’s growth –Therapy of insulin-dependent diabetes patients Non-medical domains: –Evaluation of traffic-controllers actions –summarization of meteorological data

18. Acquisition of Temporal-Abstraction Knowledge

19. Editing The Ontology Using Protégé Tools

20. Knowledge-Based Visualization and Exploration of Time-Oriented Data: The KNAVE-I and KNAVE-II Projects (Shahar and Cheng, 1999, 2000; Shahar et al., 2003) KNAVE = Knowledge-Based Navigation of Abstractions for Visualization and Explanation Interactive queries regarding both raw data and multiple levels of time-oriented abstractions derivable from these data Visualization and manipulation of query results Dynamic exploration of the results using the domain’s temporal- abstraction ontology The semantics of all operators do not depend on any specific domain, but the interface uses each domain’s ontology to compute and display specific terms and explore their relations KNAVE accesses the data through the IDAN temporal-abstraction mediator, which uses the ALMA system, a constraint-based re- implementation of RESUME, for temporal-reasoning

21. The IDAN Temporal-Abstraction Mediator (Boaz and Shahar, 2003) Temporal- Abstraction Controller Knowledge- acquisition tool Standard Medical Vocabularies Service KNAVE-II Knowledge Service Temporal - Abstraction Service (ALMA) Data Access Service Medical Expert Clinical User

22. The KNAVE-II Browsing and Exploration Interface [Shahar et al., AIM 2006] Overall pattern Raw clinical data Intermediate abstractions Medical knowledge browser Concept search

23. Moving Data Panels Around

24. Global Temporal-Granule Zoom (I)

25. Global Temporal-Granule Zoom (II)

26. Global Calendar-Based Zoom

27. Global Content-Based Zoom (I)

28. Global Content-Based Zoom (II)

29. Local Time-Sensitive Zoom

30. Exploration Operators Motion across semantic links in the domain’s knowledge base by using the semantic explorer; in particular, relations such as: - part-of - is-a - abstracted-from - subcontext Motion across abstraction types: state, gradient, rate, pattern Application of aggregation operators such as mean and distribution Dynamic change of temporal-granularity (e.g., days, months) Explanation by context-sensitive display of relevant knowledge “What- if” queries allow hypothetical assertion or retraction of data and examination of resultant patterns

31. Semantic Exploration of Temporal Abstractions

32. Explanation: A Classification Function

33. Explanation: A Persistence Function

34. Functionality and Usability Evaluation of KNAVE-II (Palo Alto Veterans Administration Health Care System )  Eight clinicians with varying medical/computer use backgrounds A second study used six additional clinicians and more difficult queries  Each user was given a 15 minute demonstration of the interface and two warm-up queries to answer  The evaluation used an online database of more than 1000 bone-marrow transplantation patients followed for 2 to 4 years  Each user was asked to answer 10 queries common in oncology protocols, about individual patients, at increasing difficulty levels  A cross-over study design compared the KNAVE-II module versus two existing methods (in the 2 nd study, control-group users chose which one): Paper charts An electronic spreadsheet (ESS)  Measures: Quantitative: time to answer and accuracy of responses Qualitative: the Standard Usability Score (SUS) and comparative ranking

35. The KNAVE-II Evaluation Results (Martins et al., MEDINFO 2004; AIM 2008) Direct Ranking comparison: KNAVE-II ranked first in preference by all users Detailed Usability Scores: The Standard Usability Scale (SUS) mean scores: KNAVE-II 69, ESS 48, Paper 46 (P=0.006) (more than 50 is user friendly) Time to answer: –In the first study, users were significantly faster using KNAVE-II as the level of difficulty increased, up to a mean of 93 seconds difference versus paper, and 27 seconds versus Excel, for the hardest query (p = 0.0006) –The second evaluation, using more difficult queries and more advanced features of KNAVE-II, emphasized the differences even further: The comparison with Excel showed a similar trend for moderately difficult queries (P=0.007) and for hard queries (p=0.002); on the average, study participants answered each of the two hardest queries 277 seconds faster using KNAVE than when using the ESS Correctness: –In the first study, using KNAVE-II significantly enhanced correctness versus using paper, especially as level of difficulty increased (P=0.01) –In the second study, 91.6% (110/120) of all of the questions asked within queries of all levels produced correct answers using KNAVE-II, versus only 57.5% (69/120) using the ESS (p<0.0001); the correctness scores for KNAVE-II versus the ESS were significantly higher for all queries

36. The VISITORS System: Interactive Exploration of Multiple Patients Supports, among other types, aggregation queries: Who [Patients] Query –Find all male patients who were older than 50 and were treated by Dr. Johnson; and, starting within the two weeks following a BMT procedure, had a one week episode of WBC_State whose value was lower than “Normal” –Show all patients whose HGB_State can be interpreted as “moderate anemia” during more than 5% of February 2006 When [Time intervals] Query –Select time intervals during which more than 10% of the specific patients have HGB state value greater than “normal”

37. Demographic Constraints: –Young (age≤20) or Old (age≥70) Male patients The VISITORS Query-Builder Interface

38. Knowledge-based constraints –Hemoglobin [HGB] State was abstracted as Normal or higher for at least seven days after the two weeks period starting from the allogenic bone marrow transplantation, –WBC counts were abstracted as Gradient = Increasing during the same period

39. The VISITORS Main Display Interface Multiple-subjects raw data Medical knowledge browser Subject groups Distribution of derived patterns over time Concept search

40. The VISITORS System: Zooming into a Panel

41. Modifying Display Parameters for a Raw-Data Concept

42. Displaying the Distribution of an Abstract Concept over Time

43. Interactive Temporal Data Mining: Temporal Association Charts [ Klimov & Shahar, IDAMP 2007] Abstractions for the same subject group are connected; support and confidence indicated by width and hue The data of each subject are connected by a line

44. Temporal Association Charts: Support and Confidence Links Three data mining parameters displayed for each temporal association link: support = 55.60% of patient group confidence = 61.00% probability actual number of patients = 25

45. Temporal Association Charts: Direct Manipulation Using a Time-Value Lens Current WBC minimal value 1.4 10 3 cells/ml 91% of patients have the “moderately_low” value of the HGB_STATE_BMT

46. New WBC minimal value 5.43 10 3 cells/ml Now only 44.4% of patients have the “moderately_low” value of the HGB_STATE_BMT Temporal Association Charts: Direct Manipulation Using a Time-Value Lens (Cont)

47. Temporal Association Charts: Using a Relative Time Line Time line is relative to the BMT_Al procedure

48. Temporal Association Charts: Using a Relative Time Line (Cont) During the first month after BMT_Al During the second month after the BMT_Al procedure

49. Temporal Association Charts: Using a Relative Time Line (Cont) 1st month 2nd month

50. Adding a New Clinical Database to The IDAN Temporal Mediator Architecture Due to local variations in terminology and data structure, linking to a new clinical database requires creation of –A term-mapping table –A unit-mapping table –A schema-mapping table The mapping tools use a vocabulary-server search engine that organizes and searches within several standard controlled medical vocabularies (ICD-9-CM, LOINC, CPT, SNOMED, NDF) Clinical databases are mapped into the standard terms and structure that are used by the clinical knowledge base, thus making the knowledge base(s) highly generic

51. The LOINC Server Search Engine

52. LOINC Search Results

53. Summary: Abstraction,Visualization, & Exploration of Time-Oriented Clinical Data Automated, intelligent interpretation of longitudinal clinical data is useful for monitoring, therapy planning, data summarization and visualization, explanation, and quality assessment Interactive query, visualization, and exploration requires runtime access to the medical domain’s ontology The visualization and exploration semantics are specific to the intelligent-exploration task, but not to any particular medical domain