2. Temporal Reasoning and Planning in Medicine Visualization and Exploration of Time-Oriented Medical Data Yuval Shahar, M.D., Ph.D.

3. The Need for Visualization of Information To be effective, care providers and other decision makers need to be able to visualize both clinical data and their multiple levels of abstraction Larkin and Simon [1987]: the benefit of visual representations is mainly due to – reduction of logical computation through the use of direct perceptual inference – reduction of necessary search for information through the use of efficient graphical representations.

4. The Granularity Issue Not all events are measured the same way Example: Birthday is accurate to the day, but age is accurate to the year The granularity may even change over time Example: Age is measured first in months, then in years

5. Standard Time Quanta Year Month Week Day Hour Minute Second Smaller... Note, not all quanta can easily be expressed in terms of other quanta. For example, how many weeks are there in a year? How many minutes are there in a month? How do you deal with mixed granularities?

6. The Granularity Denial Approach Pretend there is no granularity problem Arbitrarily choose some time quantum and use that for all measurements Works best when the most logical quantum is large, say 1 day (or larger) If all you have is a day, how do you choose the right second?

7. Complications of Ignoring Granularity Suppose we know some event occurred on March 15th, 1999 Assume that the chosen granularity is seconds Do we record 3/15/1999, 12:0:0? Or, 3/15/1999, 0:0:0? Or, something else?

8. Granularity: an Object-Oriented Approach Each event class is assigned a relevant granularity Some classes have multiple valid granularities A given timeline can specify only a small number of granularity levels

9. Object-Oriented Example

10. The Interval-Uncertainty Approach Measure each event with the most relevant granularity When viewing data at a finer granularity, introduce uncertainty Uncertainty can include up to 6 degrees of freedom (start, duration, stop)

11. Uncertainty Examples Nov. 3, 1999 can be represented as: 11/3/1999, 0:0:0 — 11/3/1999, 23:59:59 Feb. 6, 1986 at 13:37 can be truncated: 2/6/1986 Or rounded: 2/7/1986

12. More Uncertainty Examples Start Maximum Duration Minimum Duration StartStop Not all possible values need be stored:

13. Timelines A timeline is a tuple, where E is a finite set of events containing at least the special null event, and M is a measure function M:E  R +. The measure function M assigns a temporal offset to each event in E. Cousins, S.B., and Kahn, M.G. The visual display of temporal information. Artificial Intelligence in Medicine 3(6) (1991) 341–357

14. Timeline Operators New: creates a new timeline containing only the null event. Add: adds an event e to an existing timeline, may increase length of timeline as a side- effect. Slice: remove events from one or both ends of a timeline, moves the null event as needed.

15. Timeline Operators (cont.) Filter: remove all events not satisfying some predicate P; the null event cannot be removed Overlay: merges two timelines. If common events do not coincide, they are copied

16. Timeline Operator Examples InputOperationOutput Slice(e1, e2) e1 e2 Filter(“b-ness”) a1a2b1 b2 Overlay (a, a) abc abc abc bc abcc’ Overlay (b, b)

17. Grounded Timelines A grounded timeline is one that can be directly mapped to a calendar (e.g., a Julian calendar) Slice and Filter are ‘safe’. Overlay and New may cause a timeline to become ungrounded

18. Time Line Browser Prototype used to display diabetes patient data over time Created the formal definition of, and operators for, a timeline Provides GUI for manipulating timeline operators.

19. Sample Queries For the previous week, display the patient’s logbook and personal calendar Summarize the patient’s blood sugar at breakfast, lunch, dinner and nighttime over the past month

20. Logbook and Calendar MondayTuesdayWednesdayThursdayFridaySaturdaySundayMonday Mild IllnessHospitalized X-ray WorkVacation in Florida 121314151617 Work

21. Blood Sugar Summary BLDNBLDN Note that multiple slices have been overlaid to produce this result For Breakfast, Lunch, Dinner, Night:

22. Knowledge-Based Visualization and Exploration of Time-Oriented Medical Data: Desiderata Interactive composition of (temporal-abstraction) queries Visualization of query results Exploration of multiple levels of temporal abstractions The semantics of the query, visualization and exploration operators should be domain independent, but should use the terms and relations specific to each (e.g., medical) domain

23. Knowledge-Based Visualization and Exploration of Time-Oriented Data: The KNAVE-I and KNAVE-II Projects (Shahar and Cheng, 1999, 2000; Shahar et al., 2003, in press) KNAVE = Knowledge-Based Navigation of Abstractions for Visualization and Explanation Interactive queries regarding both raw data and multiple levels of time-oriented abstractions derivable from these data Visualization and manipulation of query results Dynamic exploration of the results using the domain’s temporal- abstraction ontology The semantics of all operators do not depend on any specific domain, but the interface uses each domain’s ontology to compute and display specific terms and explore their relations KNAVE accesses the data through a temporal-abstraction mediator, such as IDAN

24. The KNAVE-II Browsing and Exploration Interface [Shahar et al., AIM 2006] Overall pattern Raw clinical data Intermediate abstractions Medical knowledge browser Concept search

25. Moving Data Panels Around

26. Global Temporal-Granule Zoom (I)

27. Global Temporal-Granule Zoom (II)

28. Global Calendar-Based Zoom

29. Global Content-Based Zoom (I)

30. Global Content-Based Zoom (II)

31. Local Time-Sensitive Zoom

32. Exploration Operators Motion across semantic links in the domain’s knowledge base by using the semantic explorer; in particular, relations such as: - part-of - is-a - abstracted-from - subcontext Motion across abstraction types: state, gradient, rate, pattern Application of aggregation operators such as mean and distribution Dynamic change of temporal-granularity (e.g., days, months) Explanation by context-sensitive display of relevant knowledge “What- if” queries allow hypothetical assertion or retraction of data and examination of resultant patterns

33. Semantic Exploration of Temporal Abstractions

34. Explanation: A Classification Function

35. Explanation: A Persistence Function

36. Evaluation of the KNAVE-II Intelligent Visualization and Exploration Module [Martins et al., AIM 2008]  Site: Palo Alto Veterans Administration Health Care System  Eight clinicians with varying medical/computer use backgrounds –A second study used 6 additional clinicians and more difficult queries  Each user was given a brief demonstration of the interface  The evaluation used an online database of more than 1000 bone-marrow transplantation patients followed for 2 to 4 years  Each user was asked to answer 10 queries common in oncology protocols, about individual patients, at increasing difficulty levels  A cross-over study design compared the KNAVE-II module versus two existing methods (in the 2 nd study, users chose which): –Paper charts –An electronic spreadsheet (ESS)  Measures: –Quantitative: time to answer and accuracy of responses –Qualitative: the Standard Usability Score (SUS) and comparative ranking

37. The KNAVE-II Evaluation Results (Martins et al. AIM 2008) Direct Ranking comparison: KNAVE-II ranked first in preference by all users Detailed Usability Scores: The Standard Usability Scale (SUS) mean scores: KNAVE-II 69, ESS 48, Paper 46 (P=0.006) (more than 50 is user friendly) Time to answer: –Users were significantly faster using KNAVE-II as the level of difficulty increased, up to a mean of 93 seconds difference versus paper, and 27 seconds versus the ESS, for the hardest query (p = 0.0006) –The second evaluation, using more difficult queries and more advanced features of KNAVE-II, emphasized the differences even further: The comparison with the ESS showed a similar trend for moderately difficult queries (P=0.007) and for hard queries (p=0.002); on the average, study participants answered each of the two hardest queries 277 seconds faster using KNAVE than the ESS Correctness: –Using KNAVE-II significantly enhanced correctness versus using paper, especially as level of difficulty increased, even in the initial study (P=0.01) (99% accuracy with K-II versus only 78% paper accuracy, 1 st study; 92% with K-II vs. 57% for ESS, 2 nd study) –The correctness scores for KNAVE-II versus ESS in the second study, which used more difficult queries, are significantly higher for all queries (p<0.0001)

38. VISualizatIon and exploration of Time-Oriented raw data and abstracted concepts for multiple subject RecordS –Graphical queries enable end users to define the constraints for selecting the relevant population to further explore –Knowledge-based interpretation of the data –Visual display and interactive exploration of multiple records –Aggregation of multiple records and creation of associations amongst subject-related [temporal] patterns The VISITORS System (Klimov and Shahar, AMIA 2005; Klimov et al., AIM 2010)

39. Subject and Time Interval Queries [Klimov et al., JIIS 2010] Three types of queries: –Select subjects (Who had this pattern?) –Select Time Intervals (When did this pattern occur?) –Get Data (What were the data for these subjects?) Selection constraints include: –Demographical constraints (non-temporal): ID, age, smoking, sex, political group, … –Time and value knowledge-based constraints: measured parameters, interventions, temporal-abstraction concepts Pair-wise constraints between concepts both absolute and relative (following a reference event) time lines –Statistical constraints: filter the subjects’ data on the basis of a specific statistical function

40. VISITORS: Multiple-Patient, Multiple-Concept Intelligent Browsing and Exploration Multiple-patients raw data Knowledge browser Patient groups Distribution of derived patterns over time Concept search

41. Temporal Association Charts Data of each patient are connected by line [Klimov et al., MIM 2009] support and confidence of association rules indicated by width and hue

42. Using Temporal Abstractions for Temporal Data Mining Meaningful associations typically exist among clinically meaningful, interval-based, abstract concepts (e.g., 3-5M Moderate Anemia precedes 2M Deteriorating Renal Function), rather than among time-stamped raw data such as Hemoglobin and Creatinine values Temporal Abstraction can be used to create time-oriented, interval-based, abstract concepts and patterns –By using domain knowledge; e.g., the knowledge-based temporal abstraction method [Shahar, AIJ, 1997] –By using automated temporal discretization methods [Verduijn et al., AIM, 2007; Moskovitch et al., IDAMAP, 2009] Interval-based abstract concepts can be then be mined to discover time-intervals related patterns (TIRPs) Example: The KarmaLego algorithm [Moskovitch and Shahar, IDAMAP, 2009; AMIA. 2009]; used in several domains:, such as –Analysis of diabetes-patients data –Prediction of getting off an ICU ventilator –Classification of Hepatitis type from the course of the disease

43. Temporal Interval Related Patterns-An Example A Temporal Interval Related Pattern (TIRP) is a conjunction of temporal relations among symbolic time intervals {A 1 o B, A 1 o D, A 1 m C 1, A 1 b C 2, A 1 b A 2, B o D, B c C 1, B b C 2, B b A, C 1 b C 2, C 1 b A, C 2 o A}

44. 0.2 6 0.1 8 0.2 2 0.2 8 0.2 5 0.2 3 0.3 3 0.4 2 0.2 9 Exploration of the Diabetes TIRPs Tree: An Example [Moskovitch and Shahar, AMIA 2009] An Example [Moskovitch and Shahar, AMIA 2009]

45. Visualization of TIRPs: The KarmaLegoV Tool Enables browsing of a KarmaLego TIRP enumeration tree; includes several options: –Presenting the next level, i.e., the next time-interval related to the current TIRP, and its temporal relation –Sorting by vertical support (% of patients who have the pattern), mean horizontal support (number of instances of TIRP per patient), and interestingness measures –Visualizing the current [mean] TIRP and its instances –Visualizing the distributions of external static (non-temporal) properties, such as age and gender, or a classification outcome (e.g., recovery or not), for the patient class in which the TIRP was found

46. The KarmaLegoV Tool: An Example (I)

47. The KarmaLegoV Tool: An Example (II)

48. Knowledge-Based Visualization & Exploration of Time-Oriented Data: Conclusions Interactive query, visualization, and exploration requires runtime access to the domain’s temporal-abstraction ontology The visualization and exploration semantics can be specific to the temporal-abstraction task, but need not be specific to the domain Typical examples: Computation, visualization and exploration of multiple time-oriented records, their aggregations, and their inter- and intra- temporal relations –the VISITORS system [Klimov and Shahar 2005] –The KarmaLego framework [Moskovitch and Shahar 2009]