1. FINDING PATTERNS IN TEMPORAL DATA KRIST WONGSUPHASAWAT TAOWEI DAVID WANG CATHERINE PLAISANT BEN SHNEIDERMAN HUMAN-COMPUTER INTERACTION LAB UNIVERSITY OF MARYLAND 27th HCIL Symposium May 27, 2010

2. FINDING PATTERNS IN TEMPORAL DATA KRIST WONGSUPHASAWAT TAOWEI DAVID WANG CATHERINE PLAISANT BEN SHNEIDERMAN HUMAN-COMPUTER INTERACTION LAB UNIVERSITY OF MARYLAND 27th HCIL Symposium May 27, 2010

3. Patient ID: 45851737 12/02/2008 14:26Arrival 12/02/2008 14:36Emergency 12/02/2008 22:44ICU 12/05/2008 05:07Floor 12/14/2008 06:19Exit Time Emergency ICU Floor Exit TEMPORAL CATEGORICAL DATA A type of time series 04/26/2010 10:0031.03 04/26/2010 10:1531.01 04/26/2010 10:3031.02 04/26/2010 10:4531.08 04/26/2010 11:0031.16 Event Category Stock: Microsoft Numerical Arrival Event

4. TEMPORAL CATEGORICAL DATA Electronic Health Records: symptoms, treatment, lab test Traffic incident logs: arrival/departure time of each unit Student records: course, paper, proposal, defense, etc. Others: web logs, usability study logs, etc.

5. 10+ years work on temporal visualization (mostly on Electronic Health Records)

6. LIFELINES SINGLE RECORD [Plaisant et al. 1998] http://www.cs.umd.edu/hcil/lifelines

7. LifeLines – Single Patient

8. working with physicians at WASHINGTON HOSPITAL CENTER

9. EXAMPLE DATA Patient transfers ARRIVALArrive the hospital EMERGENCYEmergency room ICUIntensive Care Unit INTERMEDIATEIntermediate Medical Care FLOORNormal room EXIT-ALIVELeave the hospital alive EXIT-DEADLeave the hospital dead

10. TASKS within 2 days ICUFloorICU Example: Finding “Bounce backs”

11. LIFELINES 2 RECORD [Wang et al. 2008, 2009] http://www.cs.umd.edu/hcil/lifelines2

12. LifeLines2 – Search and Visualize ARF (Align-Rank-Filter) Framework Temporal Summary Multiple Records

13. ALIGNMENT Sentinel events as reference points Time Patient #45851737Arrival Emergency ICU Floor Exit Patient #43244997 Arrival Emergency ICU Floor Exit JuneJuly August

14. ALIGNMENT (2) Time shifting Time Patient #45851737Admit Emergency ICU Floor Exit Patient #43244997 Admit Emergency ICU Floor Exit 01 M 2 M

15. SIMILAN RECORD [Wongsuphasawat & Shneiderman 2009] http://www.cs.umd.edu/hcil/similan

16. Similan – Search by Similarity

18. FINDING “BOUNCE BACKS” BeforeAfter Much faster to specify new query Visualizing the results gives better understanding

19. USER STUDIES: SEARCH Exact MUST have A, B, C Record#1 Record#2 Record#3 more similar Similarity-based SHOULD have A, B, C SimilanLifeLines2 Query Record#2 Record#1 Record#3 Query

20. USER STUDIES: SEARCH Exact MUST have A, B, C Similarity-based SHOULD have A, B, C Query Record#1 Record#2 Record#3 more similar Query Record#2 Record#1 Record#3 SimilanLifeLines2 1

21. NEW STUFF Needs for an overview -> LifeFlow!

22. TASKS within 2 days ICUFloorICU Example: Finding “Bounce backs” Other questions Arrival ICU ? ? ?

23. LIFEFLOW RECORD AGGREGATE Merge multiple records into tree VISUALIZE Display the aggregation

24. AGGREGATE Aggregate by prefix #1 #2 #3 #4 Example with 4 records

25. AGGREGATE Aggregate by prefix #1 #2 #3 #4

26. VISUALIZE Inspired by the Icicle tree [Fekete 2004] Number of files

27. VISUALIZE (2) Use horizontal axis to represent time Video

28. DEMO – LIFEFLOW When the lines are combined into flow

29. FUTURE WORK Comparison Jan-Mar 2008April-June 2008 Intermediate ICU IntermediateICU Floor

30. TAKE-AWAY MESSAGE Information visualization is a powerful way to explore temporal patterns. You can work with us on new case studies.

31. TEMPORAL CATEGORICAL DATA Electronic Health Records: symptoms, treatment, lab test Traffic incident logs: arrival/departure time of each unit Student records: course, paper, proposal, defense, etc. Others: web logs, usability study logs, etc.

32. EXAMPLE – TRAFFIC INCIDENTS

33. ACKNOWLEDGEMENT D R. P HUONG H O, D R. M ARK S MITH, D AVID R OSEMAN W ASHINGTON H OSPITAL C ENTER http://www.whcenter.org N ATIONAL I NSTITUTES OF H EALTH (NIH) - G RANT CA147489 http://www.nih.gov M ICHAEL P ACK, M ICHAEL V AN D ANIKER C ENTER FOR A DVANCED T RANPORTATION T ECHNOLOGY L AB (CATT L AB ) http://www.cattlab.umd.edu

34. TAKE-AWAY MESSAGE Information visualization is a powerful way to explore temporal patterns. You can work with us on new case studies. More demos this afternoon {kristw, tw7, plaisant, ben}@cs.umd.edu http://www.cs.umd.edu/hcil/temporalviz

35. Q&A Questions? {kristw, tw7, plaisant, ben}@cs.umd.edu http://www.cs.umd.edu/hcil/temporalviz

36. THANK YOU Thank you

37. BACKUP SLIDES Junkyard...

38. LIFELINES2 8 case studies –Bounce backs –Step ups –BIPAP –Etc.

39. DR. P

40. Does not help exploring sequential patterns Needs a new overview LifeLines2’s Temporal Summary [Wang et al. 2009] Continuum’s Histogram [Andre 2007]

41. USER STUDIES 8 Extensive case studies Compared LifeLines2 with Similan –Learn advantages & disadvantages Drawing is preferred Clear cut off points is needed Working on improvements –Flexible temporal search

42. SIMILAN Compared with LifeLines2 in an experiment –Learn advantages & disadvantages –Drawing is preferred –No clear cut off points Working on improvements –Flexible temporal search

43. LIFEFLOW AGGREGATE VISUALIZE Merge multiple records into tree Display the tree

44. APPROACHES Exact Search MUST have A, B, C Similarity-based Search SHOULD have A, B, C Query Record#1 Record#2 Record#3 more similar Query Record#1 Record#2 Record#3

45. MOTIVATION RESEARCH QUESTIONS RESEARCH QUESTION#1 PRELIM. + PROPOSED WORK CONCLUSION RESEARCH QUESTION#2 PRELIM. + PROPOSED WORK

47. EXPECTED CONTRIBUTIONS 1.Design of visual representations, user interfaces and interaction techniques 2.Algorithms for flexible temporal search 3.Evaluation results 4.Open new directions for exploring temporal categorical data

48. NEEDS FOR AN OVERVIEW We learn

49. NEEDS Visualize overview or show summary Where should I start?

50. TEMPORAL VISUALIZATIONS Background and related work

51. RELATED WORK Single record Patient ID: 45851737 12/02/2008 14:26Arrival 12/02/2008 14:26Emergency 12/02/2008 22:44ICU 12/05/2008 05:07Floor 12/08/2008 10:02Floor 12/14/2008 06:19Exit Visualization E.g. LifeLines, MIDGAARD, etc.

52. RELATED WORK (2) Multiple records Visualization E.g. LifeLines2, Continuum, ActiviTree, etc. Patient ID: 45851737 12/02/2008 14:26Arrival 12/02/2008 14:26Emergency 12/02/2008 22:44ICU 12/05/2008 05:07Floor 12/08/2008 10:02Floor 12/14/2008 06:19Exit Patient ID: 45851737 12/02/2008 14:26Arrival 12/02/2008 14:26Emergency 12/02/2008 22:44ICU 12/05/2008 05:07Floor 12/08/2008 10:02Floor 12/14/2008 06:19Exit Patient ID: 45851737 12/02/2008 14:26Arrival 12/02/2008 14:26Emergency 12/02/2008 22:44ICU 12/05/2008 05:07Floor 12/08/2008 10:02Floor 12/14/2008 06:19Exit Patient ID: 45851737 12/02/2008 14:26Arrival 12/02/2008 14:26Emergency 12/02/2008 22:44ICU 12/05/2008 05:07Floor 12/08/2008 10:02Floor 12/14/2008 06:19Exit

53. More space please....

54. INFORMATION VISUALIZATION MANTRA OVERVIEW FIRST, ZOOM AND FILTER, THEN DETAILS ON DEMAND

55. RELATED WORK (3) Multiple records Visualization E.g. LifeLines2, Continuum Patient ID: 45851737 12/02/2008 14:26Arrival 12/02/2008 14:26Emergency 12/02/2008 22:44ICU 12/05/2008 05:07Floor 12/08/2008 10:02Floor 12/14/2008 06:19Exit Patient ID: 45851737 12/02/2008 14:26Arrival 12/02/2008 14:26Emergency 12/02/2008 22:44ICU 12/05/2008 05:07Floor 12/08/2008 10:02Floor 12/14/2008 06:19Exit Patient ID: 45851737 12/02/2008 14:26Arrival 12/02/2008 14:26Emergency 12/02/2008 22:44ICU 12/05/2008 05:07Floor 12/08/2008 10:02Floor 12/14/2008 06:19Exit Patient ID: 45851737 12/02/2008 14:26Arrival 12/02/2008 14:26Emergency 12/02/2008 22:44ICU 12/05/2008 05:07Floor 12/08/2008 10:02Floor 12/14/2008 06:19Exit

56. SEQUENTIAL PATTERNS within 2 days ICUFloorICU Examples: “Bounce backs” Patient #1 Patient #2 Patient #3 Patient #4

57. DESIGN AN OVERVIEW Sequential patterns Scalability vs. Loss of information