Presentation is loading. Please wait.

Presentation is loading. Please wait.

Better Logging to Improve Interactive Data Analysis Tools Sara Alspaugh......... Archana Ganapathi.......

Similar presentations


Presentation on theme: "Better Logging to Improve Interactive Data Analysis Tools Sara Alspaugh......... Archana Ganapathi......."— Presentation transcript:

1 Better Logging to Improve Interactive Data Analysis Tools Sara Alspaugh Archana Ganapathi Marti Hearst Randy Katz

2 :28: INFO AuditLogger - Audit:[timestamp= :28:01.134, user=splunk-system-user, action=search, info=granted, search_id=‘scheduler__nobody__testing__RMD56569fcf2f137b840_at_ _101256’, search=‘search index=_internal metrics per_sourcetype_thruput | head 100’, autojoin=‘1', buckets=0, ttl=120, max_count=500000, maxtime= , enable\_lookups=‘1', extra_fields=‘’, apiStartTime=‘ZERO_TIME', apiEndTime=‘Fri Sep 28 18:28: ', savedsearch_name=“sample scheduled search for dashboards (existing job case)”] event

3 :28: INFO AuditLogger - Audit:[timestamp= :28:01.134, user=splunk-system-user, action=search, info=granted, search_id=‘scheduler__nobody__testing__RMD56569fcf2f137b840_at_ _101256’, search=‘search index=_internal metrics per_sourcetype_thruput | head 100’, autojoin=‘1', buckets=0, ttl=120, max_count=500000, maxtime= , enable\_lookups=‘1', extra_fields=‘’, apiStartTime=‘ZERO_TIME', apiEndTime=‘Fri Sep 28 18:28: ', savedsearch_name=“sample scheduled search for dashboards (existing job case)”] timesta mp event

4 :28: INFO AuditLogger - Audit:[timestamp= :28:01.134, user=splunk-system-user, action=search, info=granted, search_id=‘scheduler__nobody__testing__RMD56569fcf2f137b840_at_ _101256’, search=‘search index=_internal metrics per_sourcetype_thruput | head 100’, autojoin=‘1', buckets=0, ttl=120, max_count=500000, maxtime= , enable\_lookups=‘1', extra_fields=‘’, apiStartTime=‘ZERO_TIME', apiEndTime=‘Fri Sep 28 18:28: ', savedsearch_name=“sample scheduled search for dashboards (existing job case)”] event use r timesta mp

5 :28: INFO AuditLogger - Audit:[timestamp= :28:01.134, user=splunk-system-user, action=search, info=granted, search_id=‘scheduler__nobody__testing__RMD56569fcf2f137b840_at_ _101256’, search=‘search index=_internal metrics per_sourcetype_thruput | head 100’, autojoin=‘1', buckets=0, ttl=120, max_count=500000, maxtime= , enable\_lookups=‘1', extra_fields=‘’, apiStartTime=‘ZERO_TIME', apiEndTime=‘Fri Sep 28 18:28: ', savedsearch_name=“sample scheduled search for dashboards (existing job case)”] event use r timesta mp actio n

6 :28: INFO AuditLogger - Audit:[timestamp= :28:01.134, user=splunk-system-user, action=search, info=granted, search_id=‘scheduler__nobody__testing__RMD56569fcf2f137b840_at_ _101256’, search=‘search index=_internal metrics per_sourcetype_thruput | head 100’, autojoin=‘1', buckets=0, ttl=120, max_count=500000, maxtime= , enable\_lookups=‘1', extra_fields=‘’, apiStartTime=‘ZERO_TIME', apiEndTime=‘Fri Sep 28 18:28: ', savedsearch_name=“sample scheduled search for dashboards (existing job case)”] timesta mp use r actio n event parameters execution environment configuration and version stack trace

7 Why do we need better logging? Motivation

8

9

10

11 Visualizing records of user activity to help optimize the user experience using Google Analytics Goal Flow Tool

12 Applications of Good User Activity Records Jaideep Srivastava, Robert Cooley, Makund Deshpande and Pang-Ning Tan. “Web usage mining: discovery and applications of usage patterns from web data.” SIGKDD Explorations Newsletter recommenders predictive interfaces task guidelines activity visualizations traffic analysis UX optimization

13 Examples of this in IDEA tools SYF: Systematic yet flexible (Perer and Shneiderman) –social network analysis tool –task guidelines for exploring social network data –users can provide feedback on task usefulness –records when users have completed tasks SeeDB (Parameswaran, Polyzotis, Garcia- Molina) –recommend visualizations for a given SQL query Adam Perer and Ben Shneiderman. “Systematic yet flexible discovery: guiding domain experts through exploratory data analysis.” Conference on Intelligent User Interfaces (IUI) Aditya Parameswaran, Neoklis Polyzotis, and Hector Garcia-Molina. “SeeDB: visualizing database queries efficiently.” International Conference on Very Large Databases (VLDB)

14

15 “Understanding the domain experts’ tasks is necessary to defining the systematic steps for guided discovery. Although some professions such as physicians, field biologists, and forensic scientists have specific methodologies defined for accomplishing tasks, this is rarer in data analysis. Interviewing analysts, reviewing current software approaches, and tabulating techniques common in research publications are important ways to deduce these steps.”

16 Some problems with logging ICSE 2012 study of logging best practices looks at four top OSS projects, finds logging is: –“often a subjective and arbitrary practice” –“seldom a core feature provided by the vendors” –“written as ‘after-thoughts’ after a failure” –“arbitrary decisions on when, what and where to log” Ding Yuan, Soyeon Park, and Yuanyuan Zhou. “Characterizing logging practices in open-source software.” International Conference on Software Engineering (ICSE)

17 “... it is critical to gain access to a stream of user actions. Unfortunately, systems and applications have not been written with an eye to user modeling." Eric Horvitz, Jack Breese, David Heckerman, David Hovel, and Koos Rommelse. “The Lumière project: Bayesian user modeling for inferring the goals and needs of software users.” Conference on Uncertainty in Artificial Intelligence

18 Recommendations Plan ahead to capture high-level user actions when designing the system. Track detailed provenance for all events. Observe intermediate user actions that are not “submitted” to the system. Record the metadata and statistics of the data set being analyzed. Collect user goals and feedback. Work towards a standard for logging data analysis activity records.

19 Plan ahead to capture high-level user actions when designing the system. Recommendation #1

20 High-level task: clustering in Excel

21

22 Examples of this in IDEA tools HARVEST (Gotz and Zhou) –visual analytics tool that incorporates action semantics not events as core design element –based on catalogue of common analytics actions derived through review of many analytics systems –exposes high-level actions that retain rich semantics as way of interacting with data David Gotz and Michelle Zhou. “Characterizing users’ visual analytic activity for insight provenance.” Symposium on Visual Analytics Science and Technology (VAST)

23 “...work in this area has relied on either manually recorded provenance (e.g., user notes) or automatically recorded event- based insight provenance (e.g., clicks, drags, and key-presses), both approaches have fundamental limitations.”

24 Track detailed provenance for all events. Recommendation #2

25 :28: INFO AuditLogger - Audit:[timestamp= :28:01.134, user=salspaugh, action=search, info=granted, search_id=`scheduler__nobody__testing__RMD56569fcf2f137b840_at_ _101256', search=`search source=*access_log* | eval http_success = if(status=200, true, false) | timechart count by http_success’, autojoin=`1', buckets=0, ttl=120, max_count=500000, maxtime= , enable\_lookups=`1', extra_fields=`', apiStartTime=`ZERO_TIME', apiEndTime=`Fri Sep 28 18:28: ', savedsearch_name=“”] interactively entered at search bar triggered by dashboard reload issued from external user script bad if same event is logged sources of data transformation activity

26 “...the log files do not differentiate between Show Me and Show Me Alternatives. These commands are implemented with the same code and the log entry is generated when the command is successfully executed.” Visualization recommendation in Tableau’s Show Me.

27 Record the metadata and statistics of the data set being analyzed. Recommendation #3

28 data actio n scatter plot bar chart {categorical, categorical} {categorical, quantitative} {quantitative, quantitative} Toy Example Influence Diagram Toy Example Conditional Probability Table action data P( action | data )

29 Wolfram Predictive Interface in Mathematica Recommendation ranking based on the data Initial recommendation ranking

30 Collect user goals and feedback. Recommendation #4

31

32

33

34 Work towards a standard for logging data analysis activity records. Recommendation #5

35 Conclusion Goal: improve interactive data exploration and analysis (IDEA): interfaces, recommender systems, task guidelines, predictive suggestions Problem: need better data to mine Recommendations for logging IDEA activity When you build your next system for IDEA, will you consider how you log user activity?


Download ppt "Better Logging to Improve Interactive Data Analysis Tools Sara Alspaugh......... Archana Ganapathi......."

Similar presentations


Ads by Google