1. Evaluating Implicit Measures to Improve the Search Experience SIGIR 2003 Steve Fox

2. Outline Background Approach Data Analysis Value-Add Contributions Result-Level Findings Session-Level Findings

3. Background Interested in implicit measures to improve users search experience What the user wants What satisfies them Significant implicit measures Needed to prove it! Two goals: Test association between implicit measures and user satisfaction Understand what implicit measures were useful within this association

4. Approach Architecture Internet Explorer add-in Client-Server Configured for MSN Search and Google Deployment Internal MS employees (n = 146) – work environment Implicit measures and explicit feedback SQL Server back-end

5. Approach, contd

6. Data Analysis Bayesian modeling at result and session level Trained on 80% and tested on 20% Three levels of SAT – VSAT, PSAT & DSAT Implicit measures: Result-LevelSession-Level Diff Secs, Duration SecsAverages of result-level measures (Dwell Time and Position) Scrolled, ScrollCnt, AvgSecsBetweenScroll, TotalScrollTime, MaxScroll Query count TimeToFirstClick, TimeToFirstScrollResults set count Page, Page Position, Absolute PositionResults visited VisitsEnd action Exit Type ImageCnt, PageSize, ScriptCnt Added to Favorites, Printed

7. Data Analysis, contd

8. Result-Level Findings 1. Dwell time, clickthrough and exit type strongest predictors of SAT 2. Printing and Adding to Favorites highly predictive of SAT when present 3. Combined measures predict SAT better than clickthrough

9. Result Level Findings, contd Only clickthrough Combined measures Combined measures with confidence of > 0.5 (80-20 train/test split)

10. Session-Level Findings Four findings: 1. Strong predictor of session-level SAT was result-level SAT 2. Dwell time strong predictor of SAT 3. Combination of (slightly different) implicit measures could predict SAT better than clickthrough 4. Some gene sequences predict SAT (preliminary and descriptive)

11. Session Level Findings, contd Common patterns in gene analysis, e.g. SqLrZ Session starts (S) Submit a query (q) Result list returned (L) Click a result (r) Exit on result (Z) PatternFrequency%VSAT%PSAT%DSAT Avg. VSAT Dwell Time Avg. PSAT Dwell Time Avg. DSAT Dwell Time SqLrZ1177515964611

12. Value-Add Contributions Deployed in the work setting Collected data in context of web search Rich user behavior data stream Annotated data stream with explicit judgment Used new methodology to analyze the data Gene analysis to analyze usage patterns Mapped usage patterns to SAT

13. Question(s)