ASSOCIATIVE BROWSING Evaluating 1 Jinyoung Kim / W. Bruce Croft / David Smith for Personal Information

* What do you remember about your documents? 2 Registration James Use search if you recall keywords!

* What if keyword search is not enough? 3 Registration Search first, then browse through documents!

* But I don’t remember any keywords! 4 You might remember a related concept! James William *Concept : entities and terms of interest to the user

* How can we build associations? 5 Manually? Participants wouldn’t create associations beyond simple tagging operations - Sauermann et al Participants wouldn’t create associations beyond simple tagging operations - Sauermann et al Automatically? How would it match user’s preference?

* Building the Associative Browsing Model 6 2. Concept Extraction 3. Link Extraction 4. Link Refinement 1. Document Collection Term Similarity Temporal Similarity Co-occurrence Click-based Training

* Concept: Search Engine Link Extraction and Refinement 7 Link Scoring Linear combination of link type scores S(c 1,c 2 ) = Σ i [ w i × Link i (c 1,c 2 ) ] Link Presentation Ranked list of suggested items Users click on them for browsing Link Refinement (training w i ) Maximize click-based relevance Grid Search : Maximize retrieval effectiveness (MRR) RankSVM : Minimize error in pairwise preference ConceptsDocuments Term Vector Similarity Temporal Similarity Tag Similarity String SimilarityPath / Type Similarity Co-occurrenceConcept Similarity

E VALUATION 8

* Evaluation based on Known-item Finding Data Collection Collect public documents in UMass CS dept CS dept. people competed in known-item finding tasks 30 participants, 53 search sessions in total Two rounds of user study Metrics Value of browsing % of sessions browsing was used % of sessions browsing was used & led to success Quality of browsing suggestions Mean Reciprocal Rank using clicks as judgments 10-fold cross validation over the click data collected 9

* DocTrack [Kim&Croft10] 10 Concept: Computer Architecture Relevant Documents:

* Evaluation Results Comparison with Simulation Results [Kim&Croft&Smith11] Roughly matches in terms of overall usage and success ratio The Value of Browsing Browsing was used in 30% of all sessions Browsing saved 75% of sessions when used Evaluation TypeTotal (#sessions) Browsing used Successful outcome Simulation [KCS11] 63,2609,410 (14.8%)3,957 (42.0%) User Study (1)29042 (14.5%)15 (35.7%) User Study (2)14243 (30.2%)32 (74.4%) Document Only Document + Concept

* Quality of Browsing Suggestions – CS Collection Concept Browsing (MRR) Document Browsing (MRR) 12

* Summary 13 Associative Browsing Model for Personal Information Evaluation based on User Study Any Questions?

* Evaluation by Simulated User Model [KCS11] Query generation model [Kim&Croft09] Select terms from a target document State transition model Use browsing when result looks marginally relevant Link selection model Click on browsing suggestions based on perceived relevance 14

* Community Efforts based on the Datasets 15

* Future Work 16 User Interface Evaluation Learning Method Learning Method Concept Map Visualization Query-based Concept Generation Exploratory Search Large-scale User Study Combine with Faceted Search More Features Active Learning

O PTIONAL S LIDES 17

* Role of Personalization 18 Using one person’s click data for training results in much higher learning effectiveness

* Quality of Browsing Suggestions – Person 1/2 Concept Browsing (MRR) Document Browsing (MRR) 19

* Building the Associative Browsing Model 20 Link Types Links between concepts Links between documents ConceptsDocuments Term Vector Similarity Temporal Similarity Tag Similarity String Similarity Path / Type Similarity Co-occurrenceConcept Similarity

* Quality of Browsing Suggestions (optional) For Concept Browsing For Document Browsing 21 (Using the CS Collection, Measured in MRR)

* Evaluation Results (optional) Success Ratio of Browsing Lengths of Successful Sessions