Presentation is loading. Please wait.

Presentation is loading. Please wait.

Personalizing Search on Shared Devices Ryen White and Ahmed Hassan Awadallah Microsoft Research, USA Contact:

Published byCori Pitts Modified over 8 years ago

Similar presentations

Presentation on theme: "Personalizing Search on Shared Devices Ryen White and Ahmed Hassan Awadallah Microsoft Research, USA Contact:"— Presentation transcript:

1 Personalizing Search on Shared Devices Ryen White and Ahmed Hassan Awadallah Microsoft Research, USA Contact: ryenw@microsoft.comryenw@microsoft.com

2 Shared Device Search 2011 Census: 75% of U.S. households have computer In most homes that machine is shared between multiple people Search engines use machine identifiers based on cookies, ids, etc. Assumes 1:1 mapping from IDs to people for analysis and personalization Shared devices in households Attributing activity to people (not machines) may improve personalization Some early indications of effectiveness in prior work (Singla et al., 2014)Singla et al., 2014

3 Is Shared Device Searching Common? Analyzed comScore search data (all engines, en-US) Both machine identifiers and person identifiers (users self-identify) Aside: Within-session shared device search less common: 97% sessions = single user Multi-user (66%) Variations in % machine ids = multi-user with different profile sizes 6 months = 66% 3 months = 57% 1 month = 44%

4 Handling Shared Device Use Limited current solutions in search engines (user sign-in) However: Requires user effort to sign in, People don’t sign out so their signals mixed Some solutions in other domains, e.g., streaming media Ideally this would happen automatically without user needing to explicitly log in Search activity attribution methods can help with this …

5 Activity Attribution Challenge Given a stream of data from a machine identifier, attribute observed historic and new behavior to the correct person Related work in signal processing and fraud detection Applications for: Personalization, Advertising, Privacy protection Question: What is upper bound on gain from attribution-based methods? We perform ORACLE study with perfect knowledge of who is searching New query Which user? User 1User 2User 1User 3 {k user clusters} History of search activity on machine “From devices to people: Attribution of search activity in multi-user settings” White et al., WWW2014White et al., WWW2014

6 Key Contributions Introduce attribution-based personalization (ABP) and estimate its value in ORACLE STUDY (perfect knowledge of who is searching) Show machine vs. person is meaningful for an important application: predicting searchers’ future interests Identify properties of interest models and queries for which ABP is best Learn model to predict when to apply ABP on a per-query basis

7 Key Contributions Introduce attribution-based personalization (ABP) and estimate its value in ORACLE STUDY (perfect knowledge of who is searching) Show machine vs. person is meaningful for an important application: predicting searchers’ future interests Identify properties of interest models and queries for which ABP is best Learn model to predict when to apply ABP on a per-query basis

8 Attribution-Based Personalization (ABP) Three phases: Activity attribution and interest model construction for individuals from historic activity Attribution of newly-observed activity to the correct searcher Application of that searcher’s specific interest model for personalization

9 Building Interest Models Build machine and person interest profiles based on the ODP hierarchy Use result clicks Category distributions can differ between people and machines, e.g., Sports/Tennis largest in machine, but only highest for one searcher (B) Some topics have broad interest, e.g., all searchers are interested in movies  Individualized models could matter Question is how much and when do they matter most and least?

10 Key Contributions Introduce attribution-based personalization (ABP) and estimate its value in ORACLE STUDY (perfect knowledge of who is searching) Show machine vs. person is meaningful for an important application: predicting searchers’ future interests Identify properties of interest models and queries for which ABP is best Learn model to predict when to apply ABP on a per-query basis

11 Dataset Two years of comScore logs Divided into two subsets: Model building: 6mo of comScore search logs for model building (Jan13 - Jun13) Evaluation: 1mo immediately following to evaluate predictions (Jul13) Result clicks from each person/machine used to construct interest models Machine click thresholds: MODEL BUILDING: ≥ least 100 clicks EVALUATION: ≥ 15 clicks Interest Model Building Evaluation Time 6 months1 month Per machine or person:

12 Prediction Task Given a query and interest model, predict ODP categories of next click Vary identifier type and match type Identifier type: Machine- or person-based Match type: All historic activity or on-task activity only On-task search activity: On-task historic activity as clicks associated with queries with at least one non-stopword term in common with current query On-task models more accurately reflects state-of-the-art in personalization (Bennett et al. SIGIR12; Teevan et al. WSDM11)Bennett et al. SIGIR12Teevan et al. WSDM11 Match type Identifier type Machine-basedPerson-based All activityab On-task activitycd

13 Prediction Task: Evaluation Metrics Precision (P): Did the top predicted label == actual label (1 or 0)? Recall (R): Did actual label appear in prediction? F1 score: Harmonic mean of P and R Reciprocal Rank: If actual label == predicted label, the score assigned was the reciprocal of the prediction rank position 1 ⁄ r, and 0 otherwise Averaged over all queries in evaluation dataset

14 Evaluation Method Given our evaluation set ()  {timestamp, machine identifier, person identifier, query, {result clicks}} for each query () in : For each identifier type in {machine, person}: For each match type in {all, on-task}: For each ∈ : If identifier type = person: If match type = all:If match type = on-task: Find all historic queries from searcher with ≥ 1 non-stopword terms in common with in the model building data Get clicked results for each of the queries and assign ODP categories to the clicked results Build an interest model () comprising the normalized distribution of ODP categories from the assignment Select top-weighted predicted label in, denoted 1 Compute the effectiveness of the method in relation to the ground truth Average metric values for matchtype across all ∈ to compute the overall performance metrics Obtain all historic queries from the searcher from the model building dataset If identifier type = machine: If match type = all:If match type = on-task: Find all historic queries from machine with ≥ 1 non-stopword terms in common with in the model building data Obtain all historic queries from the machine from the model building dataset

15 Prediction Results Recall slightly higher for machine  Machine-based models are a superset of the person-based models Focus on machines w/ 2+ users in the rest of our analysis Shared device searching is predictable accurately (White et al., WWW14)White et al., WWW14 Machine-based models are our baselines for each of the two match types Gains in precision, F1, and RR 11-15% in overall perf. 19-43% for on-task perf. Focus on F1 for remainder of analysis

16 Key Contributions Introduce attribution-based personalization (ABP) and estimate its value in ORACLE STUDY (perfect knowledge of who is searching) Show machine vs. person is meaningful for an important application: predicting searchers’ future interests Identify properties of interest models and queries for which ABP is best Learn model to predict when to apply ABP on a per-query basis

17 Impact of Additional Factors Properties of the interest models and query can influence utility of ABP Model Properties Model entropy: Entropy of the interest model (low, medium, high) Relative model size: Fraction of machine-based model Number of searchers on machine Query Properties Click entropy: Diversity of clicks (low, medium, high) Popularity: Frequency of query (low, medium, high) Topic: Top-level ODP category Focus on two highlighted factors (see paper for rest) Control for task effects by focusing on on-task model variants

18 Impact of Additional Factors Model entropy, i.e., diversity of the category (c) model on the machine (m) Query topic, i.e., top-level ODP category of the top-result for the query When the machine-based model is more diverse, then person-based methods perform better  More benefit from focus Compute the gains differentially based on features of models and the queries, e.g., Topics for which specific users already represented (only small n interested) Others where interests are more broad

19 Key Contributions Introduce attribution-based personalization (ABP) and estimate its value in ORACLE STUDY (perfect knowledge of who is searching) Show machine vs. person is meaningful for an important application: predicting searchers’ future interests Identify properties of interest models and queries for which ABP is best Learn model to predict when to apply ABP on a per-query basis

20 Applying Model and Query Properties Train a model to learn when to apply ABP on a per-query basis Featurized properties of the model and the query based on additional factors: 130k evaluation queries from 2.5k people (1k machines) 6mo/1mo build/test, MART-based classifier, 10 fold CV, 100 runs, Compute F1 Labels: Positives: ABP > Machine-level, Negatives: ABP  Machine-level Feature NameDescription MachineModelEntropyEntropy of the interest model constructed from machine activity RelativeModelSizeFraction of machine interest model occupied by classified historic clicks NumberOfSearchersNumber of distinct searchers QueryClickEntropyClick entropy for the query QueryPopularitycomputed based on the held-out Bing search log data QueryTopicTop-level ODP category of the query

21 Selective Application of ABP Best: 21% ABP, 9% baseline, 70% tied Predict which model best: Strong predictive performance (acc. = 0.918) > marginal baseline (0.791) Top features: MachineModelEntropy (max), RelativeModelSize (0.699 of max), QueryTopic (0.441 of max) ABP performance of 88-96% of the oracle Much better than always applying ABP Demonstrates the benefits of intelligently applying ABP for each query Always apply best Applying prediction in personalization:

22 Discussion Shared device searching common Oracle study showed clear utility from ABP Focused on click prediction; Other applications need to be examined Need to performance with automated ABP methods Alternative self-identification methods need to be examined (e.g., sign-in) Closer link between people and devices  impact on shared device usage?

23 Summary and Takeaway Introduced attribution-based personalization, performed oracle study Observe an increased accuracy in future interest predictions (11-19% in the F1-score, depending on match type) by applying this approach Gains vary by model/query properties, with selective application of method Significant opportunities to enhance personalization via tailored models Future work: More (non-oracle) studies with different ABP methods ABP methods for truly personalized ranking and recommendation at scale

24 Shared Device Searching may Matter Entropy of machine-based interest model increases with the number of searchers on the machine Different distributions in topical categories and interests

25 Shared Device Searching: Distribution Distribution of users searching Generally one dominant searcher (44-83% of queries) Decreases with other users, but still by far the most active + many other less active searchers

Download ppt "Personalizing Search on Shared Devices Ryen White and Ahmed Hassan Awadallah Microsoft Research, USA Contact:"

Similar presentations

Beliefs & Biases in Web Search

Beliefs & Biases in Web Search
Predicting User Interests from Contextual Information

Predicting User Interests from Contextual Information
Enhancing Personalized Search by Mining and Modeling Task Behavior

Enhancing Personalized Search by Mining and Modeling Task Behavior
Active Learning with Feedback on Both Features and Instances H. Raghavan, O. Madani and R. Jones Journal of Machine Learning Research 7 (2006) Presented.

Active Learning with Feedback on Both Features and Instances H. Raghavan, O. Madani and R. Jones Journal of Machine Learning Research 7 (2006) Presented.
Struggling or Exploring? Disambiguating Long Search Sessions

Struggling or Exploring? Disambiguating Long Search Sessions
Suleyman Cetintas 1, Monica Rogati 2, Luo Si 1, Yi Fang 1 Identifying Similar People in Professional Social Networks with Discriminative Probabilistic.

Suleyman Cetintas 1, Monica Rogati 2, Luo Si 1, Yi Fang 1 Identifying Similar People in Professional Social Networks with Discriminative Probabilistic.
Mustafa Cayci INFS 795 An Evaluation on Feature Selection for Text Clustering.

Mustafa Cayci INFS 795 An Evaluation on Feature Selection for Text Clustering.
1 Evaluation Rong Jin. 2 Evaluation  Evaluation is key to building effective and efficient search engines usually carried out in controlled experiments.

1 Evaluation Rong Jin. 2 Evaluation  Evaluation is key to building effective and efficient search engines usually carried out in controlled experiments.
Foreground Focus: Finding Meaningful Features in Unlabeled Images Yong Jae Lee and Kristen Grauman University of Texas at Austin.

Foreground Focus: Finding Meaningful Features in Unlabeled Images Yong Jae Lee and Kristen Grauman University of Texas at Austin.
Data Mining Methodology 1. Why have a Methodology  Don’t want to learn things that aren’t true May not represent any underlying reality ○ Spurious correlation.

Data Mining Methodology 1. Why have a Methodology  Don’t want to learn things that aren’t true May not represent any underlying reality ○ Spurious correlation.
Search Engines Information Retrieval in Practice All slides ©Addison Wesley, 2008.

Search Engines Information Retrieval in Practice All slides ©Addison Wesley, 2008.
Studies of the Onset & Persistence of Medical Concerns in Search Logs Ryen White and Eric Horvitz Microsoft Research, Redmond

Studies of the Onset & Persistence of Medical Concerns in Search Logs Ryen White and Eric Horvitz Microsoft Research, Redmond
1 Learning User Interaction Models for Predicting Web Search Result Preferences Eugene Agichtein Eric Brill Susan Dumais Robert Ragno Microsoft Research.

1 Learning User Interaction Models for Predicting Web Search Result Preferences Eugene Agichtein Eric Brill Susan Dumais Robert Ragno Microsoft Research.
Systems Engineering and Engineering Management The Chinese University of Hong Kong Parameter Free Bursty Events Detection in Text Streams Gabriel Pui Cheong.

Systems Engineering and Engineering Management The Chinese University of Hong Kong Parameter Free Bursty Events Detection in Text Streams Gabriel Pui Cheong.
Multiple Criteria for Evaluating Land Cover Classification Algorithms Summary of a paper by R.S. DeFries and Jonathan Cheung-Wai Chan April, 2000 Remote.

Multiple Criteria for Evaluating Land Cover Classification Algorithms Summary of a paper by R.S. DeFries and Jonathan Cheung-Wai Chan April, 2000 Remote.
Evaluating Search Engine

Evaluating Search Engine
Ryen W. White, Microsoft Research Jeff Huang, University of Washington.

Ryen W. White, Microsoft Research Jeff Huang, University of Washington.
Ryen White, Susan Dumais, Jaime Teevan Microsoft Research {ryenw, sdumais,

Ryen White, Susan Dumais, Jaime Teevan Microsoft Research {ryenw, sdumais,
Web Projections Learning from Contextual Subgraphs of the Web Jure Leskovec, CMU Susan Dumais, MSR Eric Horvitz, MSR.

Web Projections Learning from Contextual Subgraphs of the Web Jure Leskovec, CMU Susan Dumais, MSR Eric Horvitz, MSR.
Personalized Ontologies for Web Search and Caching Susan Gauch Information and Telecommunications Technology Center Electrical Engineering and Computer.

Personalized Ontologies for Web Search and Caching Susan Gauch Information and Telecommunications Technology Center Electrical Engineering and Computer.

Similar presentations

Ads by Google