Topics and Transitions: Investigation of User Search Behavior Xuehua Shen, Susan Dumais, Eric Horvitz

What’s next for the user?

Outline Problem Automatic Topic Tagging Predictive models Evaluation Experiments and analysis Conclusion and future directions

Problem Opportunity: Personalizing search Focus: What topics do users explore? How similar are users to each other, to special groups, and to the population at large? Data, data, data… MSN search engine log Query & clickthrough 87,449,277 rows, 36,895,634 URLs 5% sample from MSN logs, 05/29-06/29 Create predictive models of topic of queries and urls visited

Automatic Topic Tagging ODP (Open Directory Project) manually categorize URLs MSN extended methods with heuristics to cover more urls We develop a tool to automatically tag every URL in the log 15 top-level categories Arts, Business, Computers, Games, Health, Home, Kids_and_Teens, News, Recreation, Reference, Science, Shopping, Society, Sports, Adult

A Snippet multiple tagging Avg: 1.38 tags per URL ActionID ClientID ElapedTime Action Value TopCat 2 000005b8 210455 C http://www.wwltv.com Arts 3 320149 http://www.interactclaims.com/shell Undefined 549148 Q Birth certificate NULL 8 00000857 2240996 yaho 5 2240843 http://www.nextag.com Home 4 000013d1 910382 http://tv.yahoo.com/news/ap/20040530/108595548000.html 910392 french translator 910351 http://tv.zap2it.com/tveditorial/tve_main/1,1002,271|88515|1|,00.htm 6 541972 http://www.nationalenquirer.com/stories/news.cfm?instanceid=6180 Society 12 000018de 2569530 http://www.framesdirect.com Shopping 7 2568961 http://www.macrocap.com/Lower-Back-Pain 10 2569174 http://www.coolrunning.com/engine/2/2_5/193.shtml Regional Sports multiple tagging Avg: 1.38 tags per URL

Predictive Model: User Perspective Individual model Use only individual clickthrough to build a model for each user’s predictions Group model Group similar users to build a model for each group’s prediction (e.g., group users with same ‘max topic’ clickthrough) Population model Use clickthrough data for all users to build a model for all users predictions

Predictive Model: Considering Time Dependence ? Marginal model Base probability for topics Markov model Probability of moving from one topic to another Time-interval-specific Markov model User search behavior has two different patterns ? ?

Evaluation Metrics KL (Kullback-Leibler) Divergence Likelihood Top K Match the real top K topics and predicted top K’ topics

Experiment 5 weeks data (05/22-06/29) Build models based on different subsets of total data Do prediction for a “holdout set”: Other weeks data

Results from Basic Experiment Marginal model: Individual model has best performance Markov model: Consistently better than corresponding marginal model Markov model: Individual model has no best performance: Why?

Results: Training Data Size Greater amounts of training data  Markov (same for Marginal) models improve But: Individual Markov model still can’t beat Population Markov model

Results: Smoothing Using population Markov model to smooth helps individual Markov model But: smoothed individual Markov model still can’t outperform population model

Results: Time Decay Effect When time of training data decays, the prediction accuracy decreases

Results: Time-Interval-Specific Markov Model Markov Models capture short time access pattern better

Conclusion Use ODP categorization to tag URLs visited by users Construct marginal and Markov models using tagged URLs Explore performance of marginal and Markov models to predict transitions among topics Set of results relating topic transition behaviors of population, groups, and specific users

Directions Study of reliability, failure modes of automated tagging process (use of expert human taggers) Combination of query and clickthrough topics Formulating and studying different groups of people Topic-centric evaluation Application of results in personalization of search experience Interpretation of topics associated with queries Ranking of results Designs for client UI

Acknowledgement Susan and Eric for great mentoring and discussion Johnson and Muru for development support Haoyong for MSN Search Engine development environment

Backup Slides

Results from Basic Experiment Model Individ. Group Pop #URLs #Users G>P G<P G>I G<I I>P I<P W0/W1 Cur=1 Pre=1 W0/W1 Cur=1 Pre=2 W0/W1 Cur=1 Pre=3 W0/W1 Likelihood Marginal 0.274 0.274 0.176 218950 5608 2592 1240 2592 1240 Markov 0.294 0.298 0.421 207929 5508 1488 3305 1957 1423 1276 3401 Model Individ. Group Pop #URLs #Users G>P G<P G>I G<I I>P I<P Marginal 0.411 0.403 0.314 218950 5608 2539 1276 1745 1764 2816 1676 Markov 0.453 0.553 0.537 207929 5508 2568 1791 3596 701 1462 3194 Model Individ. Group Pop #URLs #Users G>P G<P G>I G<I I>P I<P Marginal 0.507 0.501 0.418 218950 5608 2504 1246 2106 1246 2883 1824 Markov 0.516 0.640 0.623 207929 5508 2554 1783 3948 542 1216 3430 Model Individ. Group Pop #URLs #Users G>P G<P G>I G<I I>P I<P Marginal 0.204 0.162 0.097 218950 5608 3763 1549 1669 3643 4268 1044 Markov 0.229 0.217 0.208 207929 5508 2540 2635 2448 2688 2707 2468 Marginal model: Individual model has best performance Markov model: Consistently better than corresponding marginal model Markov model: Population model has best performance: Why?

Results: Training Data Size 1830 585 660 910 1808 718 5508 82938 0.415 0.293 0.288 Markov 719 1284 5608 86754 0.179 0.272 Marginal I<P I>P G<I G>I G<P G>P #User #URL Pop Group Individual Model 1684 759 818 1208 1586 881 6153 87749 0.416 0.356 0.340 671 1448 91105 0.182 0.296 1458 814 842 1274 1165 891 0.419 0.395 0.374 613 1492 0.312 1337 894 915 1247 974 906 0.407 0.389 578 1560 0.323 W0/W4 Cur=1 Pre=1 W0+W1 / W4 Cur=1 Pre=1 W0+W1+W2 / W4 Cur=1 Pre=1 W0+W1+W2+W3 / W4 Cur=1 Pre=1 Greater amounts of training data  Marginal and Markov models improve But: Individual Markov model still can’t beat Population Markov model

Results: Smoothing Individual Marginal model with Jelinek- Mercer Smoothing W0 / W1 Cur=1 Pre=1 Model Individual Group Pop #URL #User G>P G<P G>I G<I I>P I<P Marginal lambda=0.0 0.274 0.176 218950 5608 2592 1240 Markov lambda =1.0 0.294 0.298 0.421 207929 5508 1488 3305 1957 1423 1276 3401 lambda=0.9 0.290 0.276 0.367 1711 3052 1248 1322 1697 2987 lambda=0.8 0.289 0.300 2137 2436 886 1089 2239 2351 lambda=0.7 0.287 0.265 2363 2183 651 864 2449 2081 lambda=0.6 0.285 0.259 2374 2131 500 677 2448 2044 lambda=0.5 0.282 0.228 2486 1828 420 563 2579 1772 lambda=0.4 0.281 2489 1827 279 384 2559 1783 lambda=0.3 0.279 2487 1817 205 259 2531 1786 lambda=0.2 0.278 0.226 2280 1802 165 191 2510 1787 lambda=0.1 0.277 0.171 2530 1180 133 131 2573 1229

Results: Smoothing (2) Population Markov model with Jelinek- Mercer Smoothing W0 / W1 Cur=1 Pre=1 Model Individual Group Pop #URL #User G>P G<P G>I G<I I>P I<P Markov lambda =1.0 0.294 0.298 0.421 207929 5508 1488 3305 1957 1423 1276 3401 lambda=0.9 0.349 1949 2468 1214 2825 lambda=0.8 0.351 1930 2500 1224 2787 lambda=0.7 0.355 1903 2554 1239 2718 lambda=0.6 0.363 1839 2661 1256 2591 lambda=0.5 0.373 1772 2788 1283 2364 lambda=0.4 0.383 1693 2922 2085 lambda=0.3 0.395 1598 3077 1155 1686 lambda=0.2 0.406 1515 3211 943 1200 lambda=0.1 0.416 1483 3284 610 597 lambda=0.0

Results: Time-Interval-Specific Differentiated Markov Model W0+W1 / W2+W3+W4 Cur=1 Pre=1 Model Individual Group Pop #URL #User G>P G<P G>I G<I I>P I<P Markov S = 1 0.412 0.460 0.495 85376 4408 798 1729 1694 683 653 2322 0.315 0.316 0.371 231209 4821 1472 2692 2022 1465 1261 2766 S = 2 0.399 0.447 0.483 125322 4584 965 2055 2031 839 773 2662 0.299 0.300 0.353 191263 4805 1541 2574 1962 1340 1289 2668 S = 5 0.398 0.435 0.474 174086 4691 1083 2280 2217 1051 876 2855 0.272 0.279 0.318 142499 4775 1689 2307 1851 1211 1385 2448 S = 10 0.393 0.429 0.468 202206 4724 1161 2381 2274 1117 929 2917 0.254 0.269 0.289 114379 4741 1839 2028 1817 1031 1464 2226 S = 15 0.389 0.423 0.465 214965 4736 1197 2467 2289 1176 940 2926 0.246 0.262 0.274 101620 4723 1903 1897 1758 997 1535 2075 S = 20 0.387 0.420 0.462 222892 4745 1210 2466 2303 1194 953 2942 0.242 0.257 0.264 93693 4708 1921 1828 1710 974 1549 2013 S = 30 0.383 0.415 0.458 232500 4758 1217 2536 2315 1221 967 2987 0.238 0.252 84085 4686 1974 1734 1668 962 1613 1923 S = 60 0.378 0.408 0.449 246607 4768 1246 2561 2339 1243 986 2988 0.232 0.248 69978 4660 1951 1637 1552 908 1624 1849 S=Infinity/ pure 0.344 0.356 0.410 316585 6153 1439 2764 1448 1215 2963 Marginal 0.302 0.183 321541 2496 1003

Results: Time Decay Effect When time of training data decays, the prediction accuracy decreases

Results: Smoothing Using Marginal distribution to smooth Markov model does not help