Download presentation

Presentation is loading. Please wait.

Published byHaley Wagner Modified over 4 years ago

1
**Statistical Models for Web Search Click Log Analysis**

Fan Guo Chao Liu Carnegie Mellon University Microsoft Research-Redmond

2
**Prologue Search Results for “CIKM” # of clicks received 3/27/2017**

CIKM'09 Tutorial, Hong Kong, China

3
**Prologue Adapt ranking to user clicks? # of clicks received 3/27/2017**

CIKM'09 Tutorial, Hong Kong, China

4
**Prologue Tools needed for non-trivial cases # of clicks received**

3/27/2017 CIKM'09 Tutorial, Hong Kong, China

5
**Motivation – Click Data Are Valuable**

One of the most extensive (yet indirect) surveys of user experience. For researchers: Help understand human interaction with IR results Design and calibrate novel models and hypotheses For practitioners: Measure, monitor and improve search engine performance. Attract more page views and clicks, boost profit 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

6
Tutorial Goals Introduce problems and applications in web search click modeling. Present latest development of click models in web search. Provide examples and discuss trade-offs for model design, implementation and evaluation. 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

7
Presenters – Fan Guo Ph.D. Student (exp. 2011), Computer Science Department, Carnegie Mellon University Advisor: Christos Faloutsos Dissertation topic: graph mining for large bioinformatics image databases 2008, M.S., CMU 2005, B.E., Tsinghua University, Beijing, China 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

8
Presenters – Chao Liu Researcher, Internet Services Research Center (ISRC), MSR- Redmond. Research focus: large-scale search/browsing log analysis for effective Web information access. 2007, Ph.D., UIUC 2005, M.S., UIUC Advisor: Jiawei Han Dissertation on statistical debugging and automated failure analysis 2003, B.S., Peking University, China 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

9
**Outline Introduction Designing click models Bayesian click models**

Selected topics on click models Conclusion 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

10
**Outline Introduction Web search click logs**

Interpret clicks as relevance feedback Building statistical models for clicks Applications of click models Designing click models Bayesian click models Selected topics on click models Conclusion 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

11
**Diverse User Feedbacks**

Click-through Browser action Dwelling time Explicit judgment Other page elements 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

12
Web Search Click Log Auto-generated data keeping important information about search activity. Query cikm Session ID f851c5af178384d12f3d Position URL Click 1 cikm2008.org 2 3 4 5 6 cikmconference.org 7 Ir.iit.edu/cikm2004 8 9 10 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

13
**Web search click log A real world example 3/27/2017**

CIKM'09 Tutorial, Hong Kong, China

14
**Web Search Click Log How large is the click log?**

search logs: 10+ TB/day In existing publications: [Craswell+08]: 108k sessions [Dupret+08] : 4.5M sessions (21 subsets * 216k sessions) [Guo +09a] : 8.8M sessions from 110k unique queries [Guo+09b]: 8.8M sessions from 110k unique queries [Chapelle+09]: 58M sessions from 682k unique queries [Liu+09a]: 0.26PB data from 103M unique queries 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

15
**Web Search Click Log How large is one ? 3/27/2017**

CIKM'09 Tutorial, Hong Kong, China

16
**Outline Introduction Web search click logs**

Interpret clicks as relevance feedback Building statistical models for clicks Applications of click models Designing click models Bayesian click models Selected topics on click models Conclusion 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

17
**Interpret Clicks: an Example**

Clicks are good… Are these two clicks equally “good”? Non-clicks may have excuses: Not relevant Not examined 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

18
**Eye-tracking User Study**

3/27/2017 CIKM'09 Tutorial, Hong Kong, China

19
Click Position-bias Normal Position Percentage Higher positions receive more user attention (eye fixation) and clicks than lower positions. This is true even in the extreme setting where the order of positions is reversed. “Clicks are informative but biased”. Reversed Impression Percentage [Joachims+07] 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

20
**Clicks as Relative Judgments**

“Clicked > Skipped Above” [Joachims02] Preference pairs: #5>#2, #5>#3, #5>#4. Use Rank SVM to optimize the retrieval function. Limitation: Confidence of judgments Little implication to user modeling 1 2 3 4 5 6 7 8 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

21
**Outline Introduction Web search click logs**

Interpret clicks as relevance feedback Building statistical models for clicks Applications of click models Designing click models Bayesian click models Selected topics on click models Conclusion 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

22
**Problem Definition Given a set of web search click logs:**

Predict clicks: output the probability of click vectors given a new order of URLs. 210 possibilities! 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

23
**The Heart of Solution Given a set of web search click logs:**

Estimate relevance: measures how good a URL is with regard to the information need of the query/user. Relevance score = 0.5 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

24
Measuring Relevance The probability of a click if the document appears at the top position. Relevance score = 0.5 indicates that on average, the document will be clicked once per 2 sessions. Bayesian click models characterize relevance using a probability distribution Relevance score Density function 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

25
Desired Properties Effective: aware of the position-bias and address it properly Scalable: linear complexity for both time and space, easy to parallel Incremental: flexible for model update based on new data 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

26
**Outline Introduction Web search click logs**

Interpret clicks as relevance feedback Building statistical models for clicks Applications of click models Designing click models Bayesian click models Selected topics on click models Conclusion 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

27
**Applications of click models**

Optimizing the retrieval function Ranking alternation based on clicks [Liu+09b] 0.90 0.10 0.08 0.05 0.20 0.72 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

28
**Applications of click models**

Optimizing the retrieval function Ranking alternation based on clicks As a feature to a learning-to-rank system (e.g., RankNet [Burges+05] ) 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

29
**Applications of click models**

Online advertising User model for sponsored search auctions 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

30
**Applications of click models**

Online advertising User model for sponsored search auctions Click through rate (CTR) prediction [Zhu+10] The image shows the CTR of ads on a commercial search engine for a typical day (Image by courtesy of Zeyuan Zhu) 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

31
**Applications of click models**

Search engine evaluation Pskip [Wang+09]: click-through-rate above last clicks; dwelling time features could also be incorporated. 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

32
**Applications of click models**

Search engine evaluation Pskip [Wang+09]: click-through-rate above last clicks; Search relevance score [Guo+09c]: average relevance score weighted by chance of examination 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

33
**Applications of click models**

User behavior analysis A preliminary work showing different user behavior patterns for navigational and informational queries [Guo+09c] 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

34
**Outline Designing click models Basic user hypotheses**

Introduction Designing click models Basic user hypotheses Modeling the first click Extending to multiple clicks Summary of model design Bayesian click models Selected topics on click models Conclusion 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

35
**Examination Hypothesis [Richardson+07]**

A document must be examined before a click. The (conditional) probability of click upon examination depends on document relevance. 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

36
**Examination Hypothesis [Richardson+07]**

The click probability could be decomposed: Global component: the examination probability which reflects the position-bias Local component: depends on the (query, URL) pair only The building block for every existing model! 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

37
**Cascade Hypothesis [Craswell+08]**

The first document is always examined. First-order Markov property: Examination at position (i+1) depends on examination and click at position i only Examination follows a strict linear order: Position i Position (i+1) 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

38
**Cascade Hypothesis [Craswell+08]**

The first document is always examined. First-order Markov property: Examination at position (i+1) depends on examination and click at position i only Examination follows a strict linear order: Position i Position (i+1) 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

39
**Cascade Hypothesis [Craswell+08]**

Limitation: examination/click rate monotonically decreases with rank, which is not always true. Some models do not follow this hypothesis (e.g., UBM) Web search data in [Guo+09a] Ads click data in [Zhu+10] 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

40
**Outline Designing click models Basic user hypotheses**

Introduction Designing click models Basic user hypotheses Modeling the first click Extending to multiple clicks Summary of model design Bayesian click models Selected topics on click models Conclusion 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

41
**Cascade model Put together two hypotheses:**

Formal model specification: P(Ci=1|Ei=0) = 0, P(Ci=1|Ei=1) = rui P(E1=1) =1, P(Ei+1=1|Ei=0) = 0 P(Ei+1=1|Ei=1, Ci=0)=1 Cascade Model = [Craswell+08] examination hypothesis cascade hypothesis modeling a single click 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

42
**Cascade model The user behavior chart: Done Examine the URL Click? No**

See Next URL? Yes Yes Done Index for URL at position i 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

43
**The chance that user may immediately abandon examination w/o a click.**

Alternatives First click in Click Chain Model [Guo+09b] as well as Dynamic Bayesian Network model [Chapelle+09] Examine the URL Click? No See Next URL? Yes Yes No The chance that user may immediately abandon examination w/o a click. Done Done 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

44
**Position-dependent parameters**

Alternatives First click in User Browsing Model [Dupret+08] Examine the URL Click? No See Next URL? Yes No i ←i+1 Yes Done Position-dependent parameters 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

45
**Outline Designing click models Basic user hypotheses**

Introduction Designing click models Basic user hypotheses Modeling the first click Extending to multiple clicks Summary of model design Bayesian click models Selected topics on click models Conclusion 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

46
**Dependent Click Model [Guo+09a]**

Generalize the cascade model to 1+ clicks: P(Ci=1|Ei=0) = 0, P(Ci=1|Ei=1) = rui P(E1=1) =1, P(Ei+1=1|Ei=0) = 0 P(Ei+1=1|Ei=1, Ci=0)=1 P(Ei+1=1|Ei=1, Ci=1)= λi λ:global parameters characterizing user browsing behavior 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

47
**Dependent Click Model [Guo+09a]**

Generalize the cascade model to 1+ clicks: 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

48
**Dependent Click Model [Guo+09a]**

DCM Algorithms: Input: for each query session, the query term, with (URL, clicked) tuple for all top-10 positions. Output: relevance for each (query, URL) pair; global parameters for user behavior Method: approximate* maximum-likelihood estimation. *Footnote: the algorithm maximizes a lower bound of log-likelihood function. 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

49
**Detour: last clicked position**

Query cikm Session ID f851c5af178384d12f3d Position URL Click 1 cikm2008.org 2 3 4 5 6 cikmconference.org 7 Ir.iit.edu/cikm2004 8 9 10 Last clicked position 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

50
**Detour: last clicked position**

Query cikm Session ID ab8dee4c4dd21e6aaf03 Position URL Click 1 cikm2008.org 2 3 4 5 cikmconference.org 6 7 Ir.iit.edu/cikm2004 8 9 10 Last clicked position 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

51
**DCM Algorithms [Guo+09a]**

The estimation formula for relevance: empirical CTR measured before last clicked position The estimation formula for global (user behavior) parameters: empirical probability of “clicked-but-not-last” 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

52
**DCM Implementation [Guo+09a]**

Details DCM Implementation [Guo+09a] Keep 3 counts for each (query, URL) pair Then 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

53
**Click Chain Model [Guo+09b]**

The examine-next probability depends on the relevance of the URL clicked: Not what I want, go to examine the next Aha, this is the right one, and I’m done! 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

54
**Click Chain Model [Guo+09b]**

The examine-next probability depends on the relevance of the URL clicked: P(Ei+1=1|Ei=1, Ci=1)= α2(1-rui) + α3rui P(Ei+1=1|Ei=1, Ci=0)= α1 where 0 < α1 ≤ 1, 0 ≤ α3< α2≤ 1 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

55
**Click Chain Model [Guo+09b]**

The full picture: 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

56
**DBN Model [Chapelle+09] Conclusion: attractive, but not satisfactory.**

There is a subtle difference between the relevance of the URL snippet and the landing page. hmmm…, this looks pretty nice errr…, it’s way out of date Conclusion: attractive, but not satisfactory. 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

57
DBN Model [Chapelle+09] The examine-next probability depends on the “satisfaction score”: P(Ei+1=1|Ei=1, Ci=1)= γ(1-sui) + 0sui P(Ei+1=1|Ei=1, Ci=0)= γ where 0 < γ ≤1 The click probability is associated with “attractiveness score”: P(Ci=1|Ei=1)= aui 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

58
**DBN Model [Chapelle+09] The full picture: 3/27/2017**

CIKM'09 Tutorial, Hong Kong, China

59
**User Browsing Model [Dupret+08]**

The examine-next probability depends on both the preceding clicked position r, and the distance to this position d. Position URL Click 1 cikm2008.org 2 3 4 5 cikmconference.org 6 … r = 0 d = 1 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

60
**User Browsing Model [Dupret+08]**

The examine-next probability depends on both the preceding clicked position r, and the distance to this position d. Position URL Click 1 cikm2008.org 2 3 4 5 cikmconference.org 6 … r = 0 d = 2 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

61
**User Browsing Model [Dupret+08]**

The examine-next probability depends on both the preceding clicked position r, and the distance to this position d. Position URL Click 1 cikm2008.org 2 3 4 5 cikmconference.org 6 … r = 2 d = 1 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

62
**User Browsing Model [Dupret+08]**

The examine-next probability depends on both the preceding clicked position r, and the distance to this position d. Position URL Click 1 cikm2008.org 2 3 4 5 cikmconference.org 6 … r = 2 d = 2 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

63
**User Browsing Model [Dupret+08]**

The examine-next probability depends on both the preceding clicked position r, and the distance to this position d. Position URL Click 1 cikm2008.org 2 3 4 5 cikmconference.org 6 … r = 2 d = 3 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

64
**User Browsing Model [Dupret+08]**

The examine-next probability depends on both the preceding clicked position r, and the distance to this position d. Users would lose patience when they browse through without issuing a click. The probability monotonically drops as d increases and r remains the same. 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

65
**User Browsing Model [Dupret+08]**

The examine-next probability depends on both the preceding clicked position r, and the distance to this position d. P(Ei=1|C1:i-1)= βri,di 55 parameters are needed for top-10 positions (0≤r<r+d≤10). Cascade hypothesis is not assumed. where ri = max{j| j <i , Cj=1}, di = i - ri 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

66
**User Browsing Model [Dupret+08]**

The full picture: 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

67
**Outline Designing click models Basic user hypotheses**

Introduction Designing click models Basic user hypotheses Modeling the first click Extending to multiple clicks Summary of model design Bayesian click models Selected topics on click models Conclusion 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

68
**Summary of Model Design**

Probability of examine the first URL Model P(E1) Cascade 1 DCM CCM 1* DBN UBM β0,1 * Footnote: it is flexible to add another parameter to specify this probability. 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

69
**Summary of Model Design**

Probability of click upon examination Model P(Ci=1|Ei=1) Cascade rdi DCM CCM rdi* DBN adi UBM *Footnote: the mean of the relevance distribution, detailed in the next part 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

70
**Summary of Model Design**

Probability of examine-next w/o a click Model P(Ei+1=1|Ei=1,Ci=0) Cascade 1 DCM CCM α1 DBN γ UBM βri+1,di+1 * *Footnote: the probability does not depend on Ei 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

71
**Summary of Model Design**

Probability of examine-next after a click Model P(Ei+1=1|Ei=1,Ci=1) Cascade -- DCM αi CCM α2(1-rdi) + α3rdi DBN γ(1-sdi) UBM βi,1 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

72
**Summary of Model Design**

Probability of examine-next after a click Model P(Ei+1=1|Ei=1,Ci=1) Cascade -- DCM αi CCM α2(1-rdi) + α3rdi DBN γ(1-sdi) UBM βi,1 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

73
**Summary of Model Design**

Size of parameter sets Model # of global params Cascade DCM 9 CCM 3 DBN 1 UBM 55 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

74
**Summary of Model Design**

Inference and estimation algorithms Model Single-Pass Details DCM Maximizing a lower bound of LL, fastest CCM No iteration needed, thanks to the Bayesian framework DBN EM-based, iterative algorithms UBM EM-based, usually takes ~30 iterations to converge 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

75
**Summary of Model Design**

Inference and estimation algorithms Model Single-Pass Details DCM Maximizing a lower bound of LL, fastest CCM No iteration needed, thanks to the Bayesian framework DBN EM-based, iterative algorithms UBM EM-based, usually takes ~30 iterations to converge 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

76
**Outline Bayesian click models Bayesian framework and the rationale**

Introduction Designing click models Bayesian click models Bayesian framework and the rationale Bayesian Browsing Model: a case study Click Chain Model in a nutshell Selected topics on click models Conclusion 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

77
**Primer: Coin-Toss Example**

1 Prior Posterior “probability” of p(H) p(H)=0.8 p(H) p(H) Bayesian Frequentist 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

78
**Primer: Coin-Toss Example**

Posterior Prior Density Function (not normalized) x x x x3(1-x) x4(1-x) 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

79
**Primer: Coin-Toss Example**

Posterior Prior Density Function (not normalized) x1(1-x) x2(1-x) x3(1-x) x3(1-x) x4(1-x)1 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

80
**Primer: Coin-Toss Example**

The graphical model for coin-toss X C1 C2 C3 C4 C5 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

81
**Primer: Coin-Toss Example**

The graphical model for coin-toss X C1 C2 C3 C4 C5 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

82
**Primer: Click Data Example**

x1 (1-x)0 (1-0.6x)0 (1+0.3x)1 (1-0.5x)0 (1-0.2x)0 … x1 (1-x)1 (1-0.6x)0 (1+0.3x)1 (1-0.5x)0 (1-0.2x)0 … x2 (1-x)1 (1-0.6x)0 (1+0.3x)2 (1-0.5x)0 (1-0.2x)0 … x3 (1-x)1 (1-0.6x)1 (1+0.3x)2 (1-0.5x)0 (1-0.2x)0 … x3 (1-x)1 (1-0.6x)1 (1+0.3x)2 (1-0.5x)1 (1-0.2x)0 … Prior Density Function (not normalized) 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

83
**Overview Representation of relevance**

A probability distribution on [0,1] for each (query, URL) pair The density function is in a polynomial form over a small set of linear factors. The coefficients of such linear factors are shared between different (query, URL) pairs. x3 (1-1x)1 (1-0.6x)1 (1+0.3x)2 (1-0.5x)1 (1-0.2x)0 … 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

84
Overview Inference: Go over each query session once, update the exponents for corresponding (query, URL) pair impressed* Analytical or numerical integration may be needed to compute the normalization constant. *Footnote: by virtue of the Bayes theorem and conditional independence relationship/assumption 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

85
**Overview Key problems: Which is the right factor to update?**

How to estimate all the coefficients? 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

86
**Why Bayesian? Modeling Benefits: Computational Benefits:**

Confidence for the URL relevance estimate Relative judgments: probability of URL i is more relevant to the query than URL j Easy to interpret: coefficients in linear factors reflect position-bias and user browsing patterns Computational Benefits: Single-pass, linear algorithms; no iterations Paralleled version is easy to implement 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

87
**Outline Bayesian click models Bayesian framework and the rationale**

Introduction Designing click models Bayesian click models Bayesian framework and the rationale Bayesian Browsing Model: a case study Click Chain Model in a nutshell Selected topics on click models Conclusion 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

88
**Variable Definition For a specific query session, let**

where 1 ≤ i ≤ M=10. S1 S2 S3 SM … E1 E2 E3 EM … C1 C2 C3 CM … 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

89
**Graphical Illustration**

Relevance S1 S2 S3 SM … Examination E1 E2 E3 EM … Click C1 C2 C3 CM … 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

90
**Inference Algorithms Compute the posterior distribution**

Details Inference Algorithms Compute the posterior distribution Conditional independence relationship induced from the graphical model How many times URLj was not clicked when it is at position (r + d) with the preceding click at position r How many times the URL j was clicked 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

91
A Toy Example Only top M=3 positions are shown, 3 query sessions and 4 distinct URLs. 4 1 3 2 Position Query Session 3 Query Session 2 Query Session 1 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

92
**A Toy Example Initialize M(M+1)/2+1 counts for each URL 4 Clicks**

r=0 d=1 r=0 d=2 r=0 d=3 r=1 d=1 r=1 d=2 r=2 d=1 4 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

93
**A Toy Example Update counts for URL 4 If not impressed, do nothing;**

If clicked, increment “clicks” by 1; Otherwise, locate the right r and d to increment. URL Clicks r=0 d=1 r=0 d=2 r=0 d=3 r=1 d=1 r=1 d=2 r=2 d=1 4 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

94
**A Toy Example Update counts for URL 4 If not impressed, do nothing;**

If clicked, increment “clicks” by 1; Otherwise, locate the right r and d to increment. URL Clicks r=0 d=1 r=0 d=2 r=0 d=3 r=1 d=1 r=1 d=2 r=2 d=1 4 1 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

95
**A Toy Example Update counts for URL 4 If not impressed, do nothing;**

If clicked, increment “clicks” by 1; Otherwise, locate the right r and d to increment. URL Clicks r=0 d=1 r=0 d=2 r=0 d=3 r=1 d=1 r=1 d=2 r=2 d=1 4 1 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

96
**A Toy Example The posterior for URL 4 Interpretation:**

The larger the probability of examination, the stronger the penalty for a non-click. URL Clicks r=0 d=1 r=0 d=2 r=0 d=3 r=1 d=1 r=1 d=2 r=2 d=1 4 1 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

97
Parameter Estimation Keep 2 counts for each parameter (one for click, and the other one for non-click) Parameter Click Non-click Non-Click β0,1 β1,1 β0,2 β1,2 β0,3 β2,1 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

98
Parameter Estimation For each position in a query session, locate the right r and d to increment. Parameter Click Non-click Non-Click β0,1 1 β1,1 β0,2 β1,2 β0,3 β2,1 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

99
Parameter Estimation For each position in a query session, locate the right r and d to increment. Parameter Click Non-click Non-Click β0,1 1 β1,1 β0,2 β1,2 β0,3 β2,1 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

100
Parameter Estimation For each position in a query session, locate the right r and d to increment. Parameter Click Non-click Non-Click β0,1 1 2 β1,1 β0,2 β1,2 β0,3 β2,1 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

101
**Parameter Estimation Maximum-Likelihood Estimate: β0,1 1 2 β1,1 β0,2**

Click Non-click Non-Click β0,1 1 2 β1,1 β0,2 β1,2 β0,3 β2,1 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

102
**Algorithm Complexities**

Details Algorithm Complexities Let Initializing and updating the counts: Time: Space: Linear to the size of the click log Almost constant storage required 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

103
**Algorithm Complexities**

Details Algorithm Complexities Let Initializing and updating the counts: Time: Space: Computing relevance scores using numerical integration with B bins: 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

104
**Summary of Algorithms Step 1: initialize counting statistics;**

Step 2: scan through the click log once and update the counts for both inference and estimation Step 3: compute parameter values; Step 4: use numerical integration to obtain relevance scores. Step 2 also applies for (linear) incremental computation! 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

105
**Outline Bayesian click models Bayesian framework and the rationale**

Introduction Designing click models Bayesian click models Bayesian framework and the rationale Bayesian Browsing Model: a case study Click Chain Model in a nutshell Selected topics on click models Conclusion 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

106
**CCM Recap The user behavior model: 3/27/2017**

CIKM'09 Tutorial, Hong Kong, China

107
**CCM Recap Graphical model: Relevance … Examination Click S1 S2 S3 E1**

SM … E1 E2 E3 EM C1 C2 C3 CM 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

108
Details CCM Inference 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

109
**Comparing with UBM Number of user behavior parameters**

Number of distinct factors for (query, URL) Number of counts needed for parameters CCM UBM 3 55 CCM UBM 22 56 CCM UBM 5 110 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

110
**Outline Selected topics on click models**

Introduction Designing click models Bayesian click models Selected topics on click models Scaling click models for Petabyte-scale data Click model evaluation Tailoring user goals to click models Conclusion 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

111
**Petabyte-Scale Data Data collected in 8 weeks**

Job k includes data between week 1 and k Both time and space costs are prohibitive for a single node. 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

112
**Introducing Map-Reduce [Dean+04]**

A Simple Task: counting # impression for each (query, URL) pair 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

113
**Introducing Map-Reduce [Dean+04]**

Machine #1 Machine #2 Machine #3 Machine #4 Extent Extent Extent Extent GetPairs GetPairs GetPairs GetPairs Map Map Map Map Sort Sort Sort Sort Count Count Count Count Output

114
**Introducing Map-Reduce [Dean+04]**

Machine #1 Machine #2 Machine #3 Machine #4 Extent Extent Extent Extent GetPairs GetPairs GetPairs GetPairs Map Map Map Map Sort “Map” puts all of the same Pairs onto one machine. This allows you to group by various fields in subsequent processes. Sort Sort Sort Count Count Count Count Output

115
**Introducing Map-Reduce [Dean+04]**

A Simple Task: counting # impression for each (query, URL) pair Map = Bucket: the intermediate key is (query, URL) pair 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

116
**Introducing Map-Reduce [Dean+04]**

Machine #1 Machine #2 Machine #3 Machine #4 Extent Extent Extent Extent GetPairs GetPairs GetPairs GetPairs “Count” carries out standard increment-by-1 over each distinct Pair. Map Map Map Map “Count” REDUCES the amount of data since each Pair has only one output value Sort Sort Sort Sort Count Count Count Count Output

117
**Introducing Map-Reduce [Dean+04]**

A Simple Task: counting # impression for each (query, URL) pair Map = Bucket: the intermediate key is (query, URL) pair Reduce = Count: it accepts a list of (key, value) tuple, and outputs the final result for each distinct key 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

118
**Introducing Map-Reduce [Dean+04]**

Machine #1 Machine #2 Machine #3 Machine #4 Extent Extent Extent Extent GetPairs GetPairs GetPairs GetPairs MAP Map Map Map Map Sort Sort Sort Sort REDUCE Count Count Count Count Output

119
**MapReduce for BBM 0 for clicks 3 2 5 1 4 6 3/27/2017**

3/27/2017 CIKM'09 Tutorial, Hong Kong, China

120
**MapReduce for BBM Map: scan the click log**

Intermediate key: (query, URL) Value: the index of linear factors (0~55 for top-10 positions) Reduce: scan the list of (key, value) The key indicates which exponent vector to update The value indicates the index of the element in the exponent vector to increment 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

121
**Empirical Results Linearly increasing computation load**

Near-constant elapsed time 3 hours 265 TB log data 1.15 billion (query, url) pairs Single machine computation load Elapse time on SCOPE 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

122
**Outline Selected topics on click models**

Introduction Designing click models Bayesian click models Selected topics on click models Scaling click models for Petabyte-scale data Click model evaluation Tailoring user goals to click models Conclusion 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

123
**Evaluation Overview - Training**

Impression Data Click Data 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

124
**Evaluation Overview - Training**

Impression Data Click Data Global Parameters M=10 Relevance Scores 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

125
**Evaluation Overview - Test**

New Impression Vector from an Existing Query Relevance Predicted Examination Predicted Clicks Global params 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

126
**Data Preprocessing [Guo+09a, 09b]**

Data are collected from a commercial search engine after query term normalization and spam removal. For each query term, split query sessions evenly into training and test sets according to the timestamp. Top frequent/infrequent query terms are removed. 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

127
**Evaluation Metrics Most popular metrics:**

Average test data log-likelihood (LL) (probability of accurately predicting the click vector, 2^10 possibilities) [Guo+09a, Guo+09b, Liu+09a, Zhu+10] Perplexity of prediction for each position (2^{average entropy} of click/no-click binary prediction for each position independently) [Dupret+08, Guo+09a, Guo+09b, Zhu+10] 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

128
**Evaluation Metrics Other Metrics:**

Click-through-rate (CTR) prediction (Especially for predicting [Chapelle+09, Zhu+10] Predicting first/last clicked positions [Guo+09a, Guo+09b] Position-bias sanity check (plot the click rate curve for top-10 positions v.s. the ground truth) [Guo+09a, Guo+09b] 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

129
**Results – Log Likelihood [Guo+09b]**

Average Log-likelihood Random guess: log(2-10) = -3.01 Optimal value: 0 Model CCM UBM DCM LL -1.171 -1.264 -1.302 Improve-ment Ratio 9.7% 14% 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

130
**Results – Log Likelihood [Guo+09b]**

Better Worse 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

131
**Results – Log Likelihood [Liu+09a]**

Better Worse 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

132
**Results – Perplexity [Guo+09b]**

Average Perplexity over top 10 positions Random guess: 2 Optimal value: 1 Model CCM UBM DCM Perplexity 1.1577 1.1590 Improve-ment Ratio 7.5% 8.3% 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

133
**Results – Perplexity [Guo+09b]**

Worse Better 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

134
**Examine/Click Position-Bias [Guo+09b]**

3/27/2017 CIKM'09 Tutorial, Hong Kong, China

135
Results – Efficiency For 1M query sessions, the estimated time in seconds: * Time for CCM and BBM includes computing posterior mean and variance using numerical integration w/ 100 bins. ** UBM converges in 34 iterations. DCM CCM* BBM* UBM** 80 150 165 5,000 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

136
**Outline Selected topics on click models**

Introduction Designing click models Bayesian click models Selected topics on click models Scaling click models for Petabyte-scale data Click model evaluation Tailoring user goals to click models Conclusion 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

137
**User Goals in Web Search**

Queries could be categorized into 2 sets: Navigational: to find the link to an existing website, e.g., bing; Informational: more exploration, multiple clicks may arise, e.g., iron man. 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

138
**Fitting Multiple Click Models**

Different user goals result in different browsing and click patterns. The straightforward mixture-modeling approach is not practical. [Dupret+08] Solution: Classify query terms a priori based on user goals. Fitting and learning 2 sets of model parameters for navigational and informational queries. 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

139
**Determining User Goals [Lee+05]**

Two-way classification for query terms based on click data using… Median position of click distribution Mean position of click distribution Average # clicks per query session … Pick the one which has best click prediction If a position receives 50% of the click, then navigational, else informational 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

140
**Empirical Results [Guo+09c]**

Improvement of click prediction for DCM: Log-Likelihood: 4.0% Perplexity: 1.3% Examination/Click position-bias: 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

141
**Outline Introduction Designing click models Bayesian click models**

Selected topics on click models Conclusion 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

142
Summary Click models A statistical tool to leverage valuable user implicit feedback in terabyte/petabyte search logs. Provide click prediction as well as relevance estimates. Application domains include learning to rank, measuring search performance, online advertising, user behavior analysis… 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

143
Summary Click models Different model designs reflect various assumption of user behaviors to explain the position-bias. The modeling choice may depend on the application scenario. 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

144
Summary Click models Efficient, single-pass, parallelizable algorithms are desired in real-world applications. Bayesian framework could be applied to click models for both modeling benefits and computational benefits. Click Chain Model and Bayesian Browsing Model represent state-of-the-art examples. 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

145
**Future Directions Bigger Context Richer inputs**

Query reformulations Personalization Richer inputs Universal search Diverse user feedback Click model v.s. Human judgments 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

146
References [Burges+05]: C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender. Learning to rank using gradient descent. ICML’05. [Chapelle+09]: O. Chapelle and Y. Zhang. A dynamic Bayesian network click model for web search ranking. WWW’09. [Craswell+08]: N. Craswell, O. Zoeter, M. Taylor, and B. Ramsey. An experimental comparison of click position-bias models. WSDM ’08. [Dean+04]: J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. OSDI’04. [Dupret+08]: G. Dupret and B. Piwowarski. A user browsing model to predict search engine click data from past observations. SIGIR’08. 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

147
References [Guo+09a]: F. Guo, C. Liu, and Y.-M. Wang. Efficient multiple-click models in web search. WSDM’09. [Guo+09b]: F. Guo, C. Liu, A. Kannan, T. Minka, M. Taylor, Y.-M. Wang, and C. Faloutsos. Click chain model in web search. WWW’09. [Guo+09c]: F. Guo, L. Li, and C. Faloutsos. Tailoring click models to user goals. WSCD’09. [Joachims02]: T. Joachims. Optimizing search engines using clickthrough data. KDD’02. [Joachims+07]: T. Joachims, L. Granka, B. Pan, H. Hembrooke, F. Radlinski, and G. Gay. Accurately interpreting clickthrough data as implicit feedback, ACM TOIS, 25(2), 2007. 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

148
References [Lee+05]: U. Lee, Z. Liu, and J. Cho. Automatic identification ofuser goals in web search. WWW’05. [Liu+09a]: C. Liu, F. Guo, and C. Faloutsos. BBM: Deriving click models from petabyte-scale data. KDD’09. [Liu+09b]: C. Liu, M. Li, and Y.-M. Wang. Post-rank reordering: resolving preference misalignments between search engines and end users. CIKM’09. [Richardson+07]: M. Richardson, E. Dominowska, and R. Ragno. Predicting clicks: estimating the click-through rate for new ads. WWW’07. [Zhu+10]: Z. Zhu, W. Chen, T. Minka, C. Zhu and Z. Chen. A novel click model and its applications to online advertising. To appear in WSDM’10. 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

149
**Acknowledgements Nick Craswell Christos Faloutsos Li-Wei He Tom Minka**

MSR, Cambridge Carnegie Mellon University MSR, ISRC-Redmond Tom Minka Anitha Kannan MSR, Cambridge MSR, Search Lab 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

150
**Acknowledgements Yi-Min Wang Mike Taylor Ethan Tu MSR, ISRC-Redmond**

MSR, Cambridge MSR, ISRC-Redmond 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

151
End 3/27/2017 CIKM'09 Tutorial, Hong Kong, China

Similar presentations

OK

Modeling (Chap. 2) Modern Information Retrieval Spring 2000.

Modeling (Chap. 2) Modern Information Retrieval Spring 2000.

© 2018 SlidePlayer.com Inc.

All rights reserved.

To make this website work, we log user data and share it with processors. To use this website, you must agree to our Privacy Policy, including cookie policy.

Ads by Google