1. A Case Study of Behavior-driven Conjoint Analysis on Yahoo
A Case Study of Behavior-driven Conjoint Analysis on Yahoo! Front Page Today Module
Wei Chu, Seung-Taek Park, Todd Beaupre, Nitin Motgi,
Amit Phadke, Seinjuti Chakraborty, Joe Zachariah
Yahoo!
Good Morning, Everyone!
Since the authors of this paper couldn’t make the trip, I will present this paper on behalf of them. Please bear with me.
This work carried out conjoint analysis on the application of Yahoo! Front Page Today Module.
Based on click through stream we observed, conjoint analysis is carried out to identify users’ preference and then cluster users having similar preferences into segments.
Presenter: Lei Guo
Yahoo!
KDD 2009

2. Outline Motivation and problem setting Our methodology
Conjoint analysis
Tensor segmentation
Experimental results
Offline analysis
Segment characteristics
Online bucket test
Conclusions and future work
I will first introduce the application, Yahoo! Front Page Today Module and the problem setting.
Then I describe our methodology.
In our application, we can collect millions of samples in short time, while traditional conjoint analysis coupled with sophisticated Monte Carlo simulation becomes computationally prohibitive.
We report a conjoint analysis technique for large-scale data analysis, that is capable of identifying users' diversified preferences from millions of click/view events and building predictive models to classify new users into segments of distinct behavior pattern.
We call it tensor segmentation.
I will also report experimental results, including offline analysis, segment characteristics and online bucket test, along with conclusions and future directions.
KDD 2009

3. Yahoo! Front Page
This is a snapshot of Yahoo! Front Page, one of the most popular pages on the Internet.
We call the module, marked by red rectangle, Today Module.
This is the most prominent panel on Front Page.
Let’s zoom in to see more details.
KDD 2009

4. Today Module
The default ``Featured'' tab in Today Module highlights one of four high-quality articles selected from a daily-refreshed article pool created by human
editors.
There are four articles at footer positions, indexed by F1, F2, F3 and F4 respectively. At default, the article at F1 is highlighted at the story position. It is featured by a large picture, a title and a short summary along with related links.
A user can click on the highlighted article at the story position to read more details if the user is interested in the article. The event is recorded as a ``story click''.
One of our goals is to increase user activities, measured by overall Click Through Rate.
To draw visitors' attention and increase the number of clicks, we would like to rank available articles according to visitors' interests, and to highlight the most attractive article at the Story position.
At default, the article at F1 is highlighted at the Story position.
Articles are selected from a hourly-refreshed article pool.
GOAL: select the most attractive article for the Story position to draw users’ attention and then increase users’ retention.
KDD 2009

5. Conjoint Analysis Current Strategy: (Agarwal, et al. NIPS 2008)
Estimated Most Popular (EMP)
estimate CTR of available articles
present the article of the highest CTR at F1
Conjoint Analysis:
personalized service at segment level
determine user clusters by conjoint analysis
present the articles with the highest SEGMENTAL CTR to user segments respectively
Our current strategy is to estimate CTR of all available articles, and present the article of the highest CTR at F1 position. We call it ``one-size-fits-all’’ Estimated Most Popular approach.
In this work we would like to further boost overall CTR by launching a partially personalized service.
User segments determined by conjoint analysis will be served with different content according to segmental interests. Articles with the highest segmental CTR will be served to user segments respectively.
In addition to optimizing overall CTR, another benefit of this study is to understand users' intention and behavior to some extend for user targeting and editorial content management.
KDD 2009

6. Data Collection Content features User profiles Click through stream
topic categories, sub-topics, URL resources, etc.
User profiles
age, gender, residential location, Yahoo! property usage, etc.
Click through stream
view only or story click for each pair of a visitor and an article
a large amount of view events are false non-click events
We collected three sets of data, including content features, user profiles and interactive data between users and articles.
Each article is summarized by a set of features, such as topic categories, sub-topics, URL resources.
Each visitor is profiled by a set of attributes as well, for example, age, gender, residential location, Yahoo! property usage.
The interaction data consists of two types of actions: view only or story click for each pair of a visitor and an article.
KDD 2009

7. Tensor Segmentation Tensor Indicator Logistic Regression KDD 2009
affinity
sports
finance
age 50
0.5
0.8
age 20
0.9
0.2
male
0.6
Tensor Indicator
the b-th feature of user
the a-th feature of item
affinity between and
1.5
Logistic Regression
0.7
We first define an indicator as a parametric function of all combinations of both article features and user attributes.
The weight w_ab means the preference of a user having feature b on an item having feature a.
Here is an example. Suppose we have a weight table in hand.
According to the table, the indicator of a male user at age 20 on a sport article would be 1.5, while his indicator on a finance article is 0.5. Similarly we can compute the scores for a male user at age 50.
The weight values in the table can be learned from our historical observations.
We employ logistic function to relate the indicator with click or not actions we observed. After optimization, we can then obtain the weights.
1.1
1.3
: action of user on item
click or view only {+1,-1}
KDD 2009

8. Tensor Segmentation User preference on item feature Clustering
each user is represented by
i.e. their preferences on item features
apply clustering techniques, e.g. K-means, to classify user segments
the cluster membership is determined by the shortest distance between to cluster centroids
With the optimal weights in hand, we can compute preferences on item features for each user.
In other words, each user can be represented by a vector of preferences on item features. Then the indicator becomes a simple dot product, between user preference vector and item feature vector.
We apply clustering techniques, such as K-Means, on the new user representation.
The number of clusters can be determined by validation in offline analysis.
For an existing or new user, we can compute their preference vectors using their original features and the weight matrix.
Then segment membership can be determined by the shortest distance between the preference vector and the centroids of clusters
KDD 2009

9. Experiments Data for offline analysis Offline metric Online metric
July random bucket data for training
Sept random bucket data for test
Offline metric
Number of story clicks on the top position in our ranking list (sorted by segmented CTR)
Online metric
CTR ( # story clicks / # views )
We collected events from a random bucket in July, 2008 for training and validation. We split the July random bucket data by a time stamp threshold for training and validation.
We also collected events from a random bucket in September 2008 for test.
In the random bucket, articles are randomly selected from a content pool to serve users.
For each user in test, we computed her membership first, and sorted all available articles in descending order according to their CTR in the test user's segment.
On click events, we measured the rank position of the article being clicked by the user.
The performance metric we used in offline analysis is the number of clicks in top position in our ranking list.
A good predictive model should have more clicks on top-ranked positions.
KDD 2009

10. Cluster Number
We varied the segment number from 1 to 20, and presented the offline performance metric on July validation data in this graph.
We observed the best validation performance at 8 clusters, but the difference compared with that at 5 clusters is not statistically significant.
Thus we selected 5 clusters in our study.
KDD 2009

11. Offline Performance KDD 2009
To verify the stability of the clusters we found in the July data, we further tested on the random bucket data collected in September 2008.
The EMP approach was utilized as the baseline model.
We also implemented two demographic segmentations for comparison purpose.
Gender segments: 3 clusters defined by users' gender, male, female, unknown.
AgeGender segments: 11 clusters defined as gender and age range combination.
We computed the lifts over the EMP baseline approach and presented the results of the top 4 rank positions in this graph.
All segmentation approaches outperform the baseline `one-size-fit-all’ model.
Tensor segmentation with 5 clusters consistently gives more lift than Gender and
AgeGender at all the top 4 positions.
KDD 2009

12. Cluster Size
We collected some characteristics in the 5 segments we discovered.
On the September data, we identified cluster membership for all
users, and plotted the population distribution in the 5 segments in this graph.
The largest cluster takes 32% of users, while the smallest cluster contains
10% of users.
Note that we didn’t constrain the segment size in clustering.
KDD 2009

13. Demographic Distribution
We further present the user composition of the 5 clusters with popular demographic categories in this Hinton graph.
We found that
Cluster c1 is of mostly female users under age 34;
Cluster c2 is of mostly male users under age 44;
Cluster c3 is for female users above age 30;
Cluster c4 is of mainly male users above age 35;
Cluster c5 is predominantly non-U.S. users.
Note that cluster membership is not solely determined by demographic information, though the demographic information gives a very strong signal.
KDD 2009

14. User Preferences in Segments
We utilized the centroid of each cluster as a representative to illustrate users' preferences on article topics.
The gray level indicates users' preference, from like (white) to dislike (black).
We found some favorite and unattractive topics by comparing the representatives' scores across segments.
For example,
Users in c2 like Sports, Music, don’t like Food.
Users in c3 like TV, OMG, don’t like Sport.
KDD 2009

15. Segment Traffic Pattern
Morning
old users
Late afternoon, young users
One interesting finding in our conjoint analysis is that visiting patterns of some segments are quite different from the others.
In this graph, we plot fraction of visitors from the 5 segments in 7 days.
We observed that more older users (c3 and c4) in the morning, while more younger users (c1 and c2) in the late afternoon.
Most users around the midnight are international users (c5).
Portion of traffic from older male users significantly decreases during weekend/holiday.
This finding suggests some tips for content management, such as programming more articles related with News, Politics and Finance in the morning of weekdays, and more articles relevant to international users around midnight.
Midnight, intl users
KDD 2009

16. Online Bucket Test
We also launched a bucket test in December 2008, starting from8:00am 12 December to 0:00am 15 December.
Three segmentation methods, Gender, AgeGender and Tensor-5, were implemented in our production.
Each of the three schemes and the control (EMP) bucket served about several million page views respectively.
We computed the story CTR and reported the corresponding lifts over the EMP control bucket for the three segmentation schemes in this Table.
The tensor segmentation with 5 clusters yields the most lift in the bucket test.
The observations in the online bucket test are consistent with our results in
offline analysis..
1. From 8:00am 12 Dec. to 0:00am 15 Dec. 2008
2. Baseline control bucket : Estimated Most Popular (EMP)
3. Each bucket served about several million page views respectively
4. We report lift over the story CTR in EMP bucket
KDD 2009

17. Conclusions
Execute conjoint analysis on large scale click through stream
Discover different characteristics and traffic patterns in segments
User intention at segment level could help editors on article content management
In future work, allow for multiple membership and then exploit a weighted sum of several segmental preferences.
In this study, we executed conjoint analysis on a large-scale click through stream of Yahoo! Front Page Today Module.
We validated the segments discovered in conjoint analysis by conducting offline and online tests.
We analyzed characteristics of users in segments and also found different visiting patterns of segments.
User intention at segment level could help editors on article content management.
As future work, we would like to exploit other clustering techniques, e.g. Gaussian mixture models that allow for multiple membership.
KDD 2009

18. Acknowledgment Raghu Ramakrishnan Scott Roy Deepak Agarwal
Bee-Chung Chen
Pradheep Elango
Ajoy Sojan
We would like to thank our colleague for many discussions and
helps on data collection, including Raghu, Scott, Deepak, Bee-Chung, Pradheep and Ajoy.
Thanks for your attention. Questions?
KDD 2009