A Case Study of Behavior-driven Conjoint Analysis on Yahoo A Case Study of Behavior-driven Conjoint Analysis on Yahoo! Front Page Today Module Wei Chu, Seung-Taek Park, Todd Beaupre, Nitin Motgi, Amit Phadke, Seinjuti Chakraborty, Joe Zachariah Yahoo! Good Morning, Everyone! Since the authors of this paper couldn’t make the trip, I will present this paper on behalf of them. Please bear with me. This work carried out conjoint analysis on the application of Yahoo! Front Page Today Module. Based on click through stream we observed, conjoint analysis is carried out to identify users’ preference and then cluster users having similar preferences into segments. Presenter: Lei Guo Yahoo! KDD 2009

Outline Motivation and problem setting Our methodology Conjoint analysis Tensor segmentation Experimental results Offline analysis Segment characteristics Online bucket test Conclusions and future work I will first introduce the application, Yahoo! Front Page Today Module and the problem setting. Then I describe our methodology. In our application, we can collect millions of samples in short time, while traditional conjoint analysis coupled with sophisticated Monte Carlo simulation becomes computationally prohibitive. We report a conjoint analysis technique for large-scale data analysis, that is capable of identifying users' diversified preferences from millions of click/view events and building predictive models to classify new users into segments of distinct behavior pattern. We call it tensor segmentation. I will also report experimental results, including offline analysis, segment characteristics and online bucket test, along with conclusions and future directions. KDD 2009

Yahoo! Front Page This is a snapshot of Yahoo! Front Page, one of the most popular pages on the Internet. We call the module, marked by red rectangle, Today Module. This is the most prominent panel on Front Page. Let’s zoom in to see more details. KDD 2009

Today Module The default ``Featured'' tab in Today Module highlights one of four high-quality articles selected from a daily-refreshed article pool created by human editors. There are four articles at footer positions, indexed by F1, F2, F3 and F4 respectively. At default, the article at F1 is highlighted at the story position. It is featured by a large picture, a title and a short summary along with related links. A user can click on the highlighted article at the story position to read more details if the user is interested in the article. The event is recorded as a ``story click''. One of our goals is to increase user activities, measured by overall Click Through Rate. To draw visitors' attention and increase the number of clicks, we would like to rank available articles according to visitors' interests, and to highlight the most attractive article at the Story position. At default, the article at F1 is highlighted at the Story position. Articles are selected from a hourly-refreshed article pool. GOAL: select the most attractive article for the Story position to draw users’ attention and then increase users’ retention. KDD 2009

Conjoint Analysis Current Strategy: (Agarwal, et al. NIPS 2008) Estimated Most Popular (EMP) estimate CTR of available articles present the article of the highest CTR at F1 Conjoint Analysis: personalized service at segment level determine user clusters by conjoint analysis present the articles with the highest SEGMENTAL CTR to user segments respectively Our current strategy is to estimate CTR of all available articles, and present the article of the highest CTR at F1 position. We call it ``one-size-fits-all’’ Estimated Most Popular approach. In this work we would like to further boost overall CTR by launching a partially personalized service. User segments determined by conjoint analysis will be served with different content according to segmental interests. Articles with the highest segmental CTR will be served to user segments respectively. In addition to optimizing overall CTR, another benefit of this study is to understand users' intention and behavior to some extend for user targeting and editorial content management. KDD 2009

Data Collection Content features User profiles Click through stream topic categories, sub-topics, URL resources, etc. User profiles age, gender, residential location, Yahoo! property usage, etc. Click through stream view only or story click for each pair of a visitor and an article a large amount of view events are false non-click events We collected three sets of data, including content features, user profiles and interactive data between users and articles. Each article is summarized by a set of features, such as topic categories, sub-topics, URL resources. Each visitor is profiled by a set of attributes as well, for example, age, gender, residential location, Yahoo! property usage. The interaction data consists of two types of actions: view only or story click for each pair of a visitor and an article. KDD 2009

Tensor Segmentation Tensor Indicator Logistic Regression KDD 2009 affinity sports finance age 50 0.5 0.8 age 20 0.9 0.2 male 0.6 Tensor Indicator the b-th feature of user the a-th feature of item affinity between and 1.5 Logistic Regression 0.7 We first define an indicator as a parametric function of all combinations of both article features and user attributes. The weight w_ab means the preference of a user having feature b on an item having feature a. Here is an example. Suppose we have a weight table in hand. According to the table, the indicator of a male user at age 20 on a sport article would be 1.5, while his indicator on a finance article is 0.5. Similarly we can compute the scores for a male user at age 50. The weight values in the table can be learned from our historical observations. We employ logistic function to relate the indicator with click or not actions we observed. After optimization, we can then obtain the weights. 1.1 1.3 : action of user on item click or view only {+1,-1} KDD 2009

Tensor Segmentation User preference on item feature Clustering each user is represented by i.e. their preferences on item features apply clustering techniques, e.g. K-means, to classify user segments the cluster membership is determined by the shortest distance between to cluster centroids With the optimal weights in hand, we can compute preferences on item features for each user. In other words, each user can be represented by a vector of preferences on item features. Then the indicator becomes a simple dot product, between user preference vector and item feature vector. We apply clustering techniques, such as K-Means, on the new user representation. The number of clusters can be determined by validation in offline analysis. For an existing or new user, we can compute their preference vectors using their original features and the weight matrix. Then segment membership can be determined by the shortest distance between the preference vector and the centroids of clusters KDD 2009

Experiments Data for offline analysis Offline metric Online metric July random bucket data for training Sept random bucket data for test Offline metric Number of story clicks on the top position in our ranking list (sorted by segmented CTR) Online metric CTR ( # story clicks / # views ) We collected events from a random bucket in July, 2008 for training and validation. We split the July random bucket data by a time stamp threshold for training and validation. We also collected events from a random bucket in September 2008 for test. In the random bucket, articles are randomly selected from a content pool to serve users. For each user in test, we computed her membership first, and sorted all available articles in descending order according to their CTR in the test user's segment. On click events, we measured the rank position of the article being clicked by the user. The performance metric we used in offline analysis is the number of clicks in top position in our ranking list. A good predictive model should have more clicks on top-ranked positions. KDD 2009

Cluster Number We varied the segment number from 1 to 20, and presented the offline performance metric on July validation data in this graph. We observed the best validation performance at 8 clusters, but the difference compared with that at 5 clusters is not statistically significant. Thus we selected 5 clusters in our study. KDD 2009

Offline Performance KDD 2009 To verify the stability of the clusters we found in the July data, we further tested on the random bucket data collected in September 2008. The EMP approach was utilized as the baseline model. We also implemented two demographic segmentations for comparison purpose. Gender segments: 3 clusters defined by users' gender, male, female, unknown. AgeGender segments: 11 clusters defined as gender and age range combination. We computed the lifts over the EMP baseline approach and presented the results of the top 4 rank positions in this graph. All segmentation approaches outperform the baseline `one-size-fit-all’ model. Tensor segmentation with 5 clusters consistently gives more lift than Gender and AgeGender at all the top 4 positions. KDD 2009

Cluster Size We collected some characteristics in the 5 segments we discovered. On the September data, we identified cluster membership for all users, and plotted the population distribution in the 5 segments in this graph. The largest cluster takes 32% of users, while the smallest cluster contains 10% of users. Note that we didn’t constrain the segment size in clustering. KDD 2009

Demographic Distribution We further present the user composition of the 5 clusters with popular demographic categories in this Hinton graph. We found that Cluster c1 is of mostly female users under age 34; Cluster c2 is of mostly male users under age 44; Cluster c3 is for female users above age 30; Cluster c4 is of mainly male users above age 35; Cluster c5 is predominantly non-U.S. users. Note that cluster membership is not solely determined by demographic information, though the demographic information gives a very strong signal. KDD 2009

User Preferences in Segments We utilized the centroid of each cluster as a representative to illustrate users' preferences on article topics. The gray level indicates users' preference, from like (white) to dislike (black). We found some favorite and unattractive topics by comparing the representatives' scores across segments. For example, Users in c2 like Sports, Music, don’t like Food. Users in c3 like TV, OMG, don’t like Sport. KDD 2009

Segment Traffic Pattern Morning old users Late afternoon, young users One interesting finding in our conjoint analysis is that visiting patterns of some segments are quite different from the others. In this graph, we plot fraction of visitors from the 5 segments in 7 days. We observed that more older users (c3 and c4) in the morning, while more younger users (c1 and c2) in the late afternoon. Most users around the midnight are international users (c5). Portion of traffic from older male users significantly decreases during weekend/holiday. This finding suggests some tips for content management, such as programming more articles related with News, Politics and Finance in the morning of weekdays, and more articles relevant to international users around midnight. Midnight, intl users KDD 2009

Online Bucket Test We also launched a bucket test in December 2008, starting from8:00am 12 December to 0:00am 15 December. Three segmentation methods, Gender, AgeGender and Tensor-5, were implemented in our production. Each of the three schemes and the control (EMP) bucket served about several million page views respectively. We computed the story CTR and reported the corresponding lifts over the EMP control bucket for the three segmentation schemes in this Table. The tensor segmentation with 5 clusters yields the most lift in the bucket test. The observations in the online bucket test are consistent with our results in offline analysis.. 1. From 8:00am 12 Dec. to 0:00am 15 Dec. 2008 2. Baseline control bucket : Estimated Most Popular (EMP) 3. Each bucket served about several million page views respectively 4. We report lift over the story CTR in EMP bucket KDD 2009

Conclusions Execute conjoint analysis on large scale click through stream Discover different characteristics and traffic patterns in segments User intention at segment level could help editors on article content management In future work, allow for multiple membership and then exploit a weighted sum of several segmental preferences. In this study, we executed conjoint analysis on a large-scale click through stream of Yahoo! Front Page Today Module. We validated the segments discovered in conjoint analysis by conducting offline and online tests. We analyzed characteristics of users in segments and also found different visiting patterns of segments. User intention at segment level could help editors on article content management. As future work, we would like to exploit other clustering techniques, e.g. Gaussian mixture models that allow for multiple membership. KDD 2009

Acknowledgment Raghu Ramakrishnan Scott Roy Deepak Agarwal Bee-Chung Chen Pradheep Elango Ajoy Sojan We would like to thank our colleague for many discussions and helps on data collection, including Raghu, Scott, Deepak, Bee-Chung, Pradheep and Ajoy. Thanks for your attention. Questions? KDD 2009