1. Interpreting Advertiser Intent in Sponsored Search BHANU C VATTIKONDA, SANTHOSH KODIPAKA, HONGYAN ZHOU, VACHA DAVE, SAIKAT GUHA, ALEX C SNOEREN 1

2. Organic results Sponsored results Query 2

3. 3 women’s shoes Ad Keyword shoes Query

4. Huge and growing industry 20% 4

5. 5 What is today’s date in Japan Query Date Japan Ad Keyword

6. 6 Virgin River Utah Query Virgin Media Ad Keyword

7. 7 Virgin River Utah Query Virgin Media Ad Keyword We use organic search results for ad keyword

8. Capturing user and advertiser intents 8

9. Overview  Mismatching advertiser and user intent  Organic results are accurate  Ad keywords capture advertiser intent  Ad keyword is very brief  Use organic results for ad keyword to interpret advertiser intent 9

10. Relevance in Sponsored Search 10

11. Ad selection pipeline Ad Corpus (M’s of ads) User Query Quick look up “virgin media” matched to “virgin river Utah” Choose ads that provide most revenue Measure the relevance of each ad to user query Virgin media ad seen as relevant to “virgin river Utah” 11 Ad retrieval RelevanceAuction

12. Supervised machine learning Training Training set (query, ad, judgement) Trained Ranker Compute numerical features E.g., no. of words common in query and ad keyword (features, judgement) Ranker can compute relevance of (query, ad) pair 12

13. Signal sources 13 Virgin River Utah Query Virgin Media Ad Keyword

14. Signal sources 14 Virgin River Utah Query Virgin Media Ad Keyword Ad Creative

15. Signal sources 15 Virgin River Utah Query Virgin Media Ad Keyword Ad Creative Landing page

16. Features from signal sources  Query and ad creative  Query: “virgin river Utah”, ad title: “virgin media | virginmedia.com”  Word bigram overlap: 0  Query and landing page  Query: “virgin river utah”, landing page title: “Virgin Media – Cable Broadband, Digital TV”  Ordered word bigram overlap: 0  Query and ad keyword  Query: “virgin river Utah”, ad keyword: “virgin media”  Word unigram overlap: 0.5 16

17. Interpreting user and advertiser intent  Query and ad keyword are very short -- 2.5 words on average  Hard to determine user and advertiser intent  Query and ad keyword may mean same without overlap or vice versa  Sneakers vs shoes  Virgin river Utah vs virgin media 17 Use organic search results to boost query and ad keyword

18. Matching user and advertiser intents 18 50% overlap 25% overlap 3% overlap

19. Improving relevance ranker  Introduce features capturing overlap between user and advertiser intents  User intent captured using search results for user query  Advertiser intent captured using search results for ad keyword  75 Features introduced:  Overlap between elements of search results  Overlap between ad creative and search results for ad keyword 19

20. Evaluation 20

21. Measuring performance of ranker Trained Ranker Validation set (query, ad, judgement) (query, ad, relevance) Compare ranker result against human judgement Compute numerical features (query, ad, judgement) 21

22. Evaluating performance of ranker 22

23. Precision-Recall  Low precision results in irrelevant ads being shown to users  Bad user experience  Wasteful spend for advertisers  Low recall would lead to missed chances  Lost revenue opportunities for search engine  Lost targeting for advertisers  Lost desirable ads for users 23

24. Dataset and approach  Data from a large search engine  1.28M (query, ad) pairs of training data  320k (query, ad) pairs of validation data  Several hundred existing features  The ranker is trained on a combination of the features we introduce and existing production system features  Compare new ranker to current ranker 24

25. Query and ad matching  Query and ad matched using different match types chosen by advertiser  Exact match  Ad keyword: “shoes” and query: “shoes”  Broad match  Ad keyword: “shoes” and query: “sneakers”  Features we introduce capture similarity between query and ad keyword 25

26. Significant improvement in broad match 26 2.7% improvement in area under precision-recall curve

27. Summary  Mismatched user and advertiser intents leads to errors in sponsored ads  Interpreting ad keywords poses a challenge due to their brevity  Organic search results for the ad keyword capture advertiser intent  2.7% gain in area under precision-recall curve over production 27

28. Thanks 28

29. Thanks 29

30. Backup 30

31. Organic results vs ad results  Organic and ad result goals are different  Ads “related” to user query  Example:  User query: “prom dresses”  Ad for Limousine service targeting “prom dresses”  Good ad but bad organic result 8 Ad keywords are for targeting