1. Retroactive Answering of Search Queries Beverly Yang Glen Jeh Google, Inc. byang@google.combyang@google.com glenj@google.comglenj@google.com Presented by: Raghavendra Aekka Kranthi Kumar Kontham

2.  Introduction  QSR Architecture  Identifying the Standing Interests Problem Definition Signals  Determining Recommendations  User Studies First person study design Third person study design  Results of User Studies  Related work  Conclusion Overview

3. Recommendations

4. Personalization

5. QSR QSR = Query-Specific (Web) Recommendations A personalization service that alerts the user when interesting new results to selected previous queries have appeared. e.g.. ``Britney Spears concert San Francisco.''

6. Visualization of Recommended Web Pages

7. Challenges  To automatically identify queries that represent standing interests.  To identify new results that the user would be interested in.

8. QSR Architecture Limit: M queries Limit: N recommendations

9. Functioning of QSR Read User’s Action Identify Top M Queries Compare 1 st Ten New Results with Old Results Display Top N Recommendations To User Submit Each Query to Search Engine Scoring Recommendations

10. Identifying Standing Interests Definition If user is interested in seeing new interesting results, it is said to be Standing Interest in Query Users’ standing interests determine how useful recommendations are.

11. Problem Definition Factors indicating Standing Interest  Prior Fulfillment Whether the user found a satisfactory result.  Query Interest Level Whether still Interested even after satisfactory result.  Need/Interest Duration How timely is the information need.

12. Sample Query Session html encode java (8 s) * RESULTCLICK (91.00 s) - 2. http://www.java2html.de/docs/api/de/java2html/util/HTMLTools.html http://www.java2html.de/docs/api/de/java2html/util/HTMLTools.html * RESULTCLICK (247.00 s) - 1. http://www.javapractices.com/Topic96.cjp http://www.javapractices.com/Topic96.cjp * RESULTCLICK (12.00 s) - 8. http://www.trialfiles.com/program-16687.html * NEXTPAGE (5.00 s) - start = 10 o RESULTCLICK (1019.00 s) - 12. http://forum.java.sun.com/thread.jspa?threadID=562942...http://forum.java.sun.com/thread.jspa?threadID=562942 o REFINEMENT (21.00 s) - html encode java utility + RESULTCLICK (32.00 s) - 7. http://www.javapractices.com/Topic96.cjp http://www.javapractices.com/Topic96.cjp o NEXTPAGE (8.00 s) - start = 10 * NEXTPAGE (30.00 s) - start = 20 (Total time: 1473.00 s)

13. Query Sessions All actions associated with a given initial query. Query Refinement New query contains at least one common term of previous one. How do we determine the Actual Query from a Query Session?

14. Signals Factors for identifying Query Interest  Number of terms E.g.: valentino rossi theaters screening movie 300 in norfolk  Number of clicks and number of refinements More actions by the user indicate more interest.  History match Query matches previous queries or clicks.

15.  Navigational E.g.: google videos  Repeated non-navigational E.g. : Britney Spears Additional Signals Session duration Topic of the query Number of long clicks

16. Interest Score Boolean signals such as repeated non-navigational signals can be used as filters.

17. Determining Interesting Results Already addressed by current web alert services. Quality of recommendation determined by user’s interest. Factors for web alerts Result moved into the top 10 results of a query. Spurious change in the ranking of a page.

18. Characteristics of Good Recommendations  New to the user  Good page  Recently “promoted”

19. Signals  History presence Do not recommend pages which are in users search engine.  Rank High Rank implies good recommendation.  Popularity and relevance score (PR) High PR score implies good recommendation.  Above drop-off Result above threshold PR score is a good recommendation.

20. Additional Signals Days elapsed since query submission signal Sole changed signal Only new result in Top 6. All poor signal All results having low PR score imply no good pages.

21. Quality Score It is a suboptimal indicator of quality. Thus alternate score with superior performance : Boolean signal “Above Dropoff” is used as filter.

22. User Studies Two human subject studies conducted on users of Google's Search History service. First person study Evaluate interest level on their own past queries and quality of recommendations Third person study Evaluate anonymous query sessions and assess the quality of recommendations

23. First Person Study design The survey displayed a maximum of 30 query sessions from the user’s own history For each query session, the user was shown a visual representation of the actions. Generate query recommendations Selecting Query Sessions Eliminate query sessions with No events No clicks and one/two refinements Non repeated navigational queries

24.  From the remaining half of the sessions selected for the survey consisted of highest ranked sessions with respect to high score.  The second half of the sessions for the survey were randomly selected. Selecting Recommendations: Generate recommendations for queries with history presence signal. Consider the current top ten results for a query. Apply Boolean signals to the result that the user has not yet seen as filters. From the remaining select top recommendations with respect to the qscore.

25. Third Person study Design Five human subjects viewed other users’ anonymized query sessions and associated recommendations. Displayed visual representation of the entire query session. Half of the recommendations selected as in first person study. Second half consisted of highest ranked new result in the top ten.

27. Identifying Standing Interest

28. Determining Quality of Recommendations

31. Conclusion Presented QSR, a new system that retroactively answers search queries representing standing interests. QSR addresses two sub problems: Automatic identification of queries that represent standing interest. Identification of new interesting results. Presented algorithms and conducted user studies to evaluate them.

32. Discussions Do the security concerns outweigh the advantages of the QSR system? What do you think was wrong in the User Studies? Why do you think Google is spending so much in research of QSR System?