1. CCF ADL, Jul 31 2011 Susan Dumais Microsoft Research http://research.microsoft.com/~sdumais Temporal Dynamics and Information Retrieval In collaboration with: Eric Horvitz, Jaime Teevan, Eytan Adar, Jon Elsas, Ed Cutrell, Dan Liebling, Richard Hughes, Merrie Ringel Morris, Evgeniy Gabrilovich, Krysta Svore, Anagha Kulkani

2. CCF ADL, Jul 31 2011 Change is Everywhere in IR Change is everywhere in digital information systems Change is everywhere in digital information systems New documents appear all the time New documents appear all the time Document content changes over time Document content changes over time Queries and query volume change over time Queries and query volume change over time What’s relevant to a query changes over time What’s relevant to a query changes over time E.g., U.S. Open 2011 (in May vs. Sept) E.g., U.S. Open 2011 (in May vs. Sept) User interaction changes over time User interaction changes over time E.g., tags, anchor text, social networks, query-click streams, etc. E.g., tags, anchor text, social networks, query-click streams, etc. Relations between entities change over time Relations between entities change over time E.g., President of the US in 2008 vs. 2004 vs. 2000 E.g., President of the US in 2008 vs. 2004 vs. 2000 Change is pervasive in digital information systems Change is pervasive in digital information systems … yet, we’re not doing much about it ! … yet, we’re not doing much about it !

3. CCF ADL, Jul 31 2011 Information Dynamics 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 Content Changes Today’s Browse and Search Experiences But, ignores … User Visitation/ReVisitation 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

4. CCF ADL, Jul 31 2011 Digital Dynamics Easy to Capture Easy to capture Easy to capture But … few tools support dynamics But … few tools support dynamics

5. CCF ADL, Jul 31 2011 Overview Change on the desktop and news Change on the desktop and news Desktop: Stuff I’ve Seen; Memory Landmarks; LifeBrowser Desktop: Stuff I’ve Seen; Memory Landmarks; LifeBrowser News: Analysis of novelty (e.g., NewsJunkie) News: Analysis of novelty (e.g., NewsJunkie) Change on the Web Change on the Web Content changes over time Content changes over time User interaction varies over time (queries, re-visitation, anchor text, query-click stream, “likes”) User interaction varies over time (queries, re-visitation, anchor text, query-click stream, “likes”) Tools for understanding Web change (e.g., Diff-IE) Tools for understanding Web change (e.g., Diff-IE) Improving Web retrieval using dynamics Improving Web retrieval using dynamics Query trends over time Query trends over time Retrieval models that leverage dynamics Retrieval models that leverage dynamics Task evolution over time Task evolution over time

6. CCF ADL, Jul 31 2011 Stuff I’ve Seen (SIS) Many silos of information Many silos of information SIS: SIS: Unified access to distributed, heterogeneous content (mail, files, web, tablet notes, rss, etc.) Unified access to distributed, heterogeneous content (mail, files, web, tablet notes, rss, etc.) Index full content + metadata Index full content + metadata Fast, flexible search Fast, flexible search Information re-use Information re-use Stuff I’ve Seen Windows-DS [Dumais et al., SIGIR 2003] SIS -> SIS -> Windows Desktop Search Windows Desktop Search

7. Example Desktop Searches Looking for: recent email from Fedor that contained a link to his new demo Initiated from: Start menu Query: from:Fedor Looking for: the pdf of a SIGIR paper on context and ranking (not sure it used those words) that someone (don’t remember who) sent me about a month ago Initiated from: Outlook Query: SIGIR Looking for: meeting invite for the last intern handoff Initiated from: Start menu Query: intern handoff kind:appointment Looking for: C# program I wrote a long time ago Initiated from: Explorer pane Query: QCluster*.* CCF ADL, Jul 31 2011 Lots of metadata … especially time Stuff I’ve Seen

8. Stuff I’ve Seen: Findings Evaluation: Evaluation: Internal to Microsoft, ~3000 users in 2004 Internal to Microsoft, ~3000 users in 2004 Methods: free-form feedback, questionnaires, usage patterns from log data, in situ experiments, lab studies for richer data Methods: free-form feedback, questionnaires, usage patterns from log data, in situ experiments, lab studies for richer data Personal store characteristics: Personal store characteristics: 5k–1500k items 5k–1500k items Information needs: Information needs: Desktop search != Web search Desktop search != Web search Short queries (1.6 words) Short queries (1.6 words) Few advanced operators in the initial query (~7%) Few advanced operators in the initial query (~7%) But … many advanced operators and query iteration in UI (48%) But … many advanced operators and query iteration in UI (48%) Filters (type, date, people); modify query; re-sort results Filters (type, date, people); modify query; re-sort results People know a lot about what they are looking for and we need to provide a way to express it ! People know a lot about what they are looking for and we need to provide a way to express it ! CCF ADL, Jul 31 2011

9. Stuff I’ve Seen: Findings Information needs: Information needs: People are important – 29% queries involve names/aliases People are important – 29% queries involve names/aliases Date is the most common sort order Date is the most common sort order Even w/ “best-match” default Even w/ “best-match” default Few searches for “best” matching object Few searches for “best” matching object Many other criteria (e.g., time, people, type), depending on task Many other criteria (e.g., time, people, type), depending on task Need to support flexible access Need to support flexible access Abstraction is important – “useful” date, people, pictures Abstraction is important – “useful” date, people, pictures Age of items retrieved Age of items retrieved Today (5%), Last week (21%), Last month (47%) Today (5%), Last week (21%), Last month (47%) Need to support episodic access to memory Need to support episodic access to memory CCF ADL, Jul 31 2011

10. Beyond Stuff I’ve Seen Better support for human memory Better support for human memory Memory Landmarks Memory Landmarks LifeBrowser LifeBrowser Phlat Phlat Beyond search Beyond search Proactive retrieval Proactive retrieval Stuff I Should See (IQ) Stuff I Should See (IQ) Temporal Gadget Temporal Gadget Using desktop index as a rich “user model” Using desktop index as a rich “user model” PSearch PSearch NewsJunkie NewsJunkie Diff-IE Diff-IE CCF ADL, Jul 31 2011

11. Memory Landmarks Importance of episodes in human memory Importance of episodes in human memory Memory organized into episodes (Tulving, 1983) Memory organized into episodes (Tulving, 1983) People-specific events as anchors (Smith et al., 1978) People-specific events as anchors (Smith et al., 1978) Time of events often recalled relative to other events, historical or autobiographical (Huttenlocher & Prohaska, 1997) Time of events often recalled relative to other events, historical or autobiographical (Huttenlocher & Prohaska, 1997) Identify and use landmarks facilitate search and information management Identify and use landmarks facilitate search and information management Timeline interface, augmented w/ landmarks Timeline interface, augmented w/ landmarks Learn Bayesian models to identify memorable events Learn Bayesian models to identify memorable events Extensions beyond search, e.g., Life Browser Extensions beyond search, e.g., Life Browser CCF ADL, Jul 31 2011

12. Memory Landmarks Search Results Memory Landmarks - General (world, calendar) - Personal (appts, photos) Linked to results by time Distribution of Results Over Time [Ringle et al., 2003]

13. CCF ADL, Jul 31 2011 Memory Landmarks: Findings Dates OnlyLandmarks + Dates 0 5 10 15 20 25 30 Search Time (s) With Landmarks Without Landmarks

14. Memory Landmarks Learned models of memorability CCF ADL, Jul 31 2011 [Horvitz et al., 2004]

15. Images & videos Appts & events Desktop & search activity Whiteboard capture Locations LifeBrowser [Horvitz & Koch, 2010] CCF ADL, Jul 31 2011

16. LifeBrowser Learned models of selective memory CCF ADL, Jul 31 2011

17. News is a stream of information w/ evolving events News is a stream of information w/ evolving events But, it’s hard to consume it as such But, it’s hard to consume it as such Personalized news using information novelty Personalized news using information novelty Identify clusters of related articles Identify clusters of related articles Characterize what a user knows about an event Characterize what a user knows about an event Compute the novelty of new articles, relative to this background knowledge (relevant & novel) Compute the novelty of new articles, relative to this background knowledge (relevant & novel) Novelty = KLDivergence (article || current_knowledge) Novelty = KLDivergence (article || current_knowledge) Use novelty score and user preferences to guide what, when, and how to show new information Use novelty score and user preferences to guide what, when, and how to show new information [Gabrilovich et al., WWW 2004] NewsJunkie Personalized news via information novelty

18. CCF ADL, Jul 31 2011 NewsJunkie in Action Novelty Score Friends say Wells is innocent Looking for two people Copycat case in Missouri Article Sequence by Time Gun disguised as cane NewsJunkie: Pizza delivery man w/ bomb incident

19. NewsJunkie in Action CCF ADL, Jul 31 2011

20. News Junkie Evaluation Experiment to evaluate algorithms for detecting novelty Experiment to evaluate algorithms for detecting novelty Task: Given background article, select set of articles that you would recommend for a friend who wants to find out what’s new about the story Task: Given background article, select set of articles that you would recommend for a friend who wants to find out what’s new about the story KL and Named Entity algorithms better than temporal KL and Named Entity algorithms better than temporal But, many types of “differences” But, many types of “differences” Recap, review of prior information Recap, review of prior information Elaboration, new information Elaboration, new information Offshoot, related but mostly about something else Offshoot, related but mostly about something else Irrelevant, not related to main story Irrelevant, not related to main story CCF ADL, Jul 31 2011

21. NewsJunkie Types of novelty, via intra-article novelty dynamics Offshoot: SARS impact on Asian stock markets Word Position Novelty Score On-topic, recap Word Position Novelty Score On topic, elaboration: SARS patient’s wife held under quarantine Word Position Novelty Score Offshoot: Swiss company develops SARS vaccine Word Position Novelty Score

22. CCF ADL, Jul 31 2011 Overview Change on the desktop and news Change on the desktop and news Desktop: Stuff I’ve Seen; Memory Landmarks; LifeBrowser Desktop: Stuff I’ve Seen; Memory Landmarks; LifeBrowser News: Analysis of novelty (e.g., NewsJunkie) News: Analysis of novelty (e.g., NewsJunkie) Change on the Web Change on the Web Content changes over time Content changes over time User interaction varies over time (queries, re-visitation, anchor text, query-click stream, “likes”) User interaction varies over time (queries, re-visitation, anchor text, query-click stream, “likes”) Tools for understanding Web change (e.g., Diff-IE) Tools for understanding Web change (e.g., Diff-IE) Improving Web retrieval using dynamics Improving Web retrieval using dynamics Query trends over time Query trends over time Retrieval models that leverage dynamics Retrieval models that leverage dynamics Task evolution over time Task evolution over time Questions?

23. CCF ADL, Jul 31 2011 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 Content Changes User Visitation/ReVisitation 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 Characterizing Web Change Large-scale Web crawls, over time Large-scale Web crawls, over time Revisited pages Revisited pages 55,000 pages crawled hourly for 18+ months 55,000 pages crawled hourly for 18+ months Unique users, visits/user, time between visits Unique users, visits/user, time between visits Pages returned by a search engine (for ~ 100k queries) Pages returned by a search engine (for ~ 100k queries) 6 million pages crawled every two days for 6 months 6 million pages crawled every two days for 6 months [Adar et al., WSDM 2009]

24. CCF ADL, Jul 31 2011 Measuring Web Page Change Summary metrics Summary metrics Number of changes Number of changes Amount of change Amount of change Time between changes Time between changes Change curves Change curves Fixed starting point Fixed starting point Measure similarity over different time intervals Measure similarity over different time intervals Within-page changes Within-page changes

25. CCF ADL, Jul 31 2011 Measuring Web Page Change Summary metrics Summary metrics Number of changes Number of changes Amount of change Amount of change Time between changes Time between changes 33% of Web pages change 33% of Web pages change 66% of visited Web pages change 66% of visited Web pages change 63% of these change every hr. 63% of these change every hr. Avg. Dice coeff. = 0.80 Avg. Dice coeff. = 0.80 Avg. time bet. change = 123 hrs. Avg. time bet. change = 123 hrs..edu and.gov pages change infrequently, and not by much.edu and.gov pages change infrequently, and not by much popular pages change more frequently, but not by much popular pages change more frequently, but not by much

26. CCF ADL, Jul 31 2011 Measuring Web Page Change Summary metrics Summary metrics Number of changes Number of changes Amount of change Amount of change Time between changes Time between changes Change curves Change curves Fixed starting point Fixed starting point Measure similarity over different time intervals Measure similarity over different time intervals Knot point Time from starting point

27. CCF ADL, Jul 31 2011 Measuring Within-Page Change DOM-level changes DOM-level changes Term-level changes Term-level changes Divergence from norm Divergence from norm cookbooks cookbooks salads salads cheese cheese ingredient ingredient bbq bbq … “Staying power” in page “Staying power” in page Time Sep. Oct. Nov. Dec.

28. CCF ADL, Jul 31 2011 Example Term Longevity Graphs

29. CCF ADL, Jul 31 2011 Revisitation on the Web 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 Content Changes User Visitation/ReVisitation 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 What was the last Web page you visited? Why did you visit (re-visit) the page? Revisitation patterns Revisitation patterns Log analyses Log analyses Toolbar logs for revisitation Toolbar logs for revisitation Query logs for re-finding Query logs for re-finding User survey to understand intent in revisitations User survey to understand intent in revisitations [Adar et al., CHI 2009]

30. 60-80% of Web pages you visit, you’ve visited before 60-80% of Web pages you visit, you’ve visited before Many motivations for revisits Many motivations for revisits CCF ADL, Jul 31 2011 Measuring Revisitation Summary metrics Summary metrics Unique visitors Unique visitors Visits/user Visits/user Time between visits Time between visits Revisitation curves Revisitation curves Histogram of revisit intervals Histogram of revisit intervals Normalized Normalized Time Interval

31. CCF ADL, Jul 31 2011 Four Revisitation Patterns Fast Fast Hub-and-spoke Hub-and-spoke Navigation within site Navigation within site Hybrid Hybrid High quality fast pages High quality fast pages Medium Medium Popular homepages Popular homepages Mail and Web applications Mail and Web applications Slow Slow Entry pages, bank pages Entry pages, bank pages Accessed via search engine Accessed via search engine

32. CCF ADL, Jul 31 2011 Relationships Between Change and Revisitation Interested in change Interested in change Monitor Monitor Effect change Effect change Transact Transact Change unimportant Change unimportant Re-find old Re-find old Change can interfere with re-finding Change can interfere with re-finding

33. CCF ADL, Jul 31 2011 Revisitation and Search (ReFinding) Repeat query (33%) Repeat query (33%) Q: microsoft research Q: microsoft research Click same and different URLs Click same and different URLs Repeat click (39%) Repeat click (39%) http://research.microsoft.com/ http://research.microsoft.com/ Q: microsoft research; msr Q: microsoft research; msr Big opportunity (43%) Big opportunity (43%) 24% “navigational revisits” 24% “navigational revisits” [Teevan et al., SIGIR 2007] [Tyler et al., WSDM 2010] [Teevan et al., WSDM 2011]

34. CCF ADL, Jul 31 2011 Building Support for Web Dynamics 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 Content Changes 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 Diff IE Temporal IR User Visitation/ReVisitation

35. CCF ADL, Jul 31 2011 Diff-IE Changes to page since your last visit Diff-IE toolbar [Teevan et al., UIST 2009] [Teevan et al., CHI 2010]

36. CCF ADL, Jul 31 2011 Interesting Features of Diff-IE Always on In-situ New to you Non-intrusive Try it: http://research.microsoft.com/en-us/projects/diffie/default.aspx http://research.microsoft.com/en-us/projects/diffie/default.aspx

37. CCF ADL, Jul 31 2011 Examples of Diff-IE in Action

38. CCF ADL, Jul 31 2011 Expected New Content

39. CCF ADL, Jul 31 2011 Monitor

40. Serendipitous Encounters

41. CCF ADL, Jul 31 2011 Unexpected Important Content

42. CCF ADL, Jul 31 2011 Understand Page Dynamics

43. CCF ADL, Jul 31 2011 Unexpected Unimportant Content Attend to Activity Edit Understand Page Dynamics Serendipitous Encounter Unexpected Important Content Expected New Content Monitor Expected Unexpected

44. CCF ADL, Jul 31 2011 Studying Diff-IE Feedback buttons Feedback buttons Survey Survey Prior to installation Prior to installation After a month of use After a month of use Logging Logging URLs visited URLs visited Amount of change when revisited Amount of change when revisited Experience interview Experience interview In situ Representative Experience Longitudinal

45. CCF ADL, Jul 31 2011 People Revisit More Perception of revisitation remains constant Perception of revisitation remains constant How often do you revisit? How often do you revisit? How often are revisits to view new content? How often are revisits to view new content? Actual revisitation increases Actual revisitation increases Last week: 45.0% of visits are revisits Last week: 45.0% of visits are revisits First week: 39.4% of visits are revisits First week: 39.4% of visits are revisits Why are people revisiting more with DIFF-IE? Why are people revisiting more with DIFF-IE? 14%

46. CCF ADL, Jul 31 2011 Revisited Pages Change More Perception of change increases Perception of change increases What proportion of pages change regularly? What proportion of pages change regularly? How often do you notice unexpected change? How often do you notice unexpected change? Amount of change seen increases Amount of change seen increases Last week: 32.4% revisits changed, by 9.5% Last week: 32.4% revisits changed, by 9.5% First week: 21.5% revisits changed, by 6.2% First week: 21.5% revisits changed, by 6.2% Diff-IE is driving visits to changed pages Diff-IE is driving visits to changed pages It supports people in understanding change It supports people in understanding change 51+% 17% 8%

47. CCF ADL, Jul 31 2011 Change by Page Type Perceptions of change reinforced Perceptions of change reinforced Pages that change a lot  change more Pages that change a lot  change more Pages that change a little  change less Pages that change a little  change less News pages Message boards, forums, news groups Search engine results Blogs you read Pages with product information Wikipedia pages Company homepages Personal home pages of people you know Reference pages (dictionaries, yellow pages, maps) Change a little Change a lot

48. Other Examples of Dynamics and User Experience Content changes Content changes Diff-IE (Teevan et al., 2008) Diff-IE (Teevan et al., 2008) Zoetrope (Adar et al., 2008) Zoetrope (Adar et al., 2008) Diffamation (Chevalier et al., 2010) Diffamation (Chevalier et al., 2010) Temporal summaries and snippets … Temporal summaries and snippets … Interaction changes Interaction changes Explicit annotations, ratings, wikis, etc. Explicit annotations, ratings, wikis, etc. Implicit interest via interaction patterns Implicit interest via interaction patterns Edit wear and read wear (Hill et al., 1992) Edit wear and read wear (Hill et al., 1992) CCF ADL, Jul 31 2011

49. Overview Change on the desktop and news Change on the desktop and news Desktop: Stuff I’ve Seen; Memory Landmarks; LifeBrowser Desktop: Stuff I’ve Seen; Memory Landmarks; LifeBrowser News: Analysis of novelty (e.g., NewsJunkie) News: Analysis of novelty (e.g., NewsJunkie) Change on the Web Change on the Web Content changes over time Content changes over time User interaction varies over time (queries, re-visitation, anchor text, query-click stream, “likes”) User interaction varies over time (queries, re-visitation, anchor text, query-click stream, “likes”) Tools for understanding Web change (e.g., Diff-IE) Tools for understanding Web change (e.g., Diff-IE) Improving Web retrieval using dynamics Improving Web retrieval using dynamics Query trends over time Query trends over time Retrieval models that leverage dynamics Retrieval models that leverage dynamics Task evolution over time Task evolution over time Questions?

50. CCF ADL, Jul 31 2011 Leveraging Dynamics for Retrieval 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 Content Changes 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 User Visitation/ReVisitation Temporal IR

51. Query frequency over time Query frequency over time Retrieval models that incorporate time Retrieval models that incorporate time Ranking algorithms typically look only at a single snapshot in time Ranking algorithms typically look only at a single snapshot in time But, both content and user interaction with the content change over time But, both content and user interaction with the content change over time Model content change on a page Model content change on a page Model user interactions Model user interactions Tasks evolve over time Tasks evolve over time CCF ADL, Jul 31 2011

52. Query Dynamics Queries sometimes mention time, but often don’t Queries sometimes mention time, but often don’t Explicit time (e.g., World Cup Soccer 2011) Explicit time (e.g., World Cup Soccer 2011) Explicit news (e.g., earthquake news) Explicit news (e.g., earthquake news) Implicit time (e.g., Harry Potter reviews; implicit “now”) Implicit time (e.g., Harry Potter reviews; implicit “now”) Queries are not uniformly distributed over time Queries are not uniformly distributed over time Often triggered by events in the world Often triggered by events in the world Using temporal query patterns to: Using temporal query patterns to: Cluster similar queries Cluster similar queries Identify events and find related news Identify events and find related news

53. Query Dynamics Q: cinema Q: cinema Discrete Fourier Transform Discrete Fourier Transform Best k components (vs. first k) Best k components (vs. first k) Best k significantly reduce reconstruction error Best k significantly reduce reconstruction error Burst detection Burst detection Bursts as deviations from moving average Bursts as deviations from moving average [Vlachos et al., SIGMOD 2004] Modeling query frequency over time (Vlachos et al.) Modeling query frequency over time (Vlachos et al.)

54. Query Dynamics Types of query popularity patterns Types of query popularity patterns Number of spikes (0, 1, multiple) Number of spikes (0, 1, multiple) Periodic (yes, no) Periodic (yes, no) Shape of rise and fall (wedge, sail, castle) Shape of rise and fall (wedge, sail, castle) Trend (flat, up, down) Trend (flat, up, down) Changes in query popularity and content to changes in user intent (i.e., what is relevant to the query) … more on this later Changes in query popularity and content to changes in user intent (i.e., what is relevant to the query) … more on this later [Kulkarni et al., WSDM 2011]

55. CCF ADL, Jul 31 2011 Model query patterns using empirical query frequency (normalized by total queries per day) Model query patterns using empirical query frequency (normalized by total queries per day) Examples: Examples: Identify “similar” queries using correlation coefficient between the normalized time series Identify “similar” queries using correlation coefficient between the normalized time series Nice use of time to identify semantic similarity between queries/entities, but not predictive Nice use of time to identify semantic similarity between queries/entities, but not predictive Using Query Dynamics to Find Similar Queries [Chien & Immorlica, WWW 2005] Q: movies Q: scott peterson Q: weather report

56. Query Dynamics via Document Dates News: date of document easy to obtain News: date of document easy to obtain Li & Croft – used publication date of documents to rank results Li & Croft – used publication date of documents to rank results Jones & Diaz – model period of time relevant to a query (temporal profile of query) Jones & Diaz – model period of time relevant to a query (temporal profile of query) Three classes of queries Three classes of queries Atemporal (0 bursts) Atemporal (0 bursts) Temporally unambiguous (1 burst) Temporally unambiguous (1 burst) Temporally ambiguous (>1 burst) Temporally ambiguous (>1 burst) Web: date of documents harder to obtain Web: date of documents harder to obtain CCF ADL, Jul 31 2011

57. Many queries to Web search engines are motivated by events in the world Many queries to Web search engines are motivated by events in the world Should you show just Web results? Should you show just Web results? Or, provide an integrated view of news and Web? Or, provide an integrated view of news and Web? Example Example Learn model to predict “newsworthiness” of a query (i.e., will a user click on news results) Learn model to predict “newsworthiness” of a query (i.e., will a user click on news results) Is the query part of a burst? [content consumption] Is the query part of a burst? [content consumption] Are the top-ranked news results very recent? [content production] Are the top-ranked news results very recent? [content production] Improve prediction using ongoing click data for this and related queries Improve prediction using ongoing click data for this and related queries [Diaz 2010] Using Query Dynamics to Identify “News” Queries

58. Temporal Retrieval Models 1 Current retrieval algorithms look only at a single snapshot of a page Current retrieval algorithms look only at a single snapshot of a page But, Web page content changes over time But, Web page content changes over time Can we can leverage this to improved retrieval? Can we can leverage this to improved retrieval? Pages have different rates of change Pages have different rates of change Different priors (using change vs. link structure) Different priors (using change vs. link structure) Terms have different longevity (staying power) Terms have different longevity (staying power) Some are always on the page; some transient Some are always on the page; some transient Language modeling approach to ranking Language modeling approach to ranking CCF ADL, Jul 31 2011 Change prior Term longevity [Elsas et al., WSDM 2010]

59. Relevance and Page Change Page change is related to relevance judgments Page change is related to relevance judgments Human relevance judgments Human relevance judgments 5 point scale – Perfect/Excellent/Good/Fair/Bad 5 point scale – Perfect/Excellent/Good/Fair/Bad Rate of Change -- 60% Perfect pages; 30% Bad pages Rate of Change -- 60% Perfect pages; 30% Bad pages Use change rate as a document prior (vs. priors based on link structure like Page Rank) Use change rate as a document prior (vs. priors based on link structure like Page Rank) Shingle prints to measure change Shingle prints to measure change CCF ADL, Jul 31 2011 Change prior

60. Relevance and Term Change Terms patterns vary over time Terms patterns vary over time Represent a document as a mixture of terms with different “staying power” Represent a document as a mixture of terms with different “staying power” Long, Medium, Short Long, Medium, Short CCF ADL, Jul 31 2011 Term longevity

61. Evaluation: Queries & Documents 18K Queries, 2.5M Judged Documents 18K Queries, 2.5M Judged Documents 5-level relevance judgment (Perfect … Bad) 5-level relevance judgment (Perfect … Bad) 2.5M Documents crawled weekly for 10 wks 2.5M Documents crawled weekly for 10 wks Navigational queries Navigational queries 2k queries identified with a “Perfect” judgment 2k queries identified with a “Perfect” judgment Assume these relevance judgments are consistent over time Assume these relevance judgments are consistent over time CCF ADL, Jul 31 2011

62. Experimental Results Baseline Static Model Dynamic Model Dynamic Model + Change Prior Change Prior CCF ADL, Jul 31 2011

63. Temporal Retrieval Models 2 Initial evaluation Initial evaluation Navigational queries; assume relevance is “static” over time Navigational queries; assume relevance is “static” over time But, relevance often changes over time But, relevance often changes over time E.g., Stanley Cup in 2011 vs. in 2010 E.g., Stanley Cup in 2011 vs. in 2010 E.g., US Open 2011 in May (golf) vs. in Sept (tennis) E.g., US Open 2011 in May (golf) vs. in Sept (tennis) E.g., March madness 2011 E.g., March madness 2011 Before event: Schedule and tickets, e.g., stubhub Before event: Schedule and tickets, e.g., stubhub During event: Real-time scores, e.g., espn, cbssports During event: Real-time scores, e.g., espn, cbssports After event: General sites, e.g., wikipedia, ncaa After event: General sites, e.g., wikipedia, ncaa Ongoing evaluation Ongoing evaluation Collecting explicit relevance judgments, query frequency, interaction data and page content over time Collecting explicit relevance judgments, query frequency, interaction data and page content over time Developing temporal IR models, temporal snippets Developing temporal IR models, temporal snippets CCF ADL, Jul 31 2011 [Kulkarni et al., WSDM 2011] [Radinsky et al., in prep]

64. Relevance over Time Query: march madness [Mar 15 – Apr 4, 2010] Query: march madness [Mar 15 – Apr 4, 2010] CCF ADL, Jul 31 2011 DuringAfter

65. Relevance over Time Query: sigir Query: sigir Why is old content ranked higher? Why is old content ranked higher? User interaction data more prevalent for older documents User interaction data more prevalent for older documents E.g., query-clicks, anchor text, etc. E.g., query-clicks, anchor text, etc. CCF ADL, Jul 31 2011 2011 2010 2009

66. Relevance over Time Query: nfl Query: nfl Again, time of query and time of event not well modeled Again, time of query and time of event not well modeled CCF ADL, Jul 31 2011 Sept - Jan Feb - Aug

67. Time Series Modeling of Queries and Clicks E.g., ny daily news E.g., ny daily news Q: weekly period Q: weekly period URL clicks: consistent ordering URL clicks: consistent ordering E.g., gold miner E.g., gold miner Q: more spikes Q: more spikes URL clicks: most clicked URL varies over time URL clicks: most clicked URL varies over time Model queries and URL clicks as time series Model queries and URL clicks as time series CCF ADL, Jul 31 2011

68. Time Series Modeling CCF ADL, Jul 31 2011

69. Experimental Results Ground truth Ground truth Actual user behavior over time Actual user behavior over time Two search-related prediction tasks Two search-related prediction tasks Predict number of URL clicks (prediction error) Predict number of URL clicks (prediction error) Ranking of URLs for search (correlation) Ranking of URLs for search (correlation) Three types of models Three types of models No historical features No historical features Historical average Historical average Time-series modeling Time-series modeling Results (preliminary) Results (preliminary) TaskPrediction ErrorRanking URLs Model Text Only 0.012 Text + Historical Avg17.610.150 Text + Time Series16.720.300 CCF ADL, Jul 31 2011

70. Task Evolution over Time Most retrieval models look at queries in isolation Most retrieval models look at queries in isolation But, people often use multiple queries to accomplish a task But, people often use multiple queries to accomplish a task Within-session tasks Within-session tasks Use previous actions in current session to improve understanding of current query Use previous actions in current session to improve understanding of current query Cross-session tasks Cross-session tasks Develop methods to predict and support task resumption over time Develop methods to predict and support task resumption over time CCF ADL, Jul 31 2011

71. Within-Session Tasks Examples Examples Q: [sigir] … given [information retrieval] vs. [iraq reconstruction] Q: [sigir] … given [information retrieval] vs. [iraq reconstruction] Q: [acl] … given [computational linguistics] vs. [knee injury] vs. [country music] Q: [acl] … given [computational linguistics] vs. [knee injury] vs. [country music] 40% of session contain multiple queries 40% of session contain multiple queries 60% of queries have at least one preceding query 60% of queries have at least one preceding query Use previous actions in current session to improve understanding of current query Use previous actions in current session to improve understanding of current query Model using ODP categories Model using ODP categories Use to predict future actions, rank current results, etc. Use to predict future actions, rank current results, etc. Health/Medicine Music Computer Science Sports Business [White et al., CIKM 2010] CCF ADL, Jul 31 2011

72. Within-Session Results Context helps Context helps Using any context source improves accuracy Using any context source improves accuracy Using more sources improves accuracy Using more sources improves accuracy Context source Models None (i.e., current query only) 0.39  Queries (i.e., all previous queries) 0.390.42 0.43 Queries + SERPClicks (i.e., all previous queries / result clicks) 0.390.46 Queries + SERPClicks + NavTrails (i.e., all previous actions) 0.390.50 0.49 Context source Percentage of queries best Between models Queries (i.e., all previous queries)25%18%22% Queries + SERPClicks (i.e., all previous queries / result clicks) 30%16%25% Queries + SERPClicks + NavTrails (i.e., all previous actions) 34%11%30% Differences across queries Differences across queries Query model wins: current query has specific intent [espn] [webmd], or first actions after a shift in interests Query model wins: current query has specific intent [espn] [webmd], or first actions after a shift in interests Context model wins: query is ambiguous [amazon] and session has a consistent intent Context model wins: query is ambiguous [amazon] and session has a consistent intent Intent model wins: session has consistent intent throughout Intent model wins: session has consistent intent throughout CCF ADL, Jul 31 2011

73. Research Missions Research missions (from Jones & Klinkner) Research missions (from Jones & Klinkner) Identify “research missions” on-the-fly during a search session Identify “research missions” on-the-fly during a search session Three general signals Three general signals Research_mission (q1, q2) classifier Research_mission (q1, q2) classifier Same_mission (q1, q2) classifier Same_mission (q1, q2) classifier Sim (topics (q1), topics (q2)) Sim (topics (q1), topics (q2)) Many features Many features Textual – similarity q1, q1 Textual – similarity q1, q1 Session-based – queries, clicks, queries since last click, etc. Session-based – queries, clicks, queries since last click, etc. Time-based – time between q1, q2, total session time, etc. Time-based – time between q1, q2, total session time, etc. Trigger Yahoo! Scratch Pad if a research mission is detected Trigger Yahoo! Scratch Pad if a research mission is detected [Donato et al., WWW 2010] [Jones & Klinckner, CIKM 2008) CCF ADL, Jul 31 2011

74. Cross-Session Tasks Many tasks extend across sessions – e.g., medical diagnosis and treatment, event planning, how-to advice, shopping research, academic research, etc. Many tasks extend across sessions – e.g., medical diagnosis and treatment, event planning, how-to advice, shopping research, academic research, etc. 10-15% of tasks continue across multiple sessions 10-15% of tasks continue across multiple sessions 20-25% of queries are from multi-session tasks 20-25% of queries are from multi-session tasks Example Example Develop methods to support task resumption over time Develop methods to support task resumption over time Same Task: Find (previous) related queries/clicks Same Task: Find (previous) related queries/clicks Task Resumption: Predict whether user will resume task Task Resumption: Predict whether user will resume task [Kotov et al., SIGIR 2011] CCF ADL, Jul 31 2011

75. Cross-Session Tasks Results Results Same Task Same Task Task Continuation Task Continuation Develop support for task continuation Develop support for task continuation [Kotov et al., SIGIR 2011] Approach Approach Classification (logistic regression, MART) Classification (logistic regression, MART) Features (query, pair-wise, session-based, history-based) Features (query, pair-wise, session-based, history-based) CCF ADL, Jul 31 2011

76. Other Examples of Dynamics and Information Systems Document dynamics, for crawling and indexing Document dynamics, for crawling and indexing Adar et al. (2009); Cho & Garcia-Molina (2000); Fetterly et al. (2003) Adar et al. (2009); Cho & Garcia-Molina (2000); Fetterly et al. (2003) Query dynamics Query dynamics Kulkarni et al. (2011); Jones & Diaz (2004); Diaz (2009); Kotov et al. (2010) Kulkarni et al. (2011); Jones & Diaz (2004); Diaz (2009); Kotov et al. (2010) Temporal retrieval models Temporal retrieval models Elsas & Dumais (2010); Liu & Croft (2004); Efron (2010); Aji et al. (2010) Elsas & Dumais (2010); Liu & Croft (2004); Efron (2010); Aji et al. (2010) Extraction of temporal entities within documents Extraction of temporal entities within documents Protocol extension for retrieving versions over time Protocol extension for retrieving versions over time E.g., Memento (Van de Sompel et al., 2010) E.g., Memento (Van de Sompel et al., 2010) CCF ADL, Jul 31 2011

77. Summary 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 Content Changes 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 Relating revisitation and change allows us to – Identify pages for which change is important – Identify interesting components within a page Diff-IE: Supports (and influences) interaction and understanding Temporal IR: Leverages change for improved IR Web content changes: page-level, term-level User Visitation/ReVisitation People revisit and re-find Web content Time on the desktop, news, real-time media

78. Challenges and Opportunities Temporal dynamics are pervasive in information systems Temporal dynamics are pervasive in information systems Influence many aspect of information systems Influence many aspect of information systems Systems: protocols, crawling, indexing, caching Systems: protocols, crawling, indexing, caching Document representations : meta-data generation, information extraction, sufficient statistics at page and term-level Document representations : meta-data generation, information extraction, sufficient statistics at page and term-level Retrieval models: term weights, document priors, etc. Retrieval models: term weights, document priors, etc. User experience and evaluation User experience and evaluation Better supporting temporal dynamics of information Better supporting temporal dynamics of information Requires digital preservation and temporal metadata extraction Requires digital preservation and temporal metadata extraction Enables richer understanding of the evolution (and prediction) of key ideas, relations, and trends over time Enables richer understanding of the evolution (and prediction) of key ideas, relations, and trends over time Time is one important example of context and IR Time is one important example of context and IR Others include: location, individuals, tasks … Others include: location, individuals, tasks … CCF ADL, Jul 31 2011

79. User Context Task/Use Context Document Context Query Words Ranked List Think Outside the (Search) Boxes Search Research

80. CCF ADL, Jul 31 2011 Thank You ! Questions/Comments … More info, http://research.microsoft.com/~sdumais http://research.microsoft.com/~sdumais Diff-IE … try it! http://research.microsoft.com/en-us/projects/diffie/default.aspx http://research.microsoft.com/en-us/projects/diffie/default.aspx

81. References Desktop and news Dumais et al. (SIGIR 2003). Ringle et al. (Interact 2003). Horvitz & Koch (2010). Gabrilovich et al. (WWW 2004). Diaz (2009). Document dynamics Adar et al. (WSDM 2009). Cho & Garcia-Molina (VLDB 2000). Fetterly et al. (WWW 2003). User interaction Adar et al. (CHI 2009). Teevan et al. (SIGIR 2007). Tyler et al. (WSDM 2010). Teevan et al. (WSDM 2011). Adar et al. (CHI 2010). Teevan et al. (UIST 2009). Teevan et al. (CHI 2010). CCF ADL, Jul 31 2011 Query dynamics Kulkarni et al. (WSDM 2011). Jones & Diaz (TOIS 2007). Vlachos et al. (SIGMOD 2004). Chien & Immorlica (WWW 2005). Temporal retrieval models Elsas & Dumais (WSDM 2010). Radinsky (in prep). Li & Croft (2004). Efron & Golovshinshy (SIRGIR 2011). Tasks over time White et al. (CIKM 2010). Donato et al. (WWW 2010). Kotov et al. (SIGIR 2011). Jones and Klinkner (CIKM 2008). General Kleinberg (). Van de Sompel et al. (Arxiv 2009).