1. The Web Changes Everything How Dynamic Content Affects the Way People Find Online Jaime Teevan Microsoft Research (CLUES) with S. Dumais, D. Liebling, E. Adar, J. Elsas, and R. Hughes

2. Information Artifacts Change

3. Digital Dynamics Easy to Capture

4. The Web Changes Everything JanuaryFebruaryMarch April May JuneJuly August September Content Changes People Revisit JanuaryFebruaryMarch April May JuneJuly August September Today’s Browse and Search Experiences But, ignores …

5. The Web Changes Everything JanuaryFebruaryMarch April May JuneJuly August September Content Changes Large scale Web crawl over time – Revisited pages Unique users, visits/user, time between visits 55,000 pages crawled hourly for 18+ months – Judged pages (relevance to a query) 6 million pages crawled every two days for 6 months

6. Measuring Web Page Change Summary metrics – Number of changes – Time between changes – Amount of change Change curves – Fixed starting point – Measure similarity over different time intervals Knot point

7. Measuring Within-Page Change DOM structure changes Term use changes – Divergence from norm cookbooks salads cheese ingredient bbq – Staying power in page Time Sep. Oct. Nov. Dec.

8. JanuaryFebruaryMarch April May JuneJuly August September Revisitation on the Web Content Changes People Revisit JanuaryFebruaryMarch April May JuneJuly August September Revisitation patterns – Log analysis – Toolbar logs for revisitation – Query logs for re-finding – User survey for intent What’s the last Web page you visited?

9. Measuring Revisitation Summary metrics – Unique visitors – Visits/user – Time between visits Revisitation curves – Histogram of revisit intervals – Normalized Time Interval

10. Four Revisitation Patterns Fast – Hub-and-spoke – Navigation within site Hybrid – High quality fast pages Medium – Popular homepages – Mail and Web applications Slow – Entry pages, bank pages – Accessed via search engine

11. Search and Revisitation How to measure? Repeat query (33%) – hci research at stanford Repeat click (39%) – http://hci.stanford.edu http:// – Query  stanford hci Big opportunity (43%) – Navigational (24%)

12. Repeat Clicks for Repeat Queries Minutes Days Weeks

13. How Revisitation and Change Relate JanuaryFebruaryMarch April May JuneJuly August September Content Changes People Revisit JanuaryFebruaryMarch April May JuneJuly August September Why did you revisit the last Web page you did?

14. Possible Relationships Interested in change – Monitor Effect change – Transact Change unimportant – Find Change can interfere – Re-find

15. Understanding the Relationship Compare summary metrics Revisits: Unique visitors, visits/user, interval Change: Number, interval, similarity 2 visits/user 3 visits/user 4 visits/user 5 or 6 visits/user 7+ visits/user Number of changesTime between changesSimilarity 2 visits/user 172.91133.260.82 3 visits/user 200.51119.240.82 4 visits/user 234.32109.590.81 5 or 6 visits/user 269.6394.540.82 7+ visits/user 341.4381.800.81

16. Comparing Change and Revisit Curves Three pages – New York Times – Woot.com – Costco Similar change patterns Different revisitation – NYT: Fast – Woot: Medium – Costco: Slow (news, forums) (retail) Time

17. Within-Page Relationship Page elements change at different rates Pages revisited at different rates Resonance can serve as a filter for interesting content

21. Building Support for Web Dynamics JanuaryFebruaryMarch April May JuneJuly August September Content Changes JanuaryFebruaryMarch April May JuneJuly August September People Revisit

22. DiffIE Changes to page since your last visit DiffIE toolbar

23. Interesting Features of DiffIE Always on In-situ New to you Non-intrusive

24. EXAMPLES OF DiffIE IN ACTION

25. Expected New Content

26. Monitor

27. Unexpected Important Content

28. Serendipitous Encounters

29. Unexpected Unimportant Content

30. Understand Page Dynamics

31. Attend to Activity

32. Edit

33. Unexpected Unimportant Content Attend to Activity Edit Understand Page Dynamics Serendipitous Encounter Unexpected Important Content Expected New Content Monitor Expected Unexpected

34. Monitor

35. Find Expected New Content

36. Methods for Studying DiffIE Feedback buttons Survey – Prior to installation – After a month of use Logging – URLs visited – Amount of change when revisited Experience interview In situ Representative Experience Longitudinal

37. People Revisit More Perception of revisitation remains constant – How often do you revisit? – How often are revisits to view new content? Actual revisitation increases – First week: 39.4% of visits are revisits – Last week: 45.0% of visits are revisits Why are people revisiting more with DiffIE? 14%

38. Revisited Pages Change More Perception of change increases – What proportion of pages change regularly? – How often do you notice unexpected change? Amount of change seen increases – First week: 21.5% revisits changed by 6.2% – Last week: 32.4% revisits changed by 9.5% DiffIE is driving visits to changed pages 51+% 17% 8%

39. Change by Page Type Perceptions of change reinforced Pages that change a lot  change more Pages that change a little  change less News pages Message boards, forums, news groups Search engine results Blogs you read Pages with product information Wikipedia pages Company homepages Personal home pages of people you know Reference pages (dictionaries, yellow pages, maps) Change little Change a lot

40. The Web Changes Everything JanuaryFebruaryMarch April May JuneJuly August September Content Changes JanuaryFebruaryMarch April May JuneJuly August September Web content changes: Page-level, term-level People revisit and re-find Web content Relating revisitation and change allows us to – Identify pages for which change is important – Identify interesting components within a page Explicit support for Web dynamics can impact how people use and understand the Web People Revisit

41. Change Adar, Teevan, Dumais & Elsas. The Web changes everything: Understanding the dynamics of Web Content. WSDM ’09 (Best Student Paper). Elsas & Dumais. Leveraging temporal dynamics of document content in relevance ranking. WSDM ’10 (to appear). Revisitation Adar, Teevan & Dumais. Large scale analysis of Web revisitation patterns. CHI ’08 (Best Paper). Teevan, Adar, Jones & Potts. Information re-retrieval: Repeat queries in Yahoo’s logs. SIGIR ’07. Tyler & Teevan. Large scale query log analysis of re-finding. WSDM ’10 (to appear). Relationship Adar, Teevan & Dumais. Resonance on the Web: Web dynamics and revisitation patterns. CHI ’09. DiffIE Teevan, Dumais, Liebling & Hughes. Changing how people view changes on the Web. UIST ’09. Teevan, Dumais & Liebling. A longitudinal study of how highlighting Web content change affects people’s Web interactions. CHI ’10 (under submission). Thank you. Jaime Teevan http://research.microsoft.com/~teevan

42. EXTRA SLIDES

43. Web Dynamics JanuaryFebruaryMarch April May JuneJuly August September Content Changes Number of studies of change [2, 7, 10, 20] Frequency and degree of change characterized Visited pages are more likely to change [2]

44. Web Dynamics JanuaryFebruaryMarch April May JuneJuly August September Content Changes People Revisit JanuaryFebruaryMarch April May JuneJuly August September People revisit on the Web a lot – Over half of page visits are revisits [2, 22] – Over a third of searches are for re-finding [23] Revisitation relates to change – 66% of revisits are to changed pages [2] – 20% of the content changes [2] – Revisiting often motivated by change [2, 15] – Change interferes with revisiting [21, 23]

45. Systems That Expose Web Change Historical access to pages – Internet Archives (archive.org) Subscription to change – RSS, Web slices – Monitoring support [15] In-situ awareness of change – symbols – Dynamo [3], Difference Engine [9], WebCQ [17]

46. HOW DiffIE WORKS

47. DiffIE DiffIE Architecture Web Cache Toolbar ComponentComparison Component IE Client Machine

48. Toolbar Compare to older versions Status message Feedback buttons See previous version Hide highlighting

49. Cache Web page representation – Hash of text in leaf DOM nodes – Some information about DOM structure Cache multiple versions of pages visited Small footprint (15-20KB) – Exact duplicates stored as pointer files – Cap count (only 6% of pages visited >5 times) Privacy preserving

50. Thread title# Posts# Views This is the title of a thread3 Something else is discussed here!1582 Read This Thread Today217 0 Comparison Component Highlighted: Additions, changes Not highlighted: Moves, deletions Look for text not present in previous version Semantic grouping of tabular information 2 Same

51. Alternate Comparison Change Deletion Addition Movement A A B B D D C C E E F F A A B B C C D D E E E E D D Node has fewer children Node has more children, child new Node has new child, child present Node has same children, child changes

52. Alternate Comparison Change Deletion Addition Movement Highlighted: Additions, changes Not highlighted: Moves, deletions Node has fewer children Node has more children, child new Node has new child, child present Node has same children, child changes

53. UNDERSTANDING DiffIE

54. DiffIE Received Positively Feedback buttons – 51% of unsolicited feedback positive (v. 10-25%) Survey – Prior to use: 2.83 / 5 – After a month: 3.20 / 5 Experience interview (conditioned on change) – 61% positive – 18% neutral – 21% negative 13%

55. Feedback Positive – “It helps make a new block of content stand out on a long page containing mostly old content.” – “Please don't take away my DiffIE!” Negative – Bugs: “Seemed to stop working on my system.” – Performance: “Had the feeling that my browser took too much time to load sites.” – Robust prototype important

56. Reported Experience with DiffIE

57. Higher value Lower value Unexpected Unimportant Content Attend to Activity Edit Understand Page Dynamics Serendipitous Encounter Unexpected Important Content Monitor Expected New Content

58. Performance Highlighting shown on page load event Appears 10s to 100s of milliseconds after load Does not interfere with browsing experience Often appears after interaction begins Notification of delay important