How to Crawl the Web Junghoo Cho Hector Garcia-Molina Stanford University

2 What is a Crawler? web init get next url get page extract urls initial urls to visit urls visited urls web pages

3 Crawling at Stanford WebBase Project BackRub Search Engine, Page Rank Google New WebBase Crawler

4 Crawling Issues (1) Load at visited web sites –robots.txt files –space out requests to a site –limit number of requests to a site per day –limit depth of crawl –multicast

5 Crawling Issues (2) Load at crawler –parallelize init get next url get page extract urls initial urls to visit urls visited urls web pages init get next url get page extract urls ?

6 Crawling Issues (3) Scope of crawl –not enough space for “all” pages –not enough time to visit “all” pages Solution: Visit “important” pages visited pages microsoft

7 Crawling Issues (4) Incremental crawling –how do we keep pages “fresh”? –how do we avoid crawling from scratch?

8 Crawl-ordering Experiment Domain: pages within Stanford Goal: Crawl HOT pages! –Experiment 1: the pages with highest PageRank (top 10%) –Experiment 2: the pages related to “admission” Experiment: –How many HOT pages did we collect when we crawl, say, 20% of the Stanford domain?

9 Experiment 1: Top 10% PageRank pages

10 Experiment 2: Admission-related Pages

11 Web Evolution Experiment How often does a web page change? How long does it take for 50% of the web to change? How do we model web changes?

12 Experimental Setup February 17 to June 24, sites visited (with permission) –identified 400 sites with highest “page rank” –contacted administrators 720,000 pages collected –3,000 pages from each site daily –start at root, visit breadth first (get new & old pages) –ran only 9pm - 6am, 10 seconds between site requests

13 How Often Does a Page Change? Example: 50 visits to page, 5 changes  average change interval = 50/5 = 10 days Is this correct? 1 day changes page visited

14 Average Change Interval fraction of pages

15 Average Change Interval — By Domain fraction of pages

16 Time for a 50% Change days fraction of unchanged pages

17 Modeling Web Evolution Poisson process with rate T is time to next event f T (t) = e - t (t > 0)

18 Change Interval of Pages for pages that change every 10 days on average interval in days fraction of changes with given interval Poisson model

19 Change Metrics Freshness –Freshness of element e i at time t is F( e i ; t ) = 1 if e i is up-to-date at time t 0 otherwise eiei eiei... webdatabase –Freshness of the database S at time t is F( S ; t ) = F( e i ; t )  N 1 N i=1

20 Change Metrics Age –Age of element e i at time t is A( e i ; t ) = 0 if e i is up-to-date at time t t - (modification e i time) otherwise eiei eiei... webdatabase –Age of the database S at time t is A( S ; t ) = A( e i ; t )  N 1 N i=1

21 Change Metrics F(e i ) A(e i ) time update refresh F( S ) = lim F(S ; t ) dt  t 1 t 0 t  F( e i ) = lim F(e i ; t ) dt  t 1 t 0 t  Time averages: similar for age...

22 Questions How often should we revisit pages to 80% of pages up-to-date? How should we revisit pages when they change at different frequencies?

23 Freshness vs. Revisit Frequency r = / f = average change frequency / average visit frequency

24 Age vs. Revisit Frequency r = / f = average change frequency / average visit frequency = Age / time to refresh all N elements

25 Trick Question Two page database e 1 changes daily e 2 changes once a week Can visit pages once a week How should we visit pages? –e 1 e 1 e 1 e 1 e 1 e 1... –e 2 e 2 e 2 e 2 e 2 e 2... –e 1 e 2 e 1 e 2 e 1 e 2... [uniform] –e 1 e 1 e 1 e 1 e 1 e 1 e 2 e 1 e 1... [proportional] –? e1e1 e2e2 e1e1 e2e2 web database

26 Proportional Often Not Good! Visit fast changing e 1  get 1/2 day of freshness Visit slow changing e 2  get 1/2 week of freshness Visiting e 2 is a better deal!

27 Selecting Optimal Refresh Frequency Analysis is complex Shape of curve is the same in all cases Holds for any distribution g( )

28 Optimal Refresh Frequency for Age Analysis is also complex Shape of curve is the same in all cases Holds for any distribution g( )

29 Comparing Policies Based on Statistics from experiment and revisit frequency of every month

30 Crawler Types In-place vs. shadow Steady vs. batch eiei eiei... webdatabase eiei... shadow database time crawler on crawler off

31 Comparison: Batch vs. Steady batch mode in-place crawler steady in-place crawler crawler running

32 Shadowing Steady Crawler crawler’s collection current collection without shadowing

33 Shadowing Batch Crawler crawler’s collection current collection without shadowing

34 Experimental Data  Freshness Pages change on average every 4 months Batch crawler works one week out of

35 Building an Incremental Crawler Variable visit frequencies (but with care…) Steady In-place improves freshness! Also need: –Page change frequency estimator –Page replacement policy –Page ranking policy (what are ‘important” pages?) See papers....

36 Summary Selecting “important pages:” Intuitive policy performs well Maintaining the collection fresh: Intuitive policy does not perform well

37 The End Thank you for your attention... For more information, visit Follow “Searchable Publications” link, and search for author = “Cho”