1 Searching the Web Baeza-Yates Modern Information Retrieval, 1999 Chapter 13

2 Introduction l Characterizing the Web l Three different forms »Search engines –AltaVista »Web directories –Yahoo »Hyperlink search –WebGlimpse

3 Challenges on the Web l Distributed data l Volatile data l Large volume l Unstructured and redundant data l Data quality l Heterogeneous data

4 Measuring the Web l The size of the Web (the number of hosts) »Netsizer, –2.7 million web servers, 65 million internet hosts, 1999 »Netcraft, –8 million web servers using different web servers, 1999 »Internet Domain Survey, –56 million internet hosts »WWW Consortium (W3C)

5 Other measures l The number of different institutions maintain Web »more than 40% of the number of Web servers l The number of Web pages »350 million in Jul [BB98, WWW7] –20,000 random queries based on a lexicon of 400,000 words extracted from Yahoo –the union of all answers from four search engines covered about 70% of the Web l The size of a page »5Kb on average with a median 2Kbs

6 Other measures (cont.) l The number of links in a page »5~15 links, 8 on average »80% of these home pages had fewer than 10 external links l Yahoo and other web directories are the glue of the Web l The size of Web size (in bytes) »5Kb*350 million=1.7 terabytes l The languages of the Web

7 Modeling the Web l Heaps’ and Zipf’s laws are also valid in the Web. »In particular, the vocabulary grows faster (larger  ) and the word distribution should be more biased (larger  ) l Heaps’ Law »An empirical rule which describes the vocabulary growth as a function of the text size. »It establishes that a text of n words has a vocabulary of size O(n  ) for 0<  <1 l Zipf’s Law »An empirical rule that describes the frequency of the text words. »It states that the i-th most frequent word appears as many times as the most frequent one divided by i , for some  >1

8 Zipf’s and Heaps’ Law Distribution of sorted word frequencies (left) and size of the vocabulary (right)

9 Search Engines l Centralized Architecture l Distributed Architecture l User Interface l Ranking l Crawling the Web l Indices

10 Typical Crawler-Indexer Architecture Query Engine (Ranking) Interface Crawler Indexer Index

11 Centralized Architecture

12 Centralized Architecture l HotBot, GoTo and Microsoft are powered by Inktomi l Magellan are powered by Excite’s internal engine l Others »Ask Jeeves, –simulates an interview »DirectHit, –ranks the Web pages in the order of their popularity

13 l Harvest »Gatherers: collect and extract indexing information from one or more Web servers »Brokers: provide the indexing mechanism and the query interface to the data data gathered »Netscape’s Catalog Server Distributed Architecture Broker Gatherer Broker User Web Object Cache Replication manager

14 User Interface l Query interface »AltaVista: OR »HotBot: AND l Answer interface »order by relevance »order by Url or date »option: find documents similar to each Web page

15 Ranking l Most search engines follow traditional »Boolean or Vector Model »Yuwono and Lee (1996) –Boolean spread –vector spread –most-cited l Hyperlink Information »WebQuery (CK97, WWW6) »Li98, Internet Computing »HITS (Kleinsberg, (SIAM98) »ARC (Cha98, WWW7) »PageRank, Google (BP98, WWW7)

16 Crawling the Web l Synonyms »spider, robot, crawler, etc. »Starting from a set of popular URLs »Partition the Web using country codes or Internet names l Crawling order »Depth-first, breadth-first »CG98, WWW7 l robot.txt »Guidelines for robot behavior includes what pages should not be indexed »e.g. dynamically generated pages, password protected pages

17 Indices l Variants of Inverted file »A short description of each Web page is complemented –creation data, size, the title and the first lines or a few headings –500bytes for each page*100million pages=50GB »30% of the text size –5KB for each page*100million pages*30%=150GB »compression –50GB l Binary Search on the sorted list of words of the inverted file

18 Indexing Granularity l Pointing to pages or to word positions is an indication of the granularity of the index »Use logical blocks instead of pages –reduce the size of the pointers (fewer blocks than documents) »Occurrences of a non-frequent word will be clustered in the same block –reduce the number of pointers l Queries are resolved as for inverted files »Obtaining a list of blocks that are then searched sequentially »Exact sequential search: 30Mb/sec »Glimpse in Harvest

19 Browsing in Web Directories

20 Combining Searching with Browsing l WebGlimpse WebGlimpse »attaches a small search box to the bottom of every HTML page »allows the search to cover the neighborhood of that page or the whole site without having to stop browsing »

21 MetaCrawlers

22 Metasearchers (cont.) l Client side metasearchers »WebCompass »WebSeeker »EchoSearch »WebFerret l Better ranking »Inquirus (LG98, WWW7) –NEC Research Institue metasearch engine

23 Dynamic Search and Software Agents l Fish search (Bra94, WWW2) » fall94.html l Shark search (HJM+98, WWW7) l Searching specific information »LaMacchia, WWW6, Internet fish construction kit »SiteHelper (NW97, WWW6) l Shopping robots »Jango »Junglee »Express

24 Summary l Characterizing the Web l Search engines »