Web Search – Summer Term 2006 VI. Web Search - Indexing (c) Wolfgang Hürst, Albert-Ludwigs-University

General Web Search Engine Architecture CLIENT QUERY ENGINE RANKING CRAWL CONTROL CRAWLER(S) USAGE FEEDBACK RESULTS QUERIES WWW COLLECTION ANALYSIS MOD. INDEXER MODULE PAGE REPOSITORY INDEXES STRUCTUREUTILITYTEXT (CF. [1] FIG. 1)

Types of (generic) indexes 1. Text index = "Traditional", text-based index "Inverted files have traditionally been the index structure choice of the web" [3] Main purpose: Identification and selection of relevant pages Special characteristics: - Size and rate of change - Consider anchor text and surrounding text

Types of (generic) indexes 2. Structure / link index = Description of the linkage between web pages Usually modeled as a graph (nodes = pages, directed edges = links) Main purpose: Provide structure information (esp. neighborhood relationships), usually to create the ranking Problem: Requires a scalable and efficient representation of a VERY large graph

3. Utility index : Stores additional, search engine dependent information needed for page selection and relevance estimation, e.g. - PageRank - Site index - special site-related characteristics etc. Main purpose: Usually to speed up processing time Types of (generic) indexes

Inverted File : Generally: term -> document (web page) - Posting (t, l) :pair of term t and location l - Sometimes: Payload field to store add. info In addition: Lexicon (dictionary) with - List of all terms in the index - Related statistics (IDF,...) Note: Similar to traditional IR but size and rate of change require special techniques Text Index (= Inverted File)

The WebBase System as an example for a distributed text index [1,3] DISTRIBUTORS INDEXERS WEB PAGES INTER- MEDIATE RUNS STAGE 1 STAGE 2 STATIS- TICIAN QUERY SERVERS INVERTED INDEX

DISTRIBUTORS INDEXERS QUERY SERVERS WebBase Architecture - 3 Types of Nodes WEB PAGES INTER- MEDIATE RUNS STAGE 1 STAGE 2 STATIS- TICIAN INVERTED INDEX......

WebBase Indexing Process - 2 Stages DISTRIBUTORS INDEXERS WEB PAGES INTER- MEDIATE RUNS STAGE 1 STAGE 2 STATIS- TICIAN QUERY SERVERS INVERTED INDEX

WebBase - Distributed inv. idx. organization DISTRIBUTORS INDEXERS WEB PAGES INTER- MEDIATE RUNS STAGE 1 STAGE 2 STATIS- TICIAN QUERY SERVERS INVERTED INDEX Two strategies : - Local inverted files - Global inverted files

WebBase - Parallelizing the indexing process DISTRIBUTORS INDEXERS WEB PAGES INTER- MEDIATE RUNS STAGE 1 STAGE 2 STATIS- TICIAN QUERY SERVERS INVERTED INDEX

Parallel index construction (Indexers) INPUT: STREAM OF WEB PAGES FROM REPOSITORY OUTPUT: SORTED RUNS / INTERMEDIATE RUNS (SORTED POSTINGS OF A SUBSET OF THE REPOSITORY) LOADINGFLUSHING MEMORY WEB PAGES MEMORY SORTED RUNS PROCESSING MEMORY PAR- SING, TOKE- NIZA- TION SOR- TING

Parallel index construction (Indexers) TIME L P F L P F L P F L P F L P F L P F Loading Processing Flushing Software pipeline to create sorted runs (multi-threaded execution)

WebBase - Collecting global statistics DISTRIBUTORS INDEXERS WEB PAGES INTER- MEDIATE RUNS STAGE 1 STAGE 2 STATIS- TICIAN QUERY SERVERS INVERTED INDEX

Coll. global statistics (Statistician) Avoid disk accesses (expensive!) Communication with the statistician only if data is already in memory (i.e. during merging or flushing) Avoid intensive communication between indexer and statistician Only send partly sorted (summarized) postings Two strategies to collect statistical info on term level: - ME strategy (during merging) - FL strategy (during flushing)

ME strategy CAT(6,2) (3,1) DOG(8,3) RAT(8,3) (4,1) CAT(4,2) (3,3) (7,1) DOG(5,2) (9,1) (DOG, 1) (CAT, 2) (RAT, 2) (DOG, 2) (CAT, 3) AGGRE- GATE INDEXERS (INVERTED LISTS) INDEXERS (LEXICON) STATISTICIAN (DOG, 3) (CAT, 5) (RAT, 2) DOG:3 CAT:5 RAT:2 DOG:3 CAT:5

FL strategy INDEXERS (SORTED RUNS) INDEXERS (LEXICON) (CAT, 1) (DOG, 2) HASH TABLE CAT(6,1) DOG(8,3) CAT(2,1) CAT(6,2) RAT(4,3) RAT (8,1) DOG(4,2) CAT(5,2) DOG(5,1) DOG(7,2) (CAT, 1) (DOG, 1) (CAT, 2) STATISTICIAN DOG:4 CAT:4 RAT:2 DOG:4 CAT:4 (RAT, 2) (DOG, 1) DOG? CAT? RAT? HASH TABLE DOG4 CAT4 RAT2 STATISTICIAN DURINGAFTER PROCESSING

Summary: ME vs. ML strategy General observations: - Relatively low overhead (both strategies) - Confirmed experimentally ("less than 5% for a 2 million page collection") ++--FL (FLUSHING) + -+ ME (MERGING) PARALLELISMMEMORY USAGE STATISTICIAN LOAD Summary of characteristics (+/-)

The WebBase System - Summary DISTRIBUTORS INDEXERS WEB PAGES INTER- MEDIATE RUNS STAGE 1 STAGE 2 STATIS- TICIAN QUERY SERVERS INVERTED INDEX

References - Indexing [1] A. ARASU, J. CHO, H. GARCIA-MOLINA, A. PAEPCKE, S. RAGHAVAN: "SEARCHING THE WEB", ACM TRANSACTIONS ON INTERNET TECHNOLOGY, VOL 1/1, AUG Chapter 4 (Indexing) [2] S. BRIN, L. PAGE: "THE ANATOMY OF A LARGE-SCALE HYPERTEXTUAL WEB SEARCH ENGINE", WWW 1998 Chapter 4 (System Anatomy) [3] S. MELNIK, S. RAGHAVAN, B. YANG, H. GARCIA-MOLINA: "BUILDING A DISTRIBUTED FULL-TEXT INDEX FOR THE WEB", ACM TRANSACTIONS ON INFORMATION SYSTEMS, VOL. 13/3, JULY 2001