Web Search – Summer Term 2006 VII. Web Search - Indexing: Structure Index (c) Wolfgang Hürst, Albert-Ludwigs-University

Structure Index (Links) Represents the links between the indexed pages Important for - Relevance calculation (PageRank, HITS,...) - Crawling (importance metrics,...) and some other applications (Web mining, etc.) Most critical issues (again): - Size and rate of change Most important requirements: - Reduce space / compression - Support required operations (random and streaming access, add / delete) - Speed

The Web Graph The structure index represents the web graph : - Node = web page - Directed edge = link Common representation techniques for graphs: a) Adjacency matrix

The Web Graph The structure index represents the web graph : - Node = web page - Directed Edge = link Common representation techniques for graphs: b) Adjacency list

The Structure Index Example: The Connectivity Server [3] Based on a data structure that supports the following operations : - Given a URL u (or a set of URLs U), return a list of pages that point to u (U), i.e. its predecessors (back links) and a list of pages that are pointed to from u (U), i.e. its successors (forward links) - Given a set of URLs U and a distance, return the respective neighborhood of U in the graph

The Connectivity Server Nodes: Array (1 node = 1 element) Edges: - OUTLIST: Adjacency list (successors) - INLIST: Inverted adjacency list (predecessors) URL DATA- BASE PTR TO URLPTR TO INLISTPTR TO OUTLIST... NODE TABLE... INLIST TABLE... OUTLIST TABLE

The Connectivity Server (cont.) Additional data structure to map ULRs to IDs (and vice versa) ID = index in the lexicographically sorted list of all crawled URLs Advantage: Compression, i.e. delta-encoding Example: GANDALF.HTM 26 7 GRAB.COM/ 41 ORIGINAL TEXT DELTA ENCODING

The Connectivity Server (cont.) Problem: Need to scan all URLs because of delta encoding (i.e. saves space at cost of speed) Solution: Include Checkpoint URLs Another problem: Updates are hard to do Several other (newer) approaches exist that take into account (e.g.) the actual web structure

S-Node Representation [4] Observations on the web structure : - Link copying: Lots of clusters with nodes containing very similar adjacency lists - Domain and URL locality: A significant fraction of links on a page point to pages from the same domain - Page similarity: Pages that have very similar adjacency lists are likely to be related Idea: Make use of these observations, e.g. by grouping related pages / similar URLs

S-Node Representation - Example PARTITION P = {N 1, N 2, N 3 } N 1 = {P 1, P 2 } N 2 = {P 3 } N 3 = {P 4, P 5 } INTRA-NODES N i N2N2 N1N1 N3N3 SUPERNODE GRAPH

S-Node Representation - Example PARTITION P = {N 1, N 2, N 3 } N 1 = {P 1, P 2 } N 2 = {P 3 } N 3 = {P 4, P 5 } N2N2 N1N1 N3N INTRA-NODES N i SUPERNODE GRAPH POSITIVE SUPEREDGES NEGATIVE SUPER- EDGES

Creating partitions 1. Initial partition : Based on URL (top two levels of DNS), e.g ad.informatik.uni-freiburg.de URL Split : Split N i s based on URL prefixes, e.g Clustered Split : Use clustering algorithm to split partitions into groups with similar adjacency lists

References - Indexing [1] A. ARASU, J. CHO, H. GARCIA-MOLINA, A. PAEPCKE, S. RAGHAVAN: "SEARCHING THE WEB", ACM TRANSACTIONS ON INTERNET TECHNOLOGY, VOL 1/1, AUG Chapter 4 (Indexing) [2] S. BRIN, L. PAGE: "THE ANATOMY OF A LARGE-SCALE HYPERTEXTUAL WEB SEARCH ENGINE", WWW 1998 Chapter 4 (System Anatomy) [3] BHARAT, BRODER, HENZINGER, KUMAR, VENKATASUBRAMINAIN: "THE CONNECTIVITY SERVER: FAST ACCESS TO LINKAGE INFORMATION ON THE WEB", WWW 1998 [4] RAGHAVAN, GARCIA-MOLINA: "REPRESENTING WEB GRAPHS", STANFORD TECHNICAL REPORT 2002

General Web Search Engine Architecture CLIENT QUERY ENGINE RANKING CRAWL CONTROL CRAWLER(S) USAGE FEEDBACK RESULTS QUERIES WWW COLLECTION ANALYSIS MOD. INDEXER MODULE PAGE REPOSITORY INDEXES STRUCTUREUTILITYTEXT (CF. [1] FIG. 1)

The evolution of search engines 1st generation : Use only "on page", text data - Word frequency, language (AltaVista, Excite, Lycos, etc.) 2nd gen. : Use off-page, web-specific data - Link (or connectivity) analysis - Click-through data (what results people click on) - Anchor-text (how people refer to a page) From 1998 (made popular by Google but everyone now) TUTORIAL ON SEARCH FROM THE WEB TO THE ENTERPRISE, SIGIR 2002

Still experimental The evolution of search engines Semantic analysis - What is it about? Focus on user need, rather than on query - Corpus reflects user needs / expectations - Integrates multiple sources of data - Help the user create a good query Context determination - Spatial (user location/target location) - Query stream (previous queries) - Personal (user profile) - Explicit (vertical search) - Implicit (on altavista.de) TUTORIAL ON SEARCH FROM THE WEB TO THE ENTERPRISE, SIGIR rd gener. : Answer the need behind the query

Still experimental The evolution of search engines 3rd gener. : Answer the need behind the query (cont.) TUTORIAL ON SEARCH FROM THE WEB TO THE ENTERPRISE, SIGIR 2002 Helping the user - UI, spell checking, query refinement, query suggestion, syntax driven feedback, context help, context transfer, etc. Integration of search and text analysis

Example: Google 3rd gener. : Answer the need behind the query

Web Search Lecture - Schedule 1. Classic IR (Basics) 2. Classic IR Exercises 3. Web Search (Basics) 4. Web Search Exercises [June, 28th till July 12th] 5. Web Search (Selected Topics) [July, 18th till July 26th]

Web Search – Summer Term 2006 Web Search Basics - (Programming) Exercises (c) Wolfgang Hürst, Albert-Ludwigs-University

General Web Search Engine Architecture CLIENT QUERY ENGINE RANKING CRAWL CONTROL CRAWLER(S) USAGE FEEDBACK RESULTS QUERIES WWW COLLECTION ANALYSIS MOD. INDEXER MODULE PAGE REPOSITORY INDEXES STRUCTUREUTILITYTEXT (CF. [1] FIG. 1)

Programming Exercises Exercise sheet 1: Tools, Library (Lucene) Exercise sheet 2: Database (and text index) Exercise sheet 3: Index (structure index) Exercise sheet 4: Search (link-based ranking)

Web Search Lecture - Schedule 1. Classic IR (Basics) 2. Classic IR Exercises 3. Web Search (Basics) 4. Web Search Exercises [June, 28th till July 12th] 5. Web Search (Selected Topics) [July, 18th till July 26th]

New Lecturnity Player Advanced replay features (developed by us) Modification of replay speed (while preserving the pitch of the voice)