Web Search – Summer Term 2006 I. General Introduction (c) Wolfgang Hürst, Albert-Ludwigs-University

Introduction: Search What is “search” (by machine)? Data bases: Relational data bases, SQL, … Search in structured data Information Retrieval Search in un- (or semi-)structured data Example: -Archive ‘All s with sender from April 1 st -3 rd, 2006’ Search in exactly specified (meta) data ‘All s that are somehow related to project x’ Search in unspecified and unstructured body

Information Retrieval (IR) Information Retrieval (IR) deals with the representation, storage, organization of, and access to information items. (Page 1, Baeza-Yates und Ribeiro-Neto [1]) Information Retrieval (IR) = Part of computer science which studies the retrieval of information (not data) from a collection of written documents. The retrieved documents aim at satisfying a user information need usually expressed in natural language. (Glossary, page 444, Baeza-Yates & Ribeiro-Neto [1]) Note: Many other definitions exist Generally, all share this common view: INFORMATION QUERY DATA / DOCUMENTS INFORMATION NEED

DOCUMENTS USERDATA SEARCH PROCESS INFORMATION RETRIEVAL SYSTEM

INFORMATION NEED DOCUMENTS INFORMATION RETRIEVAL SYSTEM USERDATA SEARCH PROCESS

INFORMATION NEED DOCUMENTS INFORMATION RETRIEVAL SYSTEM QUERY RESULT QUERY PROCESSING & SEARCHING & RANKING INDEXING INDEX USERDATA SEARCH PROCESS

Information Retrieval (IR) Main problem: Unstructured, imprecisely, and imperfectly defined data But also: The whole search process can be characterized as uncertain and vague Hence: Information is often returned in form of a sorted list (docs ranked by relevance ). INFORMATION QUERY DATA / DOCUMENTS INFORMATION NEED

‘Data Retrieval’ vs. ‘IR’ Source: C. J. van RIJSBERGEN: INFORM. RETRIEVAL ( DATA RETRIEVALINFORM. RETRIEVAL MATCHINGEXACT MATCHPARTIAL / BEST MATCH INFERENCEDEDUCTIONINDUCTION MODELDETERMINISTICPROBABILISTIC CLASSIFICATIONMONOTHETICPOLYTHETIC QUERY LANGUAGEARTIFICIALNATURAL QUERY SPECIFICATION COMPLETEINCOMPLETE ITEMS WANTEDMATCHINGRELEVANT ERROR RESPONSESENSITIVEINSENSITIVE

Summary of most imporant terms Query = The expression of the user information need in the input language provided by the information system. The most common type of input language simply allows the specification of keywords and of a few boolean connectivities. (Glossary, page 449, Baeza-Yates & Ribeiro-Neto [1]) Index = A data structure built on the text to speed up searching. (Glossary, page 443, Baeza-Yates & Ribeiro-Neto [1]) The concept of relevance = Measure to quantify relevance of a particular document for a particular user in a particular situation.

LOGICAL VIEW OF THE DOCUMENTS (INDEX) IR Process: Tasks Involved INFORMATION NEEDDOCUMENTS User Interface PERFORMANCE EVALUATION QUERY QUERY PROCESSING (PARSING & TERM PROCESSING) LOGICAL VIEW OF THE INFORMATION NEED SELECT DATA FOR INDEXING PARSING & TERM PROCESSING SEARCHING RANKING RESULTS DOCS. RESULT REPRESENTATION

Evaluation of IR Systems Standard approaches for algorithm and computer system evaluation Speed / processing time Storage requirements Correctness of used algorithms But most importantly Performance, effectiveness Questions: What is a good / better search engine? How to measure search engine quality? Etc.

Evaluation of IR Systems Another important issue: Usability, users’ perception User 1 & system 1: ‘It took me 10 min to find the information.’ Example: User 2 & system 2: ‘It took me 14 min to find the information.’

Evaluation of IR Systems Another important issue: Usability, users’ perception User 1 & system 1: ‘It took me 10 min to find the information. Those were the worst 10 minutes of my life. I really hate this system!’ Example: User 2 & system 2: ‘It took me 14 min to find the information. I never had so much fun using any search engine before!’

Some Historical Remarks 1950s: Basic idea of searching text with a computer SOURCE: AMIT SINGHAL ‘MODERN INFORMATION RETRIEVAL: A BRIEF OVERVIEW’ (CH. 1), IEEE BULLETIN, s: Key developments, e.g. The SMART system (G. Salton, Harvard/Cornell) The Crainfield evaluations 1970s and 1980s: Advancements of basic ideas But: mainly with small test collections 1990s: Establishment of TREC (Text Retrieval Conference) series (since 1992 till today) Large text collections, expansion to other fields and areas, e.g. spoken document retrieval, non-english or multi-lingual retrieval, information filtering, user interactions, WWW, video retrieval, etc.

Information Retrieval & Web Search Historically, IR was mainly motivated by text search (libraries, etc.) Today: Various other areas and data, e.g. multi media (images, video, etc.), WWW, etc. Web search : perfect example for an IR system Goal: Find best possible results (web pages) based on a) Unstructured, heterogeneous, semistructured data b) Imprecise, ambiguous, short queries (Note: ‘Best possible results‘ is also a very vague specification of the ultimate goal) But: Very different from traditional IR tasks!

Characteristics of the Web Size : The web is big! An there are lots of users! Documents : Extreme variety regarding formats, structure, quality, etc. Users : Very different skills & intensions, e.g. Find all information about related patents Find some good tourist inform. about Paris Find the phone no. of the tourist office Location : The web is a distributed system Spam : Expect manipulation instead of cooperation from the document providers Dynamic : The web keeps growing & changing

Web Search Web search is an active research area with high economical impact Many open questions & challenges for research: Improving existing systems, adapting to new scenarios (more data, spam, …), new challenges (diff. data formats, multimedia, …), new tasks (desktop search, personalization, …), etc. Many other approaches & techniques exist, e.g. Clustering, specialized search engines, meta search engines, etc. We will cover some of this here, i.e. …

Web Search Course: Rough Outline Traditional (text) retrieval: Index generation (data structures), text processing, ranking (TF*IDF, …), models (Boolean, Vector Space, Probabilistic), evaluation (precision & recall, TREC, …) Only most important concepts as required for main part of the course, i.e.: Web search (special case of IR): Special characteristics of the web, ranking (PageRank, HITs, …), crawling (Spiders, Robots), indexing, and some selected topics

Text books about (text) IR [1] RICARDO BAEZA-YATES, BERTHIER RIBEIRO-NETO: ‘MODERN INFORMATIN RETRIEVAL’, ADDISON WESLEY, 1999 [2] WILLIAM B. FRAKES, RICARDO BAEZA-YATES (EDS.): ‘INFORMATION RETRIEVAL – DATA STRUCTURES AND ALGORITHMS’, P T R PRENTICE HALL, 1992 [3] C. J. VAN RIJSBERGEN: ‘INFORMATION RETRIEVAL’, 1979, AVAILABLE ONLINE AT [4] I. WITTEN, A. MOFFAT, T. BELL: ‘MANAGING GIGABYTES’, MORGAN KAUFMANN PUBLISHING, 1999 EXCERPTS FROM A NEW BOOK ‘INTRODUCTION TO INFORMATION RETRIEVAL’ BY C. MANNING, P. RAGHAVAN, H. SCHÜTZ (TO APPEAR 2007) ARE AVAILABLE ONLINE AT Only certain topics will be covered in this course. No books on web search, but selected articles will be recommended in the lecture