Thanks to Bill Arms, Marti Hearst Documents

Last time Size of information –Continues to grow IR an old field, goes back to the ‘40s IR iterative process Search engine most popular information retrieval model Still new ones being built

Focus on documents Document will be what we: –Crawl (harvest) –Index –Retrieve with query –Evaluate –Rank IR iterative process

IR is an Iterative Process Repositories Workspace Goals

User’s Information Need Parse Query text input

Index Pre-process Collections

User’s Information Need Index Pre-process Parse Collections Rank or Match Query text input

User’s Information Need Index Pre-process Parse Collections Rank or Match Query text input Query Reformulation Evaluation

Definitions Collections consist of Documents Document –The basic unit which we will automatically index usually a body of text which is a sequence of terms –has to be digital Tokens or terms –Basic units of a document, usually consisting of text semantic word or phrase, numbers, dates, etc Collections or repositories –particular collections of documents –sometimes called a database Query –request for documents on a topic

Collection vs documents vs terms Document Collection Terms or tokens

What is a Document? A document is a digital object with an operational definition –Indexable –Can be queried and retrieved. Many types of documents –Text or part of text –Image –Audio –Video –Blogs –Data – –Tweet –Etc.

Text Documents A text digital document consists of a sequence of words and other symbols, e.g., punctuation. The individual words and other symbols are known as tokens or terms. A textual document can be: Free text, also known as unstructured text, which is a continuous sequence of tokens. Fielded text, also known as structured text, in which the text is broken into sections that are distinguished by tags or other markup. Example?

Why the focus on text? Language is the most powerful query model Language can be treated as text –Text has many interesting properties Others?

Information Retrieval from Collections of Textual Documents Major Categories of Methods 1.Exact matching (Boolean) 2.Ranking by similarity to query (vector space model) 3.Ranking of matches by importance of documents (PageRank) 4.Combination methods What happens in major search engines

Text Based Information Retrieval Most matching methods are based on Boolean operators. Most ranking methods are based on the vector space model. Web search methods combine vector space model with ranking based on importance of documents. Many practical systems combine features of several approaches. In the basic form, all approaches treat words as separate tokens with minimal attempt to interpret them linguistically.