1 CS 430 / INFO 430 Information Retrieval Lecture 4 Searching Full Text 4

2 Course Administration Assignment 1 has been posted. It is a programming assignment and is due on Saturday, September 17 at midnight. Follow the submission instructions carefully. Send questions to

3 Organization of Files for Full Text Searching TermPointer to postings ant bee cat dog elk fox gnu hog Inverted lists Word listPostings Documents store

4 Representation of Inverted Files Document store: Stores the documents. Important for user interface design. [Repositories for the storage of document collections are covered in CS 431.] Word list (vocabulary file): Stores list of terms (keywords). Designed for searching and sequential processing, e.g., for range queries, (lexicographic index). Often held in memory. Postings file: Stores an inverted list (postings list) of postings for each term. Designed for rapid merging of lists and calculation of similarities. Each list is usually stored sequentially.

5 Document Store The Documents Store holds the corpus that is being indexed. The corpus may be: primary documents, e.g., electronic journal articles or Web pages surrogates, e.g., catalog records or abstracts, which refer to the primary documents

6 Document Store The storage of the document store may be: Central (monolithic) - all documents stored together on a single server (e.g., library catalog) Distributed database - all documents managed together but stored on several servers (e.g., Medline, Westlaw, Dialog) Highly distributed - documents stored on independently managed servers (e.g., the Web) Each requires: a document ID, which is a unique identifier that can be used by the search system to refer to the document, and a location counter, which can be used to specify location of words or characters within a document.

7 Documents Store for Web Search Systems For Web search systems: A document is a Web page. The documents store is the Web. The document ID is the URL of the document. Indexes are built using a web crawler, which retrieves each page on the Web for indexing. After indexing, the local copy of each page is discarded, unless stored in a cache. (In addition to the usual word list and postings file the indexing system stores contextual information, which will be discussed in a later lecture.)

8 Inverted File Inverted file: An inverted file is list of search terms that are organized for associative look-up, i.e., to answer the questions: In which documents does a specified search term appear? Where within each document does each term appear? (There may be several occurrences.) The word list and the postings file together provide an inverted file system for free text searching. In addition, they contain the data needed to calculate weights and information that is used to display results.

9 Inverted File -- Basic Concept Word Document abacus actor aspen5 atoll11 34 Stop words are removed before building the index.

10 Inverted List -- Definitions Inverted List: A list of all the entries in an inverted file that apply to a specific word, e.g. abacus Posting: Entry in an inverted list that applies to a single instance of a term within a document, e.g., there are three postings for "abacus": abacus 3 abacus 19 abacus 22

11 Use of Inverted Files for Calculating Similarities In the term vector space, if q is query and d j a document, then q and d j have no terms in common iff q.d j = To calculate all the non-zero similarities find R. the set of all the documents, d j, that contain at least one term in the query: 2. Merge the inverted lists for each term t i in the query, with a logical or, to establish the set, R. 3. For each d j  R, calculate Similarity(q, d j ), using appropriate weights. 4. Return the elements of R in ranked order.

12 Enhancements to Inverted Files -- Concept Location: Each posting holds information about the location of each term within the document. Uses user interface design -- highlight location of search term adjacency and near operators (in Boolean searching) Frequency: Each inverted list includes the number of postings for each term. Uses term weighting query processing optimization

13 Inverted File -- Concept (Enhanced) WordPostings DocumentLocation abacus actor aspen atoll Inverted list for term actor

14 Lexicographic Order It is important that the word list can be processed sequentially, i.e, in alphabetic order. To search with wild cards, e.g. comp*, which expands to every term beginning with the letters "comp". To list results for browsing lists of search terms. This is a special case of of the mathematical concept of lexicographic order.

15 Postings File The postings file stores the elements of a sparse matrix, the term assignment matrix, with weights. It is stored as a separate inverted list for each column, i.e., a list corresponding to each term in the index file. Each element in an inverted list is called a posting, i.e., the occurrence of a term in a document Each list consists of one or many individual postings.

16 Postings File: A Linked List for Each Term 1 abacus actor aspen atoll A linked list for each term is convenient to process sequentially, but slow to update when the lists are long.

17 Length of Postings File For a common term there may be very large numbers of postings for a given term. Example: 1,000,000,000 documents 1,000,000 distinct words average length 1,000 words per document postings By Zipf's law, the 10th ranking word occurs, approximately: (10 12 /10)/10 times = times

18 Postings File Merging inverted lists is the most computationally intensive task in many information retrieval systems. Since inverted lists may be long, it is important to match postings efficiently. Usually, the inverted lists will be held on disk and paged into memory for matching. Therefore algorithms for matching postings process the lists sequentially. For efficient matching, the inverted lists should all be sorted in the same sequence. Inverted lists are commonly cached to minimize disk accesses.

19 Data for Calculating Weights The calculation of weights requires extra data to be held in the inverted file system. For each term, t j and document, d i f ij number of occurrences of t j in d i For each term, t j n j number of documents containing t j For each document, d i m i maximum frequency of any term in d i For the entire document file ntotal number of documents

20 Word List: Individual Records for Each Term The record for term j in the word list contains: term j pointer to inverted (postings) list for term j number of documents in which term j occurs (n j )

21 Decisions in Building an Inverted File: Efficiency and Query Languages Some query options may require huge computation, e.g., Regular expressions If inverted files are stored in lexicographic order, comp* can be processed efficiently *comp cannot be processed efficiently Logical operators If A and B are search terms A or B can be processed by comparing two moderate sized lists (not A) or (not B) requires two very large lists

22 Efficiency Criteria Storage Inverted files are big, typically 10% to 100% the size of the collection of documents. Update performance It must be possible, with a reasonable amount of computation, to: (a) Add a large batch of documents (b) Add a single document Retrieval performance Retrieval must be fast enough to satisfy users and not use excessive resources.

23 Word List On disk If a word list is held on disk, search time is dominated by the number of disk accesses. In memory Suppose that a word list has 1,000,000 distinct terms. Each index entry consists of the term, some basic statistics and a pointer to the inverted list, average 100 characters. Size of index is 100 megabytes, which can easily be held in memory of a dedicated computer.

24 File Structures for Inverted Files: Linear Index Advantages Can be searched quickly, e.g., by binary search, O(log n) Good for lexicographic processing, e.g., comp* Convenient for batch updating Economical use of storage Disadvantages Index must be rebuilt if an extra term is added

25 File Structures for Inverted Files: Binary Tree elk beehog cat dog fox ant gnu Input: elk, hog, bee, fox, cat, gnu, ant, dog

26 File Structures for Inverted Files: Binary Tree Advantages Can be searched quickly Convenient for batch updating Easy to add an extra term Economical use of storage Disadvantages Less good for lexicographic processing, e.g., comp* Tree tends to become unbalanced If the index is held on disk, important to optimize the number of disk accesses

27 File Structures for Inverted Files: Binary Tree Calculation of maximum depth of tree. Illustrates importance of balanced trees. Worst case: depth = n O(n) Ideal case: depth = log(n + 1)/log 2 O(log n)

28 File Structures for Inverted Files: Right Threaded Binary Tree Threaded tree: A binary search tree in which each node uses an otherwise-empty left child link to refer to the node's in- order predecessor and an empty right child link to refer to its in-order successor. Right-threaded tree: A variant of a threaded tree in which only the right thread, i.e. link to the successor, of each node is maintained. Can be used for lexicographic processing. A good data structure when held in memory Knuth vol 1, 2.3.1, page 325.

29 File Structures for Inverted Files: Right Threaded Binary Tree dog bee ant cat gnu elk fox hog NULL

30 File Structures for Inverted Files: B-trees B-tree of order m: A balanced, multiway search tree: Each node stores many keys Root has between 2 and 2m keys. All other internal nodes have between m and 2m keys. If k i is the i th key in a given internal node -> all keys in the (i-1) th child are smaller than k i -> all keys in the i th child are bigger than k i All leaves are at the same depth

31 File Structures for Inverted Files: B-trees B-tree example (order 2) Every arrow points to a node containing between 2 and 4 keys. A node with k keys has k + 1 pointers

32 File Structures for Inverted Files: B + -tree A B-tree is used as an index Data is stored in the leaves of the tree, known as buckets D 9 D D 54 D D Example: B + -tree of order 2, bucket size 4 (Implementation of B + -trees is covered in CS 432.)