Efficient Search in Large Textual Collections with Redundancy Jiangong Zhang and Torsten Suel Review by Newton Alex

Problem Searching over collections of data that include many different crawls and versions of each page – E.g. Searching the Internet archive, archives etc. Not feasible to provide full text search due to high cost of processing a query – E.g. Current indexing and query processing techniques when applied to say 10 successive crawls of the same URL will result in index sizes and query processing costs roughly 10 times that of single crawl

Proposed Solution A new and general framework that results in significant savings in the size of the inverted index and the performance of query processing for webpage collections with redundancies. Features – Content-dependent partitioning techniques, in particular Winnowing. – Non redundant indexing. Two policies with respect to indexing local sharing global sharing – Modification of Document-at-a-time query processing algorithm to take advantage of the fragment based indexes

Critique The paper does not described the data structures used or the hardware setup in detail. The framework supports deleting old unused fragments. Why is a delete required when we are interested in versioned systems? Since no duplicate fragments are maintained, deleting a fragment might result in removing fragments corresponding to other pages in the archive.

Relation to Course This paper is similar to the Google News paper. However, this paper doesn’t describe the data structures or the environment setup in detail Related to the concepts that were used in the Search engine project like inverted indexes, query matching etc. Proposes methods for creating efficient indexes for redundant data.