Performance of Compressed Inverted Indexes
Reasons for Compression Compression reduces the size of the index Compression can increase the performance of query evaluation operations
Factors Affecting Index Performance Retrieval time for index lists (index size) Complexity of decoding index lists
Standard Techniques Translate absolute location of terms into differences between locations Use bitwise encoding schemes such as Golomb-Rice or Elias coding Usually reduce an index to about 15% of the size of the collection Performance is generally equal or better than an uncompressed index
Articles Reviewed Compression of Inverted Indexes For Fast Query Evaluation Scholer, Williams, Yiannis and Zobel, 2002 School of Computer Science and Information Technology, RMIT University, Melbourne, Australia Index Compression vs. Retrieval Time of Inverted Files for XML Documents Fuhr and Govert, 2002 University of Dortmund, Germany
Article 1: Improving Performance Two techniques were chosen to attempt to improve the performance of compressed indexes: Optimization of existing bitwise compression routines Implementation of bytewise compression routines
Optimized Bitwise Compression Routines Improved existing code developed by Williams and Zobel Optimized for the Intel / Linux platform Decoding speed improved to 60% of that achieved by Williams and Zobel
Bytewise Compression Routines Integers are stored in standard binary form using only 7 bits of a byte Each integer only takes up as many bytes as necessary to store the integer 1 bit per byte is used as a flag to indicate that a byte is the final byte for the integer Decoding of the integers is much simpler than the complex bitwise encodings
Bitwise vs. Bytewise Bytewise encoding of indexes takes up nearly 20% of the original document size (33% more than bitwise encodings) Bytewise encoding provides query performance that is double that of the optimized bitwise encodings Even when the index is small enough to be stored in memory, bytewise encoding shows small improvements over uncompressed indexes
Article 2: Structured Indexes Most IR approaches in the past have ignored the structure and formatting of documents The widespread adoption of HTML and XML has created the need for improvements in structured IR
Inverted Indexes of XML Documents The document structure must be stored or referenced from the inverted index Standard schemes use a Path-In-List (PIL) approach; structure data is stored within the inverted list for each term Indexes are generally much larger than the original text when uncompressed
Compression of Inverted Lists Problem: the uncompressed PIL approach generates an index that is too large Two possible solutions were explored: Use bitwise compression schemes to compress the existing PIL representation Store only a pointer in the list that points into another data structure that models the document structure
XML Structure (XS) Tree The XS Tree is a compact representation of the structure of an XML document Size of XS Tree is generally 1-2% of the original document size XS Trees for an entire document collection can usually be kept in memory
Performance of PIL vs. XS Trees The XS Tree index, including the XS Trees, is generally 2-3 times smaller than the compressed PIL approach Both approaches yield indexes that are smaller than the document collection In both cases, compression results in retrieval performance that is far worse than uncompressed PIL. Retrieval performance of the XS Tree approach is times worse than that of the uncompressed PIL
Conclusions Retrieval performance is dependent on: the retrieval time of the index (index size) the complexity of decoding the index entries Scholer et. al. find the ideal balance with bytewise compression, which results in optimal retrieval times
Conclusions The XS Tree’s goal of compressing the size of the index is successful The complexity of decoding the XS Tree structure results in nearly unusable performance Future research should be undertaken to find a structure that is quicker to decode than the XS Tree