Compressed Prefix Sums O’Neil Delpratt Naila Rahman Rajeev Raman

Given static sequence of positive integers, such that Support Problem Minimising space for storing according to some compressibility criteria. Supporting Sum rapidly Trivial solution Explicitly store Sum values Requires: bits, support Sum in O(1) time PSDS Prefix Sums problem

Motivation: Inverted List Locations of keywords in main text Positive sequence of strictly increasing integers Term=Moses: 650, 687, 696 Bible doc: Moses…… ….Moses… ………………..Moses…

Store differences Significant space saving, standard technique [Managing Gigabytes, Witten, Moffat, Bell] We can store in PSDS => th location of keyword Direct access to individual, help to answer conjunction queries. Inverted List

String Collection Collection of non-empty stings, Store in PSDS, where Concatenate strings Store concatenated string in array Store concatenated string in compress self-index e.g. FM-index or CSA Get th string Offset =, Length = Text string: selmanjava3d programming2000” Offsets:0,6, 13,10, 24

URLs Web search engines with large database of URLs. URLs are strings URLs are 60 chars long average compressed fairly well Explicit pointer for each URL requires 64 bits

XML Documents XML Doc: selman java3d programming 2000 book titleauthoryear #doc “Java3d programming” “[cr][sp]” “selman” 5 “2000” “[cr][sp]” selman java3d programming 2000  Text nodes chars in length in avg. Compressed to average 3-4 bytes  32-bit pointer overhead for each string (naive)

Related Work [CJM]: Clark thesis,Jacobson FOCS 89, Clark,Munro SODA 96 [Geary et al.]: Geary, Raman, Rahman CPM 04,TCS 06 [Kim et al.]: Kim, Chae Na, Kim, Park WEA 05 [Gupta et al. (a)]: Gupta, Hon, Shah, Vitter DCC 06 [Gupta et al. (b)]: Gupta, Hon, Shah, Vitter WEA 06 [GV]: Grossi, Vitter STOC 00, SICOMP 05 [MG]: Witten, Moffat, Bell, Managing Gigabytes

Select Space usage: bits Time:. Bitvector Representation Write in unary is “0001” |B|= m bits B: position of the iith 1 bit in B [CJM, KIM et al.] } B:

Given the # of 1s in B is n different bit sequences Lower bound to store all L sequences bits. space usage is based on Average. Could we do better? Succinct Bound

Data-aware encoding Exploit skewed distribution. Self-delimiting encodings of values. concat. unary and binary.. add up to, average value is then.,

Data-aware encoding Golomb(b,x) Concat. in unary and in binary using or bits b=3 Golomb(3,9)= q=2 in unary(q+1)=001 and r=2 => Best encoding for inverted lists if - [MG]. - [Gupta et. al. (b)] Not achievable

a) GOLOMBSUCCINCT b) New Select DS c) Data aware PSDS. Space:, bits Time: d) Implementation and Experimental Evaluation Contributions of paper

If..- [GV, Elias] Succinct vs Golomb

Succinct PSDS Given Compute Lower-half: Lower order bits of, so we take bits Upper-half: Multiplicities bits. Upper-order bits, i.e= 0,1,1,2,2,4,5,6 Space usage: time: - [CJM] V: Simple to do Succinct i.e. 5= i.e. 5= 00101=> B: e.g [GV, Elias] get(B,4)=

New select DS = position of the th 1 bit in bitstring B of length N Extracted string & contracted string [Kim et al.] Remove zero blocks [Geary et al.]: Fast select – every block has at least a single 1 bit. Block of zeros 001…… …… XX A: P: A’:

New select DS Assume BS of N bits Results Select & rank: O(1) time, space: N+o(N) bits Select1 and select0 Partitioned BV [Delpratt, Raman, Rahman, WEA 06] In practice Joint fastest with CJM

New select DS TypicalWorst-case NewDSCJMKIMNewDSCJMKIM Input BS(1- )NNNNNN Select(1- )0.94N(1+ )0.52N(1+ )0.63N0.94N2.77N1.17N rank0.03N0.5N0.25N0.02N0.5N0.25N sum~2N ~1.94N4.27N2.42N Reliable space bound Speed evaluation: Orders.xml NewDS=0.101, CJM=0.105, KIM=0.178 oper./per sec

Data aware tree PSDS Results Space usage: bits, Time: [Gupta et al. (b)] achieved, Time:

Delete larger child Indicate nodes removed. n-1 extra bits Data aware tree PSDS

Implemented Succinct, Explicit- & Succinct- PSDS Gamma tree PSDS Remove right child nodes vs largest node = negligible difference Tree is slow Data: Lengths of Text node strings Compressibility measures Succinct measure close to GOLOMB measure Implementation and Experimental Evaluation File#Text nodesGapGOLOMBSucc. orders.xml150K Xpath.xml1.7m per node

Experimental Evaluation Results Comparative space usage for data structures Linux machine, 8 million random operation calls, 10 repeated runs Time: sec per operation FileSuccinctExplicit-Succinct- orders.xml Xpath.xml Succinct PSDS performed best

Compression of Prefix sums is important Space efficient data-aware PSDS Succinct PSDS was more appropriate in our application New select DS Future improvements Succinct- more competitive: single -decode x20 faster than single select To data aware tree PSDS Conclusions and future work

Thank you!