Suffix arrays

Suffix array We loose some of the functionality but we save space. Let s = abab Sort the suffixes lexicographically: ab, abab, b, bab The suffix array gives the indices of the suffixes in sorted order 2031

How do we build it ? Build a suffix tree Traverse the tree in DFS, lexicographically picking edges outgoing from each node and fill the suffix array. O(n) time

How do we search for a pattern ? If P occurs in T then all its occurrences are consecutive in the suffix array. Do a binary search on the suffix array Takes O(mlogn) time

Example Let S = mississippi i ippi issippi ississippi mississippi pi ppi sippi sisippi ssippi ssissippi L R Let P = issa M

How do we accelerate the search ? L R Maintain = LCP(P,L) Maintain r = LCP(P,R) Assume  ≥ r M r

L R M r If = r then start comparing M to P at + 1

L R M r > r

L R M r Someone whispers LCP(L,M) LCP(L,M) >

L R M r Continue in the right half LCP(L,M) >

L R M r LCP(L,M) <

L R M r LCP(L,M) < Continue in the left half

L R M r LCP(L,M) = start comparing M to P at + 1

Analysis If we do more than a single comparison in an iteration then max(, r ) grows by 1 for each comparison  O(m + logn) time

Construct the suffix array without the suffix tree

Linear time construction Recursively ? Say we want to sort only suffixes that start at even positions ?

Change the alphabet You in fact sort suffixes of a string shorter by a factor of 2 ! Every pair of characters is now a character

Change the alphabet a$0 aa1 ab2 b$3 ba4 bb5 $ a b a aa b 2 12

But we do not gain anything…

Divide into triples $ yab ba da b bad o abb ada bba do$

Divide into triples $ yab ba da b bad o abb ada bba do$ $ yab ba da b bad o bba dab bad o$$

Sort recursively 2/3 of the suffixes $ yab ba da b bad o abb ada bba do$ bba dab bad o$$ $ yab ba da b bad o

Sort the remaining third $ yab ba da b bad o (b, 2)(a, 5) (a, 7) (y, 1) (b, 2) (a, 5) (a, 7) (y, 1) 

Merge $ yab ba da b bad o

Merge $ yab ba da b bad o

Merge $ yab ba da b bad o

Merge $ yab ba da b bad o

Merge $ yab ba da b bad o

Merge $ yab ba da b bad o

Merge $ yab ba da b bad o

Merge $ yab ba da b bad o

Merge $ yab ba da b bad o

Merge $ yab ba da b bad o

Merge $ yab ba da b bad o

summary $ yab ba da b bad o When comparing to a suffix with index 1 (mod 3) we compare the char and break ties by the ranks of the following suffixes When comparing to a suffix with index 2 (mod 3) we compare the char, the next char if there is a tie, and finally the ranks of the following suffixes

Compute LCP’s $ yab ba da b bad o abbado$ abbadabbado$ adabbado$ ado$ badabbado$ bado$ bbadabbado$ bbado$ dabbado$ do$ o$ yabbadabbado$

Crucial observation $ yab ba da b bad o abbado$ abbadabbado$ adabbado$ ado$ badabbado$ bado$ bbadabbado$ bbado$ dabbado$ do$ o$ yabbadabbado$ LCP(i,j) = min {LCP(i,i+1),LCP(i+1,i+2),….,LCP(j-1,j)}

$ yab ba da b bad o abbado$ abbadabbado$ adabbado$ ado$ badabbado$ bado$ bbadabbado$ bbado$ dabbado$ do$ o$ yabbadabbado$ LCP(11,0) Find LCP’s of consecutive suffixes

$ yab ba da b bad o abbado$ abbadabbado$ adabbado$ ado$ badabbado$ bado$ bbadabbado$ bbado$ dabbado$ do$ o$ yabbadabbado$ LCP(8,2)

$ yab ba da b bad o abbado$ abbadabbado$ adabbado$ ado$ badabbado$ bado$ bbadabbado$ bbado$ dabbado$ do$ o$ yabbadabbado$ LCP(9,3)

$ yab ba da b bad o abbado$ abbadabbado$ adabbado$ ado$ badabbado$ bado$ bbadabbado$ bbado$ dabbado$ do$ o$ yabbadabbado$ LCP(6,4)

$ yab ba da b bad o abbado$ abbadabbado$ adabbado$ ado$ badabbado$ bado$ bbadabbado$ bbado$ dabbado$ do$ o$ yabbadabbado$ LCP(7,5)

$ yab ba da b bad o abbado$ abbadabbado$ adabbado$ ado$ badabbado$ bado$ bbadabbado$ bbado$ dabbado$ do$ o$ yabbadabbado$ LCP(1,6)

$ yab ba da b bad o abbado$ abbadabbado$ adabbado$ ado$ badabbado$ bado$ bbadabbado$ bbado$ dabbado$ do$ o$ yabbadabbado$ LCP(2,7)

$ yab ba da b bad o abbado$ abbadabbado$ adabbado$ ado$ badabbado$ bado$ bbadabbado$ bbado$ dabbado$ do$ o$ yabbadabbado$ LCP(3,8) 3

$ yab ba da b bad o abbado$ abbadabbado$ adabbado$ ado$ badabbado$ bado$ bbadabbado$ bbado$ dabbado$ do$ o$ yabbadabbado$ LCP(4,9) 3 2

$ yab ba da b bad o abbado$ abbadabbado$ adabbado$ ado$ badabbado$ bado$ bbadabbado$ bbado$ dabbado$ do$ o$ yabbadabbado$ LCP(5,10) 3 21

$ yab ba da b bad o abbado$ abbadabbado$ adabbado$ ado$ badabbado$ bado$ bbadabbado$ bbado$ dabbado$ do$ o$ yabbadabbado$ LCP(10,11) 3 210

$ yab ba da b bad o abbado$ abbadabbado$ adabbado$ ado$ badabbado$ bado$ bbadabbado$ bbado$ dabbado$ do$ o$ yabbadabbado$ The starting position deceases by 1 in every iteration. So it cannot increase more than O(n) times Analysis

$ yab ba da b bad o We need more LCPs for search Linearly many, calculate the all bottom up

abc ab bc a $ $ ca$ cabbca$ bca$ bcabbca$ bbca$ a$ abcabbca$ abbca$ Another example

Think about the LCP which we know at any point in the algorithm Analysis A successful comparison increases it by one It decreases by one when iteration starts So the number of successful comparisons is O(n)

Burrows –Wheeler (bzip2) Currently best algorithm for text Basic Idea: Sort the characters by their full context (typically done in blocks). This is called the block sorting transform. Use move-to-front encoding to encode the sorted characters. The ingenious observation is that the decoder only needs the sorted characters and a pointer to the first character of the original sequence.

S = abraca שלב I : יצירת מטריצה M שבנויה מכל ההזזות הציקליות של S: M = a b r a c a # b r a c a # a r a c a # a b a c a # a b r c a # a b r a a # a b r a c # a b r a c a

שלב II : מיון השורות בסדר לקסיקוגרפי : a b r a c a # b r a c a # a a c a # a b r c a # a b r a a # a b r a c # a b r a c a r a c a # a b LF L is the Burrows Wheeler Transform

a b r a c a # b r a c a # a r a c a # a b a c a # a b r c a # a b r a a # a b r a c # a b r a c a a b r a c a # b r a c a # a a c a # a b r c a # a b r a a # a b r a c # a b r a c a r a c a # a b Claim: Every column contains all chars. LF You can obtain F from L by sorting

a b r a c a # b r a c a # a a c a # a b r c a # a b r a a # a b r a c # a b r a c a r a c a # a b LF The “a’s” are in the same order in L and in F, Similarly for every other char.

From L you can reconstruct the string L # a r a c a b F # a r a c a b What is the first char of S ?

From L you can reconstruct the string L # a r a c a b F # a r a c a b What is the first char of S ? a

From L you can reconstruct the string L # a r a c a b F # a r a c a b abab

L # a r a c a b F # a r a c a b abr

Compression ? L # a r a c a b Compress the transform to a string of integers using move to front Then use Huffman to code the integers

a b r a c a # b r a c a # a r a c a # a b a c a # a b r c a # a b r a a # a b r a c # a b r a c a a b r a c a # b r a c a # a a c a # a b r c a # a b r a a # a b r a c # a b r a c a r a c a # a b Why is it good ? LF Characters with the same (right) context appear together

a b r a c a # b r a c a # a r a c a # a b a c a # a b r c a # a b r a a # a b r a c # a b r a c a a b r a c a # b r a c a # a a c a # a b r c a # a b r a a # a b r a c # a b r a c a r a c a # a b Sorting is equivalent to computing the suffix array. LF Can encode and decode in linear time

p i#mississi p p pi#mississ i s ippi#missi s s issippi#mi s s sippi#miss i s sissippi#m i i ssippi#mis s m ississippi # i ssissippi# m A useful tool: L  F mapping # mississipp i i #mississip p i ppi#missis s FL How do we map L’s onto F’s chars ?... Need to distinguish equal chars... unknown To implement the LF-mapping for a char  at position j in L we need the oracle occ( , j )

fr occ=2 [lr-fr+1] Substring search using the BWT ( Count the pattern occurrences ) #mississipp i#mississip ippi#missis issippi#mis ississippi# mississippi pi#mississi ppi#mississ sippi#missi sissippi#mi ssippi#miss ssissippi#m ipssm#pissiiipssm#pissii L mississippi P = si First step fr lr Inductive step: Given fr,lr for P[j+1,p] Œ Take c=P[j] P[ j ]  Find the first c in L[fr, lr] Ž Find the last c in L[fr, lr]  L-to-F mapping of these chars lr rows prefixed by char “i” s s unknown Occ() oracle is enough slide stolen from Paolo

fr occ=2 [lr-fr+1] Substring search using the BWT ( Count the pattern occurrences ) #mississipp i#mississip ippi#missis issippi#mis ississippi# mississippi pi#mississi ppi#mississ sippi#missi sissippi#mi ssippi#miss ssissippi#m ipssm#pissiiipssm#pissii L mississippi # 1 i 2 m 7 p 8 s 10 C Available info P = si First step fr lr Inductive step: Given fr,lr for P[j+1,p] Œ Take c=P[j] P[ j ]  Find the first c in L[fr, lr] Ž Find the last c in L[fr, lr]  L-to-F mapping of these chars lr rows prefixed by char “i” s s slide stolen from Paolo

fr occ=2 [lr-fr+1] Substring search using the BWT ( Count the pattern occurrences ) #mississipp i#mississip ippi#missis issippi#mis ississippi# mississippi pi#mississi ppi#mississ sippi#missi sissippi#mi ssippi#miss ssissippi#m ipssm#pissiiipssm#pissii L mississippi # 1 i 2 m 7 p 8 s 10 C Available info P = si First step fr lr rows prefixed by char “i” s s slide stolen from Paolo What if someone whispers how many “s” we have up to index 2 and up to index 5: occ(s,2), occ(s,5) ? fr = C[s] + occ(s,2) + 1 lr = C[s] + occ(s,5)

occ( a, j ) ipssm#pissiiipssm#pissii L occ(s,4) = 2

Make a bit vector for each character ipssm#pissiiipssm#pissii L occ(s,4) = rank(4) rank(i) = how many ones are there before position i ?

How do you answer rank queries ? rank(i) = how many ones are there before position i ? We can prepare a vector with all answers

Lets do it with O(n) bits per character Partition in 2n/log(n) blocks of size log(n)/ logn/2 257 Keep the answer for each prefix of the blocks There are “kinds” of blocks, prepare a table with all answers for each block

In our solution the bit vector takes Θ(n) bits and also the “additionals” take Θ (n) bits logn/2 257

Can we do it with smaller overhead : so additionals would take o(n) ? superblocks of size log 2 (n) log 2 n 7 13 Each block keeps the number of one in previous blocks that are in the same superblock

Analysis The superblock table is of size n/log (n) log 2 n 7 13 The block table is of size (loglog(n)) * n/log (n) The tables for the blocks √n log(n)loglog(n) So the additionals take o(n) space

Next step Do it without keeping the bit vectors themselves Instead keep only the compressed version of the text Saves a lot of space for compressible strings