Trie A Trie, also called a prefix tree, is a tree that records the information of strings. It likes a dictionary of those strings. Advantages comparing to BST: –Search time. –Space. –Prefix matching. –Balance.
Suffix Trie and Suffix Tree A suffix tree, Tree(T), is a compact trie that represents all the suffixes of a string T.
Suffix Trie and Suffix Tree (cont.) a b b a a a a a b b b a baab ab abaab baab aab ab b Suffix TrieSuffix Tree
Suffix Trie and Suffix Tree (cont.) Let |T| = n Suffix trie needs O(n 2 ) space. –Consider T = a n/2 b n/2 Suffix tree needs O(n) space. –Since every symbol can cause only one branch, total nodes (edges) cannot exceed O(n). –In a node we only record the starting position of corresponding suffix. –In a edge we need not to record the whole substring but the starting and ending positions of the substring.
1,1 2,5 4,5 2,5 Suffix Trie and Suffix Tree (cont.) 1:abaab 2:baab 3:aab 4:ab 5:b Suffix Tree 1:abaab$ 3:aab$ 2:baab$ 5:b$ 2,2 3,5 a baab ab Suffix Tree 3,54:ab$ 2,2
Suffix Tree: Full Text Index P occurs in T P is a prefix of some suffix of T Path for P exists in Tree(T)
Linear Time Construction 1973, Weiner gave the first linear time algorithm. 1976, McCreight gave a more readable algorithm. They were all processing from right to left. In 1992, Ukkonen gave a left-to-right on-line algorithm.
On-line Construction of Suffix Trie We have O(n 2 ) time to do this, it is quite easy. We construct Trie i from Trie i-1. We need to record the current end points of every suffix.
a a b b Trie(abaaba) a b b a a a b b a a a a a a b b a a a a a b b b a b b a a a a a b b b a a a
Two Lemmas We call the current end points of suffix i as C i. 1.If some C i ever branch out it will never branch again. 2.If C i does not branch, those C j also does not branch for j > i.
Ideas to Find Suffix Tree Thus –If one branched we never consider it again. –The branches have an order from small indices suffix to large indices. For a suffix we have 3 phases: 1.Going along original tree. 2.Branching a new leaf. 3.Growing the leaf.
Ideas to Find Suffix Tree (cont.) 3 phases: 1.Going along original tree. 2.Branching a new leaf. 3.Growing the leaf. We just record the longest suffix that is not branch yet. When it branches, we try to find next suffix. To do this, we need suffix links.
Suffix Links Suffix link is a pointer from an internal node xS to another internal node S. x , and S = *. Suffix tree of “abaabab$”: a b a abab b b b a abaabab$ baabab$ aabab$ abab$ bab$ ab$ b$ $
Using Suffix Link to Find Common Substring (1) a b a abab b b b a abaabab$ baabab$ aabab$ abab$ bab$ ab$ b$ $ Consider abaaaba
Using Suffix Link to Find Common Substring (2) a b a abab b b b a abaabab$ baabab$ aabab$ abab$ bab$ ab$ b$ $ abaaaba abaa
Using Suffix Link to Find Common Substring (3) a b a abab b b b a abaabab$ baabab$ aabab$ abab$ bab$ ab$ b$ $ abaaaba baa
Using Suffix Link to Find Common Substring (4) a b a abab b b b a abaabab$ baabab$ aabab$ abab$ bab$ ab$ b$ $ abaaaba aa
Using Suffix Link to Find Common Substring (5) a b a abab b b b a abaabab$ baabab$ aabab$ abab$ bab$ ab$ b$ $ abaaaba
Using Suffix Link to Find Common Substring (6) a b a abab b b b a abaabab$ baabab$ aabab$ abab$ bab$ ab$ b$ $ abaaaba aaba
Using Suffix Link to Find Common Substring (7) a b a abab b b b a abaabab$ baabab$ aabab$ abab$ bab$ ab$ b$ $ abaaaba aba ba a
Construct Suffix Tree Like matching, we use suffix link to find next suffix.
Time Complexity For phase 3, we do nothing. For phase 1, we totally use O(n) time. For phase 2, we do O(n) times of branch. The only bottleneck is how fast can we find next position.
Time Complexity (cont.) Consider we branch at the red circle having distance t to its parent. Next suffix can at most pass through t internal nodes. Use amortized analysis, we can easily find total internal nodes the algorithm passing through is O(n). t steps
Time Complexity (cont.) Thus, constructing suffix tree needs O(n) time and O(n) space.
Applications Longest common substring. Repeating of a pattern in a string. Approximate matching. etc.
Suffix Array hattivatti attivatti ttivatti tivatti ivatti vatti atti tti ti i ε atti attivatti hattivatti i ivatti ti tivatti tti ttivatti vatti 11 7 2 1 10 5 9 4 8 3 6 Suffix array of hattivatti: (11, 7, 2, 1, 10, 5, 9, 4, 8, 3, 6) att binary search It can construct from suffix tree in linear time. (DFS)
Special Thanks to Esko Ukkonen Hsueh-I Lu Wikipedia Some figures of this slides are based on their slides.
Suffix Tree MUMmer 1.0 MUMmer 2.1 MUMmer 3.0 Conclusion
About The System - MUMmer For rapidly aligning whole genome sequences –Assumption: the sequences are closely related –Output: Alignment of the input sequences Highlighting the exact differences in the genomes –SNPs, insertions, significant repeats, tandem repeats, reversals –Main idea Suffix tree Longest increasing subsequence (LIS) Smith-Waterman alignment
Alignment step Perform a maximal unique match (MUM) decomposition of the two genomes -> Suffix tree Sort the MUMs, and extract the longest possible set of matches in the same order -> LIS Close gaps -> Smith-Waterman alignment Output!
Step 1 (cont’) Construct a suffix tree T for genome A Add the suffixes for genome B (implement: A+dummy character+B) Find out unique matching sequence: an internal node with exactly two child nodes, such that the child nodes are leaf nodes from different genomes Find out MUM: For highly similar genomes, set MUM>=50bp For more distantly related genomes, set MUM>=20bp
Step 2 Sort, LIS=> O(KlogK) => O(N) –K: the numbers of MUMs –K<
"name": "Step 2 Sort, LIS=> O(KlogK) => O(N) –K: the numbers of MUMs –K< O(KlogK) => O(N) –K: the numbers of MUMs –K<
Step 3 Process the gap into one of the four classes –SNP Genome A: cgtcataaagt Genome B: cgtcctaaagt –Insert Genome A: cgtctaaagtggggaaaactctgg Genome B: cgtctaaagt........ Ctctgg Transposition or simple insertions –Polymorphic regions Genome A: cgtctaaagtggggaaaactctgg Genome B: cgtctaaagta tgacaggctctgg Should be aligned –Repeat Genome A: aaggaaggaaggagct Genome B: aaggaagg.... agct
Result and Discussion Comparing two strain of tuberculosis –H37Rv and CDC1551 –>99% identical –Be able to catalog all SNPs all insertions of every length All tandem repeat with different copy numbers –Performance (DEC Alpha 4100) 5s for step1 45s for step 2 5s for step 3
Comparing two Mycoplasma genome –M.genitalium (580074nt) and M.pneumoniae (226000nt) –Performance (DEC Alpha 4100) 6.5s for step1 0.02s for step 2 116s for step 3 –FASTA: many hours
Result and Discussion (cont’) FASTA 25mers MUMmer
Result and Discussion (cont’) Comparing human and mouse –222930bp of human chromosome 12 (accession no. U47924) and 227538bp of mouse chromosome 6 (accession no. AC002397) – Performance -29s 1.6s for step1
MUMmer 2.1 Fast algorithms for large-scale genome alignment and comparison by Delcher, Phillippy, Carlton and Salzberg Nucleic Acids Research 2002 http://www.tigr.org/software/mummer/MUMmer2.pdf
Agenda Key in MUMmer 1 Improvements in MUMmer 2.1 Technical improvements in MUMmer 2.1 Application to DNA sequence alignment – Alignment of incomplete genomes
MUMmer 1 Key –Built a suffix tree containing 2 input sequences –Find all maximal unique matches (MUMs) between them. MUM (Maximal Unique Matches) –A subsequence occurred in 2 exactly matching copies, once in each input sequence –Cannot be extended in either direction
Improvements in MUMmer 2.1 Fast and less memory, by a factor of nearly three Able to align DNA or protein sequence MUM1MUM2 Time74s (1GHz)27s (1GHz) Mem293MB100MB To align 4.7 Mb genome of E. coli and 3.0Mb large chromosome of V.cholerae
Technical improvements in MUMmer 2.1 A reduction in amount of memory used to store suffix trees –Kurtz (1999) technique is used An alternative algorithm to find initial exact matches Cluster matches
Alternative to find initial exact matches MUMmer 1: –Built a suffix tree containing 2 input sequences MUMmer 2: –Chang-Lawler (1994) method is used running time is reduced –Built a suffix tree storing only one sequence (reference) –2nd sequence (query) streamed against the suffix tree memory usage is reduced by at least half once the suffix tree is built, arbitrarily long/multiple queries can be streamed.
Alternative to find initial exact matches (cont.) MUMmer 2 (cont.): –Identify where the query sequence would branch off from the tree, to find all matches –Unique match Wherever a branch occurs at a tree position with just a single leaf beneath it –Maximal match Using suffix links to find next match (extended match) By checking the character immediately preceding the start of this match, we can determine whether it is a maximal match To find all maximal matches, it is in time proportional to the length of the query
Maximal match suffix links is used to find extended match
Cluster matches MUMmer 1: –Align 2 complete sequences, no rearrangement. –That is, to find a single longest alignment. MUMmer 2: –After matches are identified, the interval length between matches are checked. –If the interval length between matches is less than a user- defined gap length, the matches are joined into a cluster.
Alignment of incomplete genomes DNA sequencing Shotgun sequencing Terms Finishing NUCmer (NUCleotide MUMmer)
DNA sequencing –Human genomes are approx. 3 billion bases. –Sequencing machine can generate sequences for fragments in 500-600 bp long. –In order to read DNA, genome is broken up into tiny of pieces (reads), each is read individually. –After all pieces are read, they are assembled in the correct order.
Shotgun sequencing Extract DNA Fragment DNA Clone DNA Sequence both ends of clones –500-600 bp each read Assemble –reads are assembled to reconstruct the genome Finish sequencing (close gaps)
Terms Raw sequence –Unassembled sequence reads Contig –Overlapping reads are joined into longer composite sequences, called contigs Finished sequence –Complete sequence of a genome with no gaps and an accuracy of >99.9% Full shotgun coverage –Genome coverage in random raw sequence required to produce finished sequence, 8-10 fold (8-10X)
Finishing The process of –Determining the order and orientation of all the contigs, and –Generating additional sequence to fill in all the gaps between them (closing all the gaps) Finishing phase is dispensed in many projects. However, MUMmer 2.1 is used as a base to build a program for the “finishing phase” –Align the multiple contigs to a completed reference genome, and –Align one set of contigs to another set of contigs
NUCmer (NUC leotide MUM mer ) Built based on MUMmer 2.1 to develop a miltiple-contig alignment program 3 steps Outputs
NUCmer – 3 steps Step 1 –Input: 2 DNA sequences in 2 multi-fasta files representing partial or complete assemblies. –Each DNA sequence is represented as contig sequence –Creates a map of all contig positions within each of the multi-fasta files –Concatenates the two files seperately –Runs MUMmer to find all exact matches between two genomes –These matches are mapped back to the separate contigs
NUCmer – 3 steps (cont.) Step 2 ： Clustering –MUMs (output of step 1) are clustered together if they are separated in user-defined distance Step 3 –Run a modified Smith-Waterman DP alignment to align the sequence between MUMs (output of step 2) Result –Alignment of “every sequence contig in the 1st file” to “every sequence contig in the 2nd file” –Order, orientation, and coverage identity percentage of contigs