Fa05CSE 182 CSE182-L4: Keyword matching. Fa05CSE 182 Backward scoring Defin S b [i,j] : Best scoring alignment of the suffixes s[i+1..n] and t[j+1..m]

Slides:



Advertisements
Similar presentations
Fa07CSE 182 CSE182-L4: Database filtering. Fa07CSE 182 Summary (through lecture 3) A2 is online We considered the basics of sequence alignment –Opt score.
Advertisements

Hidden Markov Models (1)  Brief review of discrete time finite Markov Chain  Hidden Markov Model  Examples of HMM in Bioinformatics  Estimations Basic.
Gapped BLAST and PSI-BLAST Altschul et al Presenter: 張耿豪 莊凱翔.
BLAST Sequence alignment, E-value & Extreme value distribution.
OUTLINE Scoring Matrices Probability of matching runs Quality of a database match.
Combinatorial Pattern Matching CS 466 Saurabh Sinha.
Lecture 8 Alignment of pairs of sequence Local and global alignment
Introduction to Bioinformatics
Searching Sequence Databases
. Class 4: Fast Sequence Alignment. Alignment in Real Life u One of the major uses of alignments is to find sequences in a “database” u Such collections.
Lecture outline Database searches
Sequence Alignment Storing, retrieving and comparing DNA sequences in Databases. Comparing two or more sequences for similarities. Searching databases.
Heuristic alignment algorithms and cost matrices
March 2006Vineet Bafna Database Filtering. March 2006Vineet Bafna Project/Exam deadlines May 2 – Send to me with a title of your project May 9 –
Fa05CSE 182 L3: Blast: Keyword match basics. Fa05CSE 182 Silly Quiz TRUE or FALSE: In New York City at any moment, there are 2 people (not bald) with.
Fa05CSE 182 CSE182-L5: Position specific scoring matrices Regular Expression Matching Protein Domains.
Sequence Alignment vs. Database Task: Given a query sequence and millions of database records, find the optimal alignment between the query and a record.
1 1. BLAST (Basic Local Alignment Search Tool) Heuristic Only parts of protein are frequently subject to mutations. For example, active sites (that one.
Fa05CSE 182 L3: Blast: Local Alignment and other flavors.
Sequence Alignment Variations Computing alignments using only O(m) space rather than O(mn) space. Computing alignments with bounded difference Exclusion.
Fa05CSE 182 CSE182-L4: Scoring matrices, Dictionary Matching.
Alignment methods June 26, 2007 Learning objectives- Understand how Global alignment program works. Understand how Local alignment program works.
Similar Sequence Similar Function Charles Yan Spring 2006.
Sequence Alignment III CIS 667 February 10, 2004.
Practical algorithms in Sequence Alignment Sushmita Roy BMI/CS 576 Sep 16 th, 2014.
Fa05CSE 182 CSE182-L5: Scoring matrices Dictionary Matching.
Alignment IV BLOSUM Matrices. 2 BLOSUM matrices Blocks Substitution Matrix. Scores for each position are obtained frequencies of substitutions in blocks.
Sequence similarity. Motivation Same gene, or similar gene Suffix of A similar to prefix of B? Suffix of A similar to prefix of B..Z? Longest similar.
Protein Sequence Comparison Patrice Koehl
Sequence alignment, E-value & Extreme value distribution
CSE182-L5: Scoring matrices Dictionary Matching
Alignment methods II April 24, 2007 Learning objectives- 1) Understand how Global alignment program works using the longest common subsequence method.
Sequence comparison: Significance of similarity scores Genome 559: Introduction to Statistical and Computational Genomics Prof. James H. Thomas.
Heuristic methods for sequence alignment in practice Sushmita Roy BMI/CS 576 Sushmita Roy Sep 27 th,
Alignment Statistics and Substitution Matrices BMI/CS 576 Colin Dewey Fall 2010.
CISC667, S07, Lec5, Liao CISC 667 Intro to Bioinformatics (Spring 2007) Pairwise sequence alignment Needleman-Wunsch (global alignment)
Evolution and Scoring Rules Example Score = 5 x (# matches) + (-4) x (# mismatches) + + (-7) x (total length of all gaps) Example Score = 5 x (# matches)
Gapped BLAST and PSI- BLAST: a new generation of protein database search programs By Stephen F. Altschul, Thomas L. Madden, Alejandro A. Schäffer, Jinghui.
Classifier Evaluation Vasileios Hatzivassiloglou University of Texas at Dallas.
Pairwise Sequence Alignment. The most important class of bioinformatics tools – pairwise alignment of DNA and protein seqs. alignment 1alignment 2 Seq.
Pairwise alignment of DNA/protein sequences I519 Introduction to Bioinformatics, Fall 2012.
1 Lecture outline Database searches –BLAST –FASTA Statistical Significance of Sequence Comparison Results –Probability of matching runs –Karin-Altschul.
BLAST Anders Gorm Pedersen & Rasmus Wernersson. Database searching Using pairwise alignments to search databases for similar sequences Database Query.
Comp. Genomics Recitation 3 The statistics of database searching.
Sequence Alignment Csc 487/687 Computing for bioinformatics.
BLAST: Basic Local Alignment Search Tool Altschul et al. J. Mol Bio CS 466 Saurabh Sinha.
Sequence Comparison Algorithms Ellen Walker Bioinformatics Hiram College.
BLAST Slides adapted & edited from a set by Cheryl A. Kerfeld (UC Berkeley/JGI) & Kathleen M. Scott (U South Florida) Kerfeld CA, Scott KM (2011) Using.
Pairwise Local Alignment and Database Search Csc 487/687 Computing for Bioinformatics.
Pairwise Sequence Alignment Part 2. Outline Summary Local and Global alignments FASTA and BLAST algorithms Evaluating significance of alignments Alignment.
©CMBI 2005 Database Searching BLAST Database Searching Sequence Alignment Scoring Matrices Significance of an alignment BLAST, algorithm BLAST, parameters.
Sequence Alignment.
Construction of Substitution matrices
Doug Raiford Phage class: introduction to sequence databases.
Step 3: Tools Database Searching
The statistics of pairwise alignment BMI/CS 576 Colin Dewey Fall 2015.
©CMBI 2005 Database Searching BLAST Database Searching Sequence Alignment Scoring Matrices Significance of an alignment BLAST, algorithm BLAST, parameters.
Dynamic programming with more complex models When gaps do occur, they are often longer than one residue.(biology) We can still use all the dynamic programming.
BLAST: Database Search Heuristic Algorithm Some slides courtesy of Dr. Pevsner and Dr. Dirk Husmeier.
Using BLAST To Teach ‘E-value-tionary’ Concepts Cheryl A. Kerfeld 1, 2 and Kathleen M. Scott 3 1.Department of Energy-Joint Genome Institute, Walnut Creek,
Your friend has a hobby of generating random bit strings, and finding patterns in them. One day she come to you, excited and says: I found the strangest.
Substitution Matrices and Alignment Statistics BMI/CS 776 Mark Craven February 2002.
Vineet Bafna. How can we compute the local alignment itself?
9/6/07BCB 444/544 F07 ISU Dobbs - Lab 3 - BLAST1 BCB 444/544 Lab 3 BLAST Scoring Matrices & Alignment Statistics Sept6.
Database Scanning/Searching FASTA/BLAST/PSIBLAST G P S Raghava.
Pairwise Sequence Alignment (cont.)
BLAST Slides adapted & edited from a set by
Sequence alignment, E-value & Extreme value distribution
BLAST Slides adapted & edited from a set by
Searching Sequence Databases
Presentation transcript:

Fa05CSE 182 CSE182-L4: Keyword matching

Fa05CSE 182 Backward scoring Defin S b [i,j] : Best scoring alignment of the suffixes s[i+1..n] and t[j+1..m] Q: What is the score of the best alignment of the two strings s and t? HW: Write the recurrences for S b

Fa05CSE 182 Forward/Backward computations F[j] = Score of the best scoring alignment of s[1..n/2] and t[1..j] –F[j] = S[n/2,j] B[j] = Score of the best scoring alignment of s[n/2+1..n] and t[j+1..m] –B[j] = S b [n/2,j] n/2 j 1 m

Fa05CSE 182 Forward/Backward computations At the optimal coordinate, j –F[j]+B[j]=S[n,m] In O(nm) time, and O(m) space, we can compute one of the coordinates on the optimum path. n/2 j 1 m

Fa05CSE 182 Forward, Backward computation There exists a coordinate, j –F[j]+B[j]=S[n,m] In O(nm) time, and O(m) space, we can compute one of the coordinates on the optimum path.

Fa05CSE 182 Linear Space Alignment Align(1..n,1..m) –For all 1<=j <= m Compute F[j]=S(n/2,j) –For all 1<=j <= m Compute B[j]=S b (n/2,j) –j* = max j {F[j]+B[j] } –X = Align(1..n/2,1..j*) –Y = Align(n/2..n,j*..m) –Return X,j*,Y

Fa05CSE 182 Linear Space complexity T(nm) = c.nm + T(nm/2) = O(nm) Space = O(m)

Fa05CSE 182 Summary We considered the basics of sequence alignment –Opt score computation –Reconstructing alignments –Local alignments –Affine gap costs –Space saving measures Can we recreate Blast?

Fa05CSE 182 Blast and local alignment Concatenate all of the database sequences to form one giant sequence. Do local alignment computation with the query.

Fa05CSE 182 Large database search Query (m) Database (n) Database size n=100M, Querysize m=1000. O(nm) = computations

Fa05CSE 182 Why is Blast Fast?

Fa05CSE 182 Silly Question! True or False: No two people in new york city have the same number of hair

Fa05CSE 182 Observations Much of the database is random from the query’s perspective Consider a random DNA string of length n. –Pr[A]=Pr[C] = Pr[G]=Pr[T]=0.25 Assume for the moment that the query is all A’s (length m). What is the probability that an exact match to the query can be found?

Fa05CSE 182 Basic probability Probability that there is a match starting at a fixed position i = 0.25 m What is the probability that some position i has a match. Dependencies confound probability estimates.

Fa05CSE 182 Basic Probability:Expectation Q: Toss a coin: each time it comes up heads, you get a dollar –What is the money you expect to get after n tosses? –Let X i be the amount earned in the i-th toss  Total money you expect to earn

Fa05CSE 182 Expected number of matches Expected number of matches can still be computed.  Let X i =1 if there is a match starting at position i, X i =0 otherwise  Expected number of matches = i

Fa05CSE 182 Expected number of exact Matches is small! Expected number of matches = n*0.25 m –If n=10 7, m=10, Then, expected number of matches = –If n=10 7, m=11 expected number of hits = 2.38 –n=10 7,m=12, Expected number of hits = 0.5 < 1 Bottom Line: An exact match to a substring of the query is unlikely just by chance.

Fa05CSE 182 Observation 2 What is the pigeonhole principle?  Suppose we are looking for a database string with greater than 90% identity to the query (length 100)  Partition the query into size 10 substrings. At least one much match the database string exactly

Fa05CSE 182 Why is this important? Suppose we are looking for sequences that are 80% identical to the query sequence of length 100. Assume that the mismatches are randomly distributed. What is the probability that there is no stretch of 10 bp, where the query and the subject match exactly? Rough calculations show that it is very low. Exact match of a short query substring to a truly similar subject is very high. –The above equation does not take dependencies into account –Reality is better because the matches are not randomly distributed

Fa05CSE 182 Just the Facts Consider the set of all substrings of the query string of fixed length W. –Prob. of exact match to a random database string is very low. –Prob. of exact match to a true homolog is very high. –Keyword Search (exact matches) is MUCH faster than sequence alignment

Fa05CSE 182 BLAST Consider all (m-W) query words of size W (Default = 11) Scan the database for exact match to all such words For all regions that hit, extend using a dynamic programming alignment. Can be many orders of magnitude faster than SW over the entire string Database (n)

Fa05CSE 182 Why is BLAST fast? Assume that keyword searching does not consume any time and that alignment computation the expensive step. Query m=1000, random Db n=10 7, no TP SW = O(nm) = 1000*10 7 = computations BLAST, W=11 E(#11-mer hits)= 1000* (1/4) 11 * 10 7 =2384 Number of computations = 2384*100*100=2.384*10 7 Ratio=10 10 /(2.384*10 7 )=420 Further speed improvements are possible

Fa05CSE 182 Keyword Matching How fast can we match keywords? Hash table/Db index? What is the size of the hash table, for m=11 Suffix trees? What is the size of the suffix trees? Trie based search. We will do this in class. AATCA 567

Fa05CSE 182 Related notes How to choose the alignment region? –Extend greedily until the score falls below a certain threshold What about protein sequences? –Default word size = 3, and mismatches are allowed. Like sequences, BLAST has been evolving continuously –Banded alignment –Seed selection –Scanning for exact matches, keyword search versus database indexing

Fa05CSE 182 P-value computation How significant is a score? What happens to significance when you change the score function A simple empirical method: Compute a distribution of scores against a random database. Use an estimate of the area under the curve to get the probability. OR, fit the distribution to one of the standard distributions.

Fa05CSE 182 Z-scores for alignment Initial assumption was that the scores followed a normal distribution. Z-score computation: –For any alignment, score S, shuffle one of the sequences many times, and recompute alignment. Get mean and standard deviation –Look up a table to get a P-value

Fa05CSE 182 Blast E-value Initial (and natural) assumption was that scores followed a Normal distribution 1990, Karlin and Altschul showed that ungapped local alignment scores follow an exponential distribution Practical consequence: –Longer tail. –Previously significant hits now not so significant

Fa05CSE 182 Exponential distribution Random Database, Pr(1) = p What is the expected number of hits to a sequence of k 1’s Instead, consider a random binary Matrix. Expected # of diagonals of k 1s

Fa05CSE 182 As you increase k, the number decreases exponentially. The number of diagonals of k runs can be approximated by a Poisson process In ungapped alignments, we replace the coin tosses by column scores, but the behaviour does not change (Karlin & Altschul). As the score increases, the number of alignments that achieve the score decreases exponentially

Fa05CSE 182 Blast E-value Choose a score such that the expected score between a pair of residues < 0 Expected number of alignments with a particular score For small values, E-value and P-value are the same

Fa05CSE 182 Blast Variants 1.What is mega-blast? 2.What is discontiguous mega- blast? 3.Phi-Blast/Psi-Blast? 4.BLAT? 5.PatternHunter? Longer seeds. Seeds with don’t care values Later Database pre-processing Seeds with don’t care values

Fa05CSE 182 Silly Quiz Name a famous Bioinformatics Researcher Name a famous Bioinformatics Researcher who is a woman

Fa05CSE 182 Scoring DNA DNA has structure.

Fa05CSE 182 DNA scoring matrices So far, we considered a simple match/mismatch criterion. The nucleotides can be grouped into Purines (A,G) and Pyrimidines. Nucleotide substitutions within a group (transitions) are more likely than those across a group (transversions)

Fa05CSE 182 Scoring proteins Scoring protein sequence alignments is a much more complex task than scoring DNA –Not all substitutions are equal Problem was first worked on by Pauling and collaborators In the 1970s, Margaret Dayhoff created the first similarity matrices. –“One size does not fit all” –Homologous proteins which are evolutionarily close should be scored differently than proteins that are evolutionarily distant –Different proteins might evolve at different rates and we need to normalize for that

Fa05CSE 182 PAM 1 distance Two sequences are 1 PAM apart if they differ in 1 % of the residues. PAM 1 (a,b) = Pr[residue b substitutes residue a, when the sequences are 1 PAM apart] 1% mismatch

Fa05CSE 182 PAM1 matrix Align many proteins that are very similar –Is this a problem? PAM1 distance is the probability of a substitution when 1% of the residues have changed Estimate the frequency P b|a of residue a being substituted by residue b. S(a,b) = log 10 (P ab /P a P b ) = log 10 (P b|a /P b )

Fa05CSE 182 PAM 1

Fa05CSE 182 PAM distance Two sequences are 1 PAM apart when they differ in 1% of the residues. When are 2 sequences 2 PAMs apart? 1 PAM 2 PAM

Fa05CSE 182 Higher PAMs PAM 2 (a,b) = ∑ c PAM 1 (a,c). PAM 1 (c,b) PAM 2 = PAM 1 * PAM 1 (Matrix multiplication) PAM 250 –= PAM 1 *PAM 249 –= PAM 1 250

Fa05CSE 182 Note: This is not the score matrix: What happens as you keep increasing the power?

Fa05CSE 182 Scoring using PAM matrices Suppose we know that two sequences are 250 PAMs apart. S(a,b) = log 10 (P ab /P a P b )= log 10 (P b|a /P b ) = log 10 (PAM 250 (a,b)/P b )

Fa05CSE 182 BLOSUM series of Matrices Henikoff & Henikoff: Sequence substitutions in evolutionarily distant proteins do not seem to follow the PAM distributions A more direct method based on hand-curated multiple alignments of distantly related proteins from the BLOCKS database. BLOSUM60 Merge all proteins that have greater than 60%. Then, compute the substitution probability. –In practice BLOSUM62 seems to work very well.

Fa05CSE 182 PAM vs. BLOSUM What is the correspondence? PAM1 Blosum1 PAM2 Blosum2 Blosum62 PAM250 Blosum100

Fa05CSE 182 Dictionary Matching, R.E. matching, and position specific scoring

Fa05CSE 182 Keyword search Recall: In BLAST, we get a collection of keywords from the query sequence, and identify all db locations with an exact match to the keyword. Question: Given a collection of strings (keywords), find all occrrences in a database string where they keyword might match.

Fa05CSE 182 Dictionary Matching Q: Given k words (s i has length l i ), and a database of size n, find all matches to these words in the database string. How fast can this be done? 1:POTATO 2:POTASSIUM 3:TASTE P O T A S T P O T A T O dictionary database

Fa05CSE 182 Dict. Matching & string matching How fast can you do it, if you only had one word of length m? –Trivial algorithm O(nm) time –Pre-processing O(m), Search O(n) time. Dictionary matching –Trivial algorithm (l 1 +l 2 +l 3 …)n –Using a keyword tree, l p n (l p is the length of the longest pattern) –Aho-Corasick: O(n) after preprocessing O(l 1 +l 2..) We will consider the most general case

Fa05CSE 182 Direct Algorithm P O P O P O T A S T P O T A T O P O T A T O Observations: When we mismatch, we (should) know something about where the next match will be. When there is a mismatch, we (should) know something about other patterns in the dictionary as well.

Fa05CSE 182 PO T A TO T UISM SET A The Trie Automaton Construct an automaton A from the dictionary –A[v,x] describes the transition from node v to a node w upon reading x. –A[u,’T’] = v, and A[u,’S’] = w –Special root node r –Some nodes are terminal, and labeled with the index of the dictionary word. 1:POTATO 2:POTASSIUM 3:TASTE w vu S r

Fa05CSE 182 An O(l p n) algorithm for keyword matching Start with the first position in the db, and the root node. If successful transition –Increment current pointer –Move to a new node –If terminal node “success” Else –Retract ‘current’ pointer –Increment ‘start’ pointer –Move to root & repeat

Fa05CSE 182 Illustration: PO T A TO T UISM SET A P O T A S T P O T A T O l c v S 1

Fa05CSE 182 Idea for improving the time P O T A S T P O T A T O Suppose we have partially matched pattern i (indicated by l, and c), but fail subsequently. If some other pattern j is to match –Then prefix(pattern j) = suffix [ first c-l characters of pattern(i)) l c 1:POTATO 2:POTASSIUM 3:TASTE P O T A S S I U M T A S T E Pattern i Pattern j

Fa05CSE 182 Improving speed of dictionary matching Every node v corresponds to a string s v that is a prefix of some pattern. Define F[v] to be the node u such that s u is the longest suffix of s v If we fail to match at v, we should jump to F[v], and commence matching from there Let lp[v] = |s u | PO T A TO T UISM SET A S

Fa05CSE 182 An O(n) alg. For keyword matching Start with the first position in the db, and the root node. If successful transition –Increment current pointer –Move to a new node –If terminal node “success” Else (if at root) –Increment ‘current’ pointer –Mv ‘start’ pointer –Move to root Else –Move ‘start’ pointer forward –Move to failure node

Fa05CSE 182 Illustration P O T A S T P O T A T O POTATO T UISM SET A lc v S 1

Fa05CSE 182 Time analysis In each step, either c is incremented, or l is incremented Neither pointer is ever decremented (lp[v] < c-l). l and c do not exceed n Total time <= 2n P O T A S T P O T A T O lc

Fa05CSE 182 Blast: Putting it all together Input: Query of length m, database of size n Select word-size, scoring matrix, gap penalties, E-value cutoff

Fa05CSE 182 Blast Steps 1.Generate an automaton of all query keywords. 2.Scan database using a “Dictionary Matching” algorithm (O(n) time). Identify all hits. 3.Extend each hit using a variant of “local alignment” algorithm. Use the scoring matrix and gap penalties. 4.For each alignment with score S, compute the bit-score, E- value, and the P-value. Sort according to increasing E-value until the cut-off is reached. 5.Output results.

Fa05CSE 182 Protein Sequence Analysis What can you do if BLAST does not return a hit? –Sometimes, homology (evolutionary similarity) exists at very low levels of sequence similarity. A: Accept hits at higher P-value. –This increases the probability that the sequence similarity is a chance event. –How can we get around this paradox? –Reformulated Q: suppose two sequences B,C have the same level of sequence similarity to sequence A. If A& B are related in function, can we assume that A& C are? If not, how can we distinguish?