1. Significance in protein analysis
Swapan ‘Shop’ Mallick
Bioinformatics Group
Institute of Biotechnology
University of Helsinki

2. Overview The need for statistics Example: BLOSUM Example: BLAST
What do the scores mean?
How can you compare two scores?
Example: BLAST
Problems with BLAST
Review of Distributions
Distribution of random BLAST results
P-values and e-values
Statistics of BLAST
Summary and Conclusion
Exercise
Statistics underly many tools which we use for protein analysis, including:
alignment tools,
eg: BLAST
protein classification eg, PFAM (HMM),
Statistics is very important for bioinformatics. It is very easy to have a computer analyze the data and give you back a result. The problem is to decide whether the answer the computer gives you is any good at all.
How statistically significant is the answer?
What is the probability that this answer could have been obtained by random chance?
Is a certain pattern of amino acids or nucleotides important information that tells you something about a sequence, or is it nothing more than a fluctuation in the random background noise?
These are the underlying questions you need to ask whenever you do a database search or other type of bioinformatics analysis.

3. The need for statistics
Statistics is very important for bioinformatics.
It is very easy to have a computer analyze the data and give you back a result.
Problem is to decide whether the answer the computer gives you is any good at all.
Questions:
How statistically significant is the answer?
What is the probability that this answer could have been obtained by random? What does this depend on?
Statistics underly many tools which we use for protein analysis, including:
alignment tools,
eg: BLAST
protein classification eg, PFAM (HMM),
Other questions:
Is a certain pattern of amino acids or nucleotides important information that tells you something about a sequence, or is it nothing more than a fluctuation in the random background noise?
These are the underlying questions you need to ask whenever you do a database search or other type of bioinformatics analysis.

4. Basics N n
Sample
Population

5. Basics N
Descriptive statistics n
Sample
Population
Probability

6. Example: BLOSUM
The BLOSUM matrix assigns a probability score for each residue pair in an alignment based on:
the frequency with which that pairing is known to occur within conserved blocks of related proteins.
Simple since size of population = size of sample
BLOSUM matrices are constructed from observations which lead to observed probabilities

7. BLOSUM substitution matrices
BLOSUM matrices are used in ‘log-odds’ form based on actually observed substitutions.
This is because:
Ease of use: ‘Scores’ can be just added (the raw probabilities would have to be multiplied)
Ease of interpretation:
S=0 : substitution is just as likely to occur as random
S<0 : substitution is more likely to occur randomly than observed
S>0 : substitution is less likely to occur randomly than observed
Observed v. expected.
But I’m going to describe what the numbers mean, and how they relate to one another.
For example, we intuitively know that 6 is better than 5. But how much better?
shows traceback

8. Substitution matrices
Score of amino acid a with amino acid b
Pab is the observed frequency that residues a and b are correlated because of homology
Lambda is a scaling factor equal to 0.347, set so that the scores can be rounded off to sensible integers
shows traceback
fafb is the expected frequency of seeing residues a and b paired together, which is just the product of the frequency of residue a multiplied by the frequency of residue b
Source: Where did the BLOSUM62 alignment score matrix come from? Eddy S., Nat. Biotech. 22 Aug 2004

9. Substitution matrices
Lambda is a scaling factor equal to 0.347, set so that the scores can be rounded off to sensible integers
Pab is the observed frequency that residues a and b are correlated because of homology
shows traceback
fafb is the expected frequency of seeing residues a and b paired together, which is just the product of the frequency of residue a multiplied by the frequency of residue b

11. ii) Compare S=5 and S=10. Ratio is based on exponential function
i) S=0 : O/E ratio=1
ii) Compare S=5 and S=10. Ratio is based on exponential function
iii) S=-10: O/E ratio = ≈ 1/32.
iv) Ratio of scores S1, S2 in terms of probabilities of observed/random = i)
ii)
iii) ie the correspondence of two aas in an alignment that accurately represents
homology (evolutionary descent) is one tenth as frequent
as the chance alignment of these amino acids.
iv) eg: The ratio of probabilities for scores : 10, 5 would be about 5.6

12. ii) Compare S=5 and S=10. Ratio is based on exponential function
i) S=0 : O/E ratio=1
ii) Compare S=5 and S=10. Ratio is based on exponential function
iii) S=-10: O/E ratio = ≈ 1/32.
iv) Ratio of scores S1, S2 in terms of probabilities of observed/random =
32.1 i)
ii)
iii) ie the correspondence of two aas in an alignment that accurately represents
homology (evolutionary descent) is one tenth as frequent
as the chance alignment of these amino acids.
iv) eg: The ratio of probabilities for scores : 10, 5 would be about 5.6
5.7

13. ii) Compare S=5 and S=10. Ratio is based on exponential function
i) S=0 : O/E ratio=1
ii) Compare S=5 and S=10. Ratio is based on exponential function
iii) S=-10: O/E ratio = ≈ 1/32.
iv) Ratio of scores S1, S2 in terms of probabilities of observed/random =
32.1 i)
ii)
iii) ie the correspondence of two aas in an alignment that accurately represents
homology (evolutionary descent) is one tenth as frequent
as the chance alignment of these amino acids.
iv) eg: The ratio of probabilities for scores : 10, 5 would be about 5.6
5.7

14. ii) Compare S=5 and S=10. Ratio is based on exponential function
i) S=0 : O/E ratio=1
ii) Compare S=5 and S=10. Ratio is based on exponential function
iii) S=-10: O/E ratio = ≈ 1/32.
iv) Ratio of scores S1, S2 in terms of probabilities of observed/random =
32.1 i)
ii)
iii) ie the correspondence of two aas in an alignment that accurately represents
homology (evolutionary descent) is one tenth as frequent
as the chance alignment of these amino acids.
iv) eg: The ratio of probabilities for scores : 10, 5 would be about 5.6
5.7

15. Example: BLAST Motivations
Exact algorithms are exhaustive but computationally expensive.
Exact algorithms are impractical for comparing a query sequence to millions of other sequences in a database (database scanning),
and so, database scanning requires heuristic alignment algorithm (at the cost of optimality).

16. Interpret BLAST results - Description
ID (GI #, refseq #, DB-specific ID #) Click to access the record in GenBank
Gene/sequence Definition
Expect value – lower, better. It tells the possibility that this is a random hit
Bit score – higher, better. Click to access the pairwise alignment
Links

17. Problems with BLAST Why do results change?
How can you compare results from different BLAST tools which may report different types of values?
How are results (eg evalue) affected by query
There are _many_ values reported in the output – what do they mean?

18. Example: Importance of Blast statistics
But, first a review.

19. Review What is a distribution?
A plot showing the frequency of a given variable or observation.

20. Review What is a distribution?
A plot showing the frequency of a given variable or observation.

21. Features of a Normal Distribution
Symmetric Distribution
Has an average or mean value at the centre
Has a characteristic width called the standard deviation (S.D. = σ)
Most common type of distribution known
m = mean

22. Standard Deviations (Z-score)
Z value is the number of standard deviations you are away….
Disadvantages of Z-scores:
Absolute value is lost
Same score in different sample => different z-score

23. Mean, Median & Mode
Mode
Median
Mean

24. Mean, Median, Mode
In a Normal Distribution the mean, mode and median are all equal
In skewed distributions they are unequal
Mean - average value, affected by extreme values in the distribution
Median - the “middlemost” value, usually half way between the mode and the mean
Mode - most common value

25. Different Distributions
Unimodal Bimodal

26. Other Distributions Binomial Distribution Poisson Distribution
Extreme Value Distribution

27. Binomial Distribution1
1 1
P(x) = (p + q)n

28. Poisson Distribution P(x) x
Proportion of samples m
= 10
=0.1
= 1
= 2
= 3
P(x)
Poisson distribution is like a normal distribution at high values of mu x

29. Review What is a distribution? What is a null hypothesis?
A plot showing the frequency of a given variable or observation.
What is a null hypothesis?
A statistician’s way of characterizing “chance.”
Generally, a mathematical model of randomness with respect to a particular set of observations.
The purpose of most statistical tests is to determine whether the observed data can be explained by the null hypothesis.

30. Review What is a distribution? What is a null hypothesis?
A plot showing the frequency of a given variable or observation.
What is a null hypothesis?
A statistician’s way of characterizing “chance.”
Generally, a mathematical model of randomness with respect to a particular set of observations.
The purpose of most statistical tests is to determine whether the observed data can be explained by the null hypothesis.

31. Review Examples of null hypotheses:
Sequence comparison using shuffled sequences.
A normal distribution of log ratios from a microarray experiment.
LOD scores from genetic linkage analysis when the relevant loci are randomly sprinkled throughout the genome.

32. Empirical score distribution
The picture shows a distribution of scores from a real database search using BLAST.
This distribution contains scores from non-homologous and homologous pairs.
High scores from homology.

33. Empirical null score distribution
This distribution is similar to the previous one, but generated using a randomized sequence database.

34. Review
What is a p-value?

35. Review What is a p-value?
The probability of observing an effect as strong or stronger than you observed, given the null hypothesis. I.e., “How likely is this effect to occur by chance?”
Pr(x > S|null)

36. Review
What is the name of the distribution created by sequence similarity scores, and what does it look like?
Extreme value distribution, or Gumbel distribution.
It looks similar to a normal distribution, but it has a larger tail on the right.
Arises from sampling the extreme end of a normal distribution
A distribution which is “skewed” due to its selective sampling
Skew can be either right or left
In the limit of sufficiently large sequence lengths m and n, the statistics of HSP scores are characterized by two parameters, K and lambda. Most simply, the expected number of HSPs with score at least S is given by the formula
Equation gives Evalue.
We call this the E-value for the score S.
This formula makes eminently intuitive sense. Doubling the length of either sequence should double the number of HSPs attaining a given score. Also, for an HSP to attain the score 2x it must attain the score x twice in a row, so one expects E to decrease exponentially with score. The parameters K and lambda can be thought of simply as natural scales for the search space size and the scoring system respectively.

37. Review
What is the name of the distribution created by sequence similarity scores, and what does it look like?
Extreme value distribution, or Gumbel distribution.
It looks similar to a normal distribution, but it has a larger tail on the right.
Arises from sampling the extreme end of a normal distribution
A distribution which is “skewed” due to its selective sampling
Skew can be either right or left
In the limit of sufficiently large sequence lengths m and n, the statistics of HSP scores are characterized by two parameters, K and lambda. Most simply, the expected number of HSPs with score at least S is given by the formula
Equation gives Evalue.
We call this the E-value for the score S.
This formula makes eminently intuitive sense. Doubling the length of either sequence should double the number of HSPs attaining a given score. Also, for an HSP to attain the score 2x it must attain the score x twice in a row, so one expects E to decrease exponentially with score. The parameters K and lambda can be thought of simply as natural scales for the search space size and the scoring system respectively.

38. Statistics
BLAST (and also local i.e. Smith-Waterman and BLAT scores) between random, unrelated sequences follow the Gumbel Extreme Value Distribution (EVD)
Pr(s>S) = 1-exp(-Kmn e-lS)
This is the probability of randomly encountering a score greater than S.
S alignment score
m,n query sequence lengths, and length of database resp.
K, l parameters depending on scoring scheme and sequence composition
Bit score : S’ = lS – log(K) log(2)
We're interested in high scores S. Note that the bigger S gets, the smaller e-vS gets, and the smaller that gets, the closer exp(-Kmne-vS) gets to 1, and the closer the lower bound for P(s>S) gets to zero. That is, big S yield small P.
Notice that the function 1 - exp(-Kmne-vS) is not the distribution itself, but the area under its right-tail. Recall that areas are associated with probabilities.
In addition, if Kmne-vS is close to zero (ie as S gets bigger), then exp(-Kmne-vS) is well approximated by 1 - Kmne-vS. In that case the lower bound above can be well approximated by Kmne-vS. This value is called the expect.
According to Setabul and Meidanis in their book "Introduction to Computational Molecular Biology", it is interpreted as the expected number of distinct segment pairs between two random sequences with score above S
Nice website:

39. BLAST output revisited
S’ S E n m
 K
From: Expasy BLAST

40. Review EVD for random blast
Upper tail behaviour: Pr( s > S ) ~ Kmn e-lS
This is the EXPECT value = Evalue
This is the EXPECT value that you see on the NCBI web site.

41. How to Calculate E-values
Think of the databank as one very long random sequence, length G
Alignments with s>S occur randomly across the genome, with a Poisson distribution
Pr (highest-scoring alignment s>S) ~ KmGe-lS
Pr( no alignment s>S ) ~ 1 - KmGe-lS
Expected number m of alignments with s>S given by
1-e-m ~ 1 - KmGe-lS (Poisson property)
m ~ -log(KmG) + lS
Threshold S ~ [log(KmG) + m ]/l

42. Summary
Want to be able to compare scores in sequences of different compositions or different scoring schemes
Score: S = sum(match) – sum(gap costs)
Notice that the bit score is a function of the database. Though this is better, this means that as the size of the database grows, the bit score for the same alignment can drop! This also means that the E-value will change.

43. Summary
Want to be able to compare scores in sequences of different compositions or different scoring schemes
Score: S = sum(match) – sum(gap costs)
Bit score
S’ = lS – log(K) log(2)
Notice that the bit score is a function of the database. Though this is better, this means that as the size of the database grows, the bit score for the same alignment can drop! This also means that the E-value will change.

44. Score and bit score grow linearly with the length of the alignment
Summary
Want to be able to compare scores in sequences of different compositions or different scoring schemes
Score: S = sum(match) – sum(gap costs)
Bit score
S’ = lS – log(K) log(2)
Notice that the bit score is a function of the database. Though this is better, this means that as the size of the database grows, the bit score for the same alignment can drop! This also means that the E-value will change.

45. Score and bit score grow linearly with the length of the alignment
Summary
Want to be able to compare scores in sequences of different compositions or different scoring schemes
Score: S = sum(match) – sum(gap costs)
Bit score
S’ = lS – log(K) log(2)
E-value of bit score
E = mn2-S’

46. Score and bit score grow linearly with the length of the alignment
Summary
E-Value shrinks really fast as bit score grows
Want to be able to compare scores in sequences of different compositions or different scoring schemes
Score: S = sum(match) – sum(gap costs)
Bit score
S’ = lS – log(K) log(2)
E-value of bit score
E = mn2-S’

47. Score and bit score grow linearly with the length of the alignment
Summary
E-Value shrinks really fast as bit score grows
Want to be able to compare scores in sequences of different compositions or different scoring schemes
Score: S = sum(match) – sum(gap costs)
Bit score
S’ = lS – log(K) log(2)
E-value of bit score
E = mn2-S’
E-Value grows linearly with the product of target and query sizes.

48. Score and bit score grow linearly with the length of the alignment
Summary
E-Value shrinks really fast as bit score grows
Want to be able to compare scores in sequences of different compositions or different scoring schemes
Score: S = sum(match) – sum(gap costs)
Bit score
S’ = lS – log(K) log(2)
E-value of bit score
E = mn2-S’
E-Value grows linearly with the product of target and query sizes.
Doubling target set size and doubling query length have the same effect on e-value

49. Conclusion
You should now be able to compare BLAST results from different databases, converting values if they are reported differently (which happens frequently)
You should now know why BLAST results might change from one day to the next, even on the same server
You should understand also the dependance of query length on E-value.
Statistical rankings are reported for (almost) every database search tool. When making comparisons between databases, between sequences it is useful to know how the statistics are derived to know if comparisons are meaningful.

50. THE END

51. Supplemental
Section

52. What is the structure of my sequence?
Look through: Patterns in sequences (Searching for information within sequences) - Some common problems and their solutions:
What is the structure of my sequence?
(clickable!)
Statistics underly many tools which we use for protein analysis, including:
alignment tools,
eg: BLAST
protein classification eg, PFAM (HMM),