1. Comparing Database Search Methods & Improving the Performance of PSI-BLAST Stephen Altschul

2. “Gold standards” for protein classification Traditional curated sequence databases with family and superfamily classifications: PIR SWISS-PROT Structure-based protein domain classification: SCOP

3. Measuring retrieval accuracy Sequence Search RelatedUnrelated Positive TP True Positive FP False Positive P = TP + FP Negative FN False Negative TN True Negative N = FN + TN R = TP + FN U = FP + TN Sensitivity: TP/RSpecificity: TP/P

4. Receiver Operating Characteristic curve False +True – False – True +

5. Random retrieval on a ROC plot

6. Line of fixed sensitivity

7. Line of fixed specificity

8. Line of fixed crossover ratio

10. ROC score: area under the ROC curve

11. Region of interest in ROC analysis

13. Truncated ROC, or ROC n curve 0 10 –3 Fraction unrelated accepted

14. ROC n score: area under the ROC n curve

15. Questions concerning ROC analysis What false-positive cutoff value should be used? When does it make sense to pool the results of database searches? When are the ROC scores for two different methods significantly different?

16. Marginal ratio of true to false positives

17. Definition of the ROC n score t: Total number of related sequences t i : Number of related sequences (true positives) returned before the ith false positive

18. “Random distribution” of ROC n scores Bootstrap resampling can be used to assign a statistical significance to differences in ROC n scores. Under reasonable assumptions, the distribution of bootstrapped ROC n scores is approximately normal. Resampling a small subset in a large database is equivalent to resampling the subset with independent Poisson distributions with mean 1.

19. Bootstrap resampling of false positives 12345678910 Retrieval Ranking of the Database 1348104 527 The false records are the noise. 41048 Only false records are resampled with replacement. 12357 The true records are well characterized.

20. Mean and variance for the normal distribution of ROC n scores yielded by resampling only the false positives

21. Mean and variance for the normal distribution of the difference of two ROC n scores, yielded by resampling only the false positives

22. PSI-BLAST in a nutshell With a protein sequence as query, use BLAST to search a protein sequence database. Collapse significant local alignments (those with E- value less than or equal to a set threshold h) into a multiple alignment, using the residues of the query sequence as alignment-column placeholders. Abstract a position-specific score matrix from the multiple alignment. Search the database with the score matrix as query. Iterate a fixed number of times, or until convergence.

23. Protocol for evaluating PSI-BLAST For each query sequence, search a comprehensive protein sequence database (e.g. NCBI’s nr) through a fixed number of PSI-BLAST iterations, or until convergence. Use the resulting position-specific score matrix to search the “gold standard” database. Pool the search results for ROC analysis.

24. The effect of acceptance threshold h on PSI-BLAST accuracy

25. Some ideas for improving PSI-BLAST 1. New statistical parameters 2. Smith-Waterman alignment 3. Substitution matrix frequency ratios 4. Apply SEG to database sequences 5. Composition-based statistics 6. “Concentrated” accounting of gaps 7. “Dispersed” accounting of gaps 8. Exponentiate Henikoff weights 9. Reverse sequence normalization 10. Window for amino acid composition 11. Use pseudocounts with composition window 12. Vary gap costs 13. Generalized affine gap costs 14. Substitution score offset 15. Information-dependent pseudocount parameter 16. Database-sequence length- normalization 17. Restricted score rescaling 18. Adjust purging percentage 19. Adjust pseudocount parameter 20. Adjust acceptance threshold

26. The effect of composition-based statistics on PSI-BLAST accuracy

27. Composition-based statistics Statistics based on “standard” amino acid frequencies can differ by orders of magnitude from those based upon the peculiar composition of two proteins. Standard protein: 4.5 % N DNA pol III, β chain [M. genitalium]: 12.1 % N DNA pol III, β chain [C. jejuni]: 7.6 % N Depending upon the composition assumed, a search of nr with M. genitalium DNA pol III as query yields different E-values for C. jejuni DNA pol III, as well as for the highest-scoring false positive: “Standard” statistics: 10 -10 0.0002 Composition-based statistics: 0.001 0.2 At a threshold of 0.0001, “standard” statistics yield 54 true positives, while at 0.1, composition-based statistics yield 55 true positives.

28. The effect of dispersed accounting of gaps on PSI-BLAST accuracy

29. The effect of restricted score rescaling and parameter tuning on PSI-BLAST accuracy

30. Accuracy of PSI-BLAST Program versionROC 100 score Original h = 10 -6 0.758 ± 0.005 + Composition-based statistics h = 0.002 0.879 ± 0.003 + “Dispersed” gap accounting h = 0.005 0.884 ± 0.002 + Restricted score rescaling b = 9 ; p = 0.94 0.895 ± 0.003

31. PSI-BLAST accuracy as a function of the number of iterations

32. Literature ROC analysis Swets, J.A. (1988) Science 240:1285-1293 Gribskov, M. & Robinson, N.L. (1996) Comput. Chem. 20:25-33 PSI-BLAST Altschul, S.F. et al. (1997) Nucl. Acids Res. 25:3389-3402 Composition-based statistics Karplus, K. et al. (1998) Bioinformatics 14:846-856 Schäffer, A.A. et al. (1999) Bioinformatics 15:1000-1011 Mott, R. (2000) J. Mol. Biol. 300:649-659 Statistics of ROC n resampling Schäffer, A.A. et al. (2001) Nucl. Acids Res. 29:2994-3005 Spouge, J.L. & Czabarka, E. (2002) ISMB Poster 133A

33. Acknowledgements Analysis of ROC n score distribution John Spouge Eva Czabarka Improvements to PSI- BLAST Alejandro Schäffer L. Aravind Thomas Madden Sergei Shavirin John Spouge Yuri Wolf Eugene Koonin