Presentation on theme: "AB RF Proteome Informatics Research Group iPRG 2012: A Study on Detecting Modified Peptides in a Complex Mixture ABRF 2012, Orlando, FL 3/17-20/2012."— Presentation transcript:
AB RF Proteome Informatics Research Group iPRG 2012: A Study on Detecting Modified Peptides in a Complex Mixture ABRF 2012, Orlando, FL 3/17-20/2012
AB RF Proteome Informatics Research Group IPRG2012 STUDY: DESIGN 2
AB RF Proteome Informatics Research Group Study Goals Primary:Evaluate the ability of participants to identify modified peptides present in a complex mixture Secondary:Find out why result sets might differ between participants Tertiary:Produce a benchmark dataset, along with an analysis resource 3
AB RF Proteome Informatics Research Group Study Design Use a common, rich dataset Use a common sequence database Allow participants to use the bioinformatic tools and methods of their choosing Use a common reporting template Report results at an estimated 1% FDR (at the spectrum level) Ignore protein inference 4
AB RF Proteome Informatics Research Group Sample Tryptic digest of yeast (RM8323 – NIST), spiked with 69 synthetic modified peptides (tryptic peptides from 6 different proteins – sPRG) – Phospho (STY) – Sulfo (Y) – Mono-, di-, trimethyl (K) – Mono-, dimethyl (R) – Acetyl (K) – Nitro (Y) 5
AB RF Proteome Informatics Research Group Supplied Study Materials 5600 TripleTOF dataset (i.e. WIFF file) – WIFF, mzML, dta, MGF (de-isotoped);– conversions by MS Data Converter 1.1.0 – MGF (not de-isotoped – conversion by Mascot Distiller 2.4) 1 fasta file (UniProtKB/SwissProt S. cerevisiae, human, + 1 bovine protein + trypsin from Dec. 2011) 1 template (Excel) 1 on-line survey (Survey Monkey) 6
AB RF Proteome Informatics Research Group Instructions to Participants 1.Retrieve and analyze the data file in the format of your choosing, with the method(s) of your choosing 2.Report the peptide to spectrum matches in the provided template 3.Report measures of reliability for PTM site assignments (optional) 4.Fill out the survey 5.Attach a 1-2 page description of the methodology employed 7
AB RF Proteome Informatics Research Group iPRG 2012 STUDY: PARTICIPATION 8
AB RF Proteome Informatics Research Group Study advertised on the ABRF website and listserv and by direct invitation from iPRG members 1. Email participation request to iPRGxxxx@gmail.com 2. Send official study letter with instructions 3. All further communication (e.g., questions, submission) through iPRGxxx.email@example.com iPRG members Participant Questions / Answers Anonymizer Soliciting Participants and Logistics 9
AB RF Proteome Informatics Research Group Participants (i) – overall numbers 24 submissions – One participant submitted two result sets 9 initialed iPRG member submissions (with appended i) 2 vendor submissions (identifiable by appended v) 10
AB RF Proteome Informatics Research Group About the Participant 11
AB RF Proteome Informatics Research Group About the Participants Lab 12
AB RF Proteome Informatics Research Group Participation in sPRG Study 13 Only one participant indicated he used sPRG information to aid his analysis. This person was one of the least successful in identifying the spiked-in peptides!
AB RF Proteome Informatics Research Group Search Engine Used 14
AB RF Proteome Informatics Research Group Site Localization Software 15 4 participants did not list using software for site localization.
AB RF Proteome Informatics Research Group Summary of Submitted Results 16 Only reported modified peptides
AB RF Proteome Informatics Research Group Summary of IDs and Localizations 17 Peptide Identification in all Spectra Site Localization in Spectra With Interesting Modifications
AB RF Proteome Informatics Research Group Overlap of spectrum identifications 18 7840 agreed on by 3 or more participants
AB RF Proteome Informatics Research Group AnAndromeda/MaxQuantMGMS-GFDBpFpFindScScaffold ASA-ScoreMMMyriMatchPkPEAKSSMSpectrum Mill ByByonicsMOMODaPkDBPEAKSDBSqSequest IHIn-house softwareOOMSSAPPiProtein PilotSTSpectraST IPIDPickerOtOtherPPrProtein ProspectorTPPTransProteomic Pipeline MMascotP/PPPep/Prot ProphetPRPhosphoRSXLExcel MDeMascot Delta ScorePDProteomeDiscovererPWProteoWizardXTX!Tandem MDiMascot Distiller Room for improvement in thresholding? 19 71755v58288v3356493128i112115840987133i94158i97053i42424i77777i2306840104i87048i9265334284i2311774564141515278147603141524551111821 PeaklistmgfmzML mgf mzMLmgf_ndmgfmgf_ndmgfmzML mgf_ndmgf mzMLmgfmgf_ndmgf mgf_ndmgfmgf_nd mzML mgf_nd WIFF Spectral Pre-ProcessingPk PPiMDiOtSMpFPkMDiPD PkDBPWPkDBSq Peptide IdentificationByPk MPPiMOPPrpFMMGMP/PPSMpFMPkMPPiMOMMPD PkDB OSTMMOTPPPkDBSq STXTIHXT Ot Discovery of Unexpected ModsByPk MPPiSTPPrpFIHMOSMpFMPkPPiOM PkDB Site LocalizationPk MDePPiMPPrpFMIHMSMpFMPkMASIHAnMDePD PkDBOMMOPkDBScOtSq STIH Results FilteringByPk IHPPiP/PP PPrpFIPIHXLIHXLpFMPkMXLScMPR PkDBXLOtTPP XLPkDB NTT21121?11121?12222?1222?2 Experience 5-10 years >10 years< 1 year3-4 years 5-10 years >10 years 5-10 years 3-4 years 5-10 years >10 years1-2 years >10 years1-2 years >10 years 1-2 years >10 years 1-2 years >10 years
AB RF Proteome Informatics Research Group ESR and FDR Extraordinary Skill Rate or High False Discovery Rate? ESR + FDR = 100* (Y<3P+YD)/total ids Y 20 24 participants 3 for consensus
AB RF Proteome Informatics Research Group Characteristics of consensus spectra 7840 spectra >=3 participants agreeing on sequence Consensus requires agreement on Sequence, but not modification localization 21
AB RF Proteome Informatics Research Group Peak lists Two types of peak lists were supplied – Deisotoped and non deisotoped Can only tell fragment charge state from non- deisotoped Requires search engine to be able to de-isotope spectrum 22
AB RF Proteome Informatics Research Group Peaklists 23 Number of spectra with undefined precursor charge state Deisotoped 1031 (304 in consensus results) Non-deisotoped 6094 (1140 in consensus results) For 1013 out of 7840 consensus spectra the precursor m/z differ by greater than 0.02 Da between deisotoped and non-deisotoped peak list. For 238 consensus spectra the peak lists had different specified charge state – 193 consensus results only possible with deisotoped peak list – 45 consensus results only possible with non-deisotoped peak list – For 19 consensus results multiple people who searched the nd peak list agreed on a confident different answer – For 4 consensus results multiple people who searched the deisotoped peak list agreed on a confident different answer
AB RF Proteome Informatics Research Group Mixed Spectra 24 465.19 2+ 464.59 3+ Non-deisotoped peaklist 465.19 2+ Deisotoped peaklist
AB RF Proteome Informatics Research Group Synthetic Peptide ID by Peptide 25 Sulfo Phospho Nitro Trimethyl Methyl (R) Dimethyl (R) Acetyl Methyl (K) Dimethyl (K) # participants
AB RF Proteome Informatics Research Group Correct Localization of Modified Synthetic Peptides 27 70 synthetic modified peptides were spiked into sample. 7 of these were confidently found by no participant Correct localization & name of modification reported
AB RF Proteome Informatics Research Group FLR of Modified Synthetic Peptides 28 FLR = 100% * # PSMs wrong localization of s,t,y,k,r # PSMs wrong + right localization of s,t,y,k,r Ignored PSMs contain mods of residues other than s,t,y,k,r. Sample handling mods (n,q,d,e, etc). 5% 1%1-2% 1%<10.510% <30% 0.01<5% <1%
AB RF Proteome Informatics Research Group Incorrect Localization by Peptide 29 71755v58288v 33564 93128i 1121158409 87133i94158i97053i42424i77777i 23068 40104i87048i 92653 34284i 2311774564141515278147603141524551111821 EKLLDFIK AEGSEIRLAK1 VDATEESDLAQQYGVR TITLEVEPSDTIENVK ESTLHLVLR AEFAEVSK1 LKLVSELWDAGIK DQGGELLSLR TYETTLEK1 NGDTASPKEYTAGR144141 LKAEGSEIR TVIDYNGER4344441133 ADEGISFR YKPESDELTAEK GTRDYSPR1 VPQVSTPTLVEVSR ADEGISFRGLFIIDDK ALAPEYAK TIAQDYGVLK111121111 THILLFLPKSVSDYEGK536811241 WVTFISLLFLFSSAYSR IFSIVEQR TLSDYNIQK31 GILRQITVNDLPVGR NVAVDELSR LDELRDEGK ESTLHLVLRLR1 DEGKASSAK SVSDYEGK111111 LVQAFQFTDK LVNEVTEFAK1 GLFIIDDKGILR1 FPKAEFAEVSK LKAQLGPDESK DISLSDYK1123136 FKDLGEENFK Number of PSMs with Incorrect Site Localization – Mod Loc Confidence Y Present as sulfo-Tyr Present as phospho S-10 often mislocalized as S-12 or Y-14 Present as mono, di, tri methyl K often mislocalized at R
AB RF Proteome Informatics Research Group 30 DISLSDY(Phospho)K DISLSDY(Sulfo)K Observe modified fragment ions. Observe unmodified fragment ions. Spectrum looks essentially identical to unmodified peptide spectrum Phospho vs Sulfo
AB RF Proteome Informatics Research Group Conclusions Reasonable number of participants from around the globe, mainly experienced users but a few first-timers Large spread in number of spectra identified False negatives (NS) are generally much higher than false positives, so there is generally room for improvement Peak list was a significant factor on performance Varied performance in detecting PTMs Most participants struggled with sulfation Multiply phosphorylated harder to find than singly Most common errors in site assignment were: Reporting sulfo(Y) as phospho(ST) Mis-assignment of site/s in multiply phosphorylated peptides 31
AB RF Proteome Informatics Research Group What did the participants think? 22 out of 24 participants found the study useful 32 Too many modifications at the same time. Manual validation is necessary and the right time necessary for this study is too demanding for this challenge. The spiked proteins made it possible to game the study - look for the uncommon modifications only on the spikes. Of course we didn't do this. Overall I'd say this was a flawed but very interesting ABRF study.
AB RF Proteome Informatics Research Group 33 Before After Participants Confidence in Analyzing PTM Data
AB RF Proteome Informatics Research Group 34 How difficult do you think this study was? What was your total analysis time for the entire project?
AB RF Proteome Informatics Research Group 35 Based on this study, would you consider participating in future ABRF studies?
AB RF Proteome Informatics Research Group Thank you! Questions? iPRG Nuno Bandeira Robert Chalkley(chair) Matt Chambers Karl Clauser John Cottrell Eric Deutsch Eugene Kapp Henry Lam Hayes McDonald Tom Neubert (EB liaison) Ruixiang Sun Dataset Creation Chris Colangelo Anonymizer: Jeremy Carver, UCSD THANK YOU TO ALL STUDY PARTICIPANTS! 36