iPRG 2012: A Study on Detecting Modified Peptides in a Complex Mixture ABRF 2012, Orlando, FL 3/17-20/2012

iPRG2012 Study: DESIGN

Study Goals Primary: Evaluate the ability of participants to identify modified peptides present in a complex mixture Secondary: Find out why result sets might differ between participants Tertiary: Produce a benchmark dataset, along with an analysis resource

Study Design Use a common, rich dataset Use a common sequence database Allow participants to use the bioinformatic tools and methods of their choosing Use a common reporting template Report results at an estimated 1% FDR (at the spectrum level) Ignore protein inference

Sample Tryptic digest of yeast (RM8323 – NIST), spiked with 69 synthetic modified peptides (tryptic peptides from 6 different proteins – sPRG) Phospho (STY) Sulfo (Y) Mono-, di-, trimethyl (K) Mono-, dimethyl (R) Acetyl (K) Nitro (Y)

Supplied Study Materials 5600 TripleTOF dataset (i.e. WIFF file) WIFF, mzML, dta, MGF (de-isotoped);– conversions by MS Data Converter 1.1.0 MGF (not de-isotoped – conversion by Mascot Distiller 2.4) 1 fasta file (UniProtKB/SwissProt S. cerevisiae, human, + 1 bovine protein + trypsin from Dec. 2011) 1 template (Excel) 1 on-line survey (Survey Monkey)

Instructions to Participants Retrieve and analyze the data file in the format of your choosing, with the method(s) of your choosing Report the peptide to spectrum matches in the provided template Report measures of reliability for PTM site assignments (optional) Fill out the survey Attach a 1-2 page description of the methodology employed

iPRG 2012 STUDY: PARTICIPATION

Soliciting Participants and Logistics Study advertised on the ABRF website and listserv and by direct invitation from iPRG members 1. Email participation request to ‘iPRGxxxx@gmail.com’ Participant 2. Send official study letter with instructions iPRG members Questions / Answers 3. All further communication (e.g., questions, submission) through ‘iPRGxxx.anonymous@gmail.com’ “Anonymizer” 9

Participants (i) – overall numbers 24 submissions One participant submitted two result sets 9 initialed iPRG member submissions (with appended ‘i’) 2 vendor submissions (identifiable by appended ‘v’)

About the Participant 11

About the Participant’s Lab 12

Participation in sPRG Study Only one participant indicated he used sPRG information to aid his analysis. This person was one of the least successful in identifying the spiked-in peptides!

Search Engine Used 14

Site Localization Software 4 participants did not list using software for site localization.

Summary of Submitted Results Only reported modified peptides

Summary of IDs and Localizations Peptide Identification in all Spectra Site Localization in Spectra With Interesting Modifications There is a very wide range in the total number of spectra with identified peptides. Once one focuses only on the spectra containing modifcations for which the ability to localize the modification to a particular residue, the range is much narrower. The 5 rightmost participants went so far as to reported only spectra of modified peptides.

Overlap of spectrum identifications 7840 agreed on by 3 or more participants We selected 3 participants agreeing as the threshold for denoting consensus agreement. Consensus requires agreement on sequence, so do note that this still allows for disagreement on modification localization.

Room for improvement in thresholding? 33564 93128i 11211 58409 87133i 94158i 97053i 42424i 77777i 23068 40104i 87048i 92653 34284i 23117 74564 14151 52781 47603 14152 45511 11821 Peaklist mgf mzML mgf_nd   WIFF Spectral Pre-Processing Pk PPi MDi Ot SM pF PD PkDB PW Sq Peptide Identification By M O PPr MG P/PP ST MM TPP XT IH Discovery of Unexpected Mods MO Site Localization MDe AS An Sc Results Filtering IP XL PR NTT 2 1 ? Experience 5-10 years >10 years < 1 year 3-4 years 1-2 years The green (NS) bars represent the room for improving confidence threshold setting. If one could improve the decision making about confidence in a peptide spectral match, without increasing FDR then the sum of the heights of the blue (YS) and green (NS) bars appears to be within reach for many participants to substantially improve their overall identification totals. The gray bar corresponds to spectra, which for that participant were reported as Peptide Identification Certainty N and their reported sequence differs from the consensus sequence. These spectra are part of the consensus set for which 3 or more participants reported Peptide Identification Certainty Y and agreed on the sequence, but may have differed on site localization if modified. I.e. they are part of the blue bar of at least 3 other participants. Participants who allowed for semi-tryptic peptides generally were toward the left of the plot, although only about 2 % of the consensus results were semi-tryptic. An Andromeda/MaxQuant MG MS-GFDB pF pFind Sc Scaffold AS A-Score MM MyriMatch Pk PEAKS SM Spectrum Mill By Byonics MO MODa PkDB PEAKSDB Sq Sequest IH In-house software O OMSSA PPi Protein Pilot ST SpectraST IP IDPicker Ot Other PPr Protein Prospector TPP TransProteomic Pipeline M Mascot P/PP Pep/Prot Prophet PR PhosphoRS XL Excel MDe Mascot Delta Score PD ProteomeDiscoverer PW ProteoWizard XT X!Tandem MDi Mascot Distiller

ESR and FDR Extraordinary Skill Rate or High False Discovery Rate? ESR + FDR = 100* (Y<3P+YD)/total ids Y 24 participants 3 for consensus When particular identifications are reported by less than 3 participants it is difficult to tell whether that represents extraordinary skill or just another false positive. On the other hand, disagreement with the consensus (YD) is more likely to indicate a wrong answer. Consequently, the YD rate serves as a surrogate for the minimum FDR level. Note that many participants, especially those to the far right tend to have a YD rate much greater than 1%, which suggests they have underestimated their FDR level. Although the study was requested to be performed at 1% FDR, these far right participants reported much lower total numbers of ids. The 5 rightmost participants reported only modified peptides.

Characteristics of consensus spectra 7840 spectra >=3 participants agreeing on sequence The total numbers on the bars is > 7840 because some of the consensus spectra contain more than 1 modification. Consensus requires agreement on Sequence, but not modification localization

Peak lists Two types of peak lists were supplied Deisotoped and non deisotoped Can only tell fragment charge state from non-deisotoped Requires search engine to be able to de-isotope spectrum

Peaklists Number of spectra with undefined precursor charge state Deisotoped 1031 (304 in consensus results) Non-deisotoped 6094 (1140 in consensus results) For 1013 out of 7840 consensus spectra the precursor m/z differ by greater than 0.02 Da between deisotoped and non-deisotoped peak list. For 238 consensus spectra the peak lists had different specified charge state 193 consensus results only possible with deisotoped peak list 45 consensus results only possible with non-deisotoped peak list For 19 consensus results multiple people who searched the nd peak list agreed on a confident different answer For 4 consensus results multiple people who searched the deisotoped peak list agreed on a confident different answer

Mixed Spectra 465.19 2+ 464.59 3+ 465.19 2+ Deisotoped peaklist Non-deisotoped peaklist

Synthetic Peptide ID by Peptide Trimethyl Sulfo Methyl (K) Methyl (R) Phospho Dimethyl (K) Counts correspond to the number of participants reporting at least 1 PSM for the spiked synthetic peptide modified with the correct localization reported (case sensitive string compare) and the correct modification name. The localization certainty may have been reported as either Y or N. PSM’s containing modifcation of residues other than s,t,y,k,r were excluded. Dimethyl (R) Acetyl Nitro # participants # participants

Synthetic Peptide ID by Participant 71755v 58288v 33564 93128i 11211 58409 87133i 94158i 97053i 42424i 77777i 23068 40104i 87048i 92653 34284i 23117 74564 14151 52781 47603 14152 45511 11821 Acetyl (K) 1 Dimethyl (K) Dimethyl (R) Methyl (K) Methyl (R) Nitro (Y) Phospho (STY) Sulfo (Y) Trimethyl (K) Red corresponds to the presence of at least 1 PSM for a spiked synthetic peptide modified with the correct localization reported (case sensitive string compare) and the correct modification name. The localization certainty may have been reported as either Y or N. PSM’s containing modifcation of residues other than s,t,y,k,r were excluded.

Correct Localization of Modified Synthetic Peptides 70 synthetic modified peptides were spiked into sample. 7 of these were confidently found by no participant Correct localization & name of modification reported

FLR of Modified Synthetic Peptides FLR = 100% * # PSMs wrong localization of s,t,y,k,r # PSMs wrong + right localization of s,t,y,k,r Ignored PSMs contain mods of residues other than s,t,y,k,r . Sample handling mods (n,q,d,e, etc). Values below second chart are participant’s estimate of the reliability of their site localizations. 5% 1% 1-2% <1 0.5 10% <30% 0.01 <5% <1%

Incorrect Localization by Peptide Number of PSM’s with Incorrect Site Localization – Mod Loc Confidence Y Present as sulfo-Tyr Present as phospho S-10 often mislocalized as S-12 or Y-14 Present as mono, di, tri methyl K often mislocalized at R 71755v 58288v 33564 93128i 11211 58409 87133i 94158i 97053i 42424i 77777i 23068 40104i 87048i 92653 34284i 23117 74564 14151 52781 47603 14152 45511 11821 EKLLDFIK AEGSEIRLAK 1 VDATEESDLAQQYGVR TITLEVEPSDTIENVK ESTLHLVLR AEFAEVSK LKLVSELWDAGIK DQGGELLSLR TYETTLEK NGDTASPKEYTAGR 4 LKAEGSEIR TVIDYNGER 3 ADEGISFR YKPESDELTAEK GTRDYSPR VPQVSTPTLVEVSR ADEGISFRGLFIIDDK ALAPEYAK TIAQDYGVLK 2 THILLFLPKSVSDYEGK 5 6 8 WVTFISLLFLFSSAYSR IFSIVEQR TLSDYNIQK GILRQITVNDLPVGR NVAVDELSR LDELRDEGK ESTLHLVLRLR DEGKASSAK SVSDYEGK LVQAFQFTDK LVNEVTEFAK GLFIIDDKGILR FPKAEFAEVSK LKAQLGPDESK DISLSDYK FKDLGEENFK Incorrect localizations were limited to spectra from a few particular peptides.

Phospho vs Sulfo DISLSDY(Phospho)K Observe modified fragment ions. DISLSDY(Sulfo)K Observe modified fragment ions. Observe ‘unmodified’ fragment ions. Spectrum looks essentially identical to unmodified peptide spectrum

Conclusions Reasonable number of participants from around the globe, mainly experienced users but a few first-timers Large spread in number of spectra identified False negatives (NS) are generally much higher than false positives, so there is generally room for improvement Peak list was a significant factor on performance Varied performance in detecting PTMs Most participants struggled with sulfation Multiply phosphorylated harder to find than singly Most common errors in site assignment were: Reporting sulfo(Y) as phospho(ST) Mis-assignment of site/s in multiply phosphorylated peptides

What did the participants think? “The spiked proteins made it possible to game the study - look for the uncommon modifications only on the spikes. Of course we didn't do this. Overall I'd say this was a flawed but very interesting ABRF study.” 22 out of 24 participants found the study useful “Too many modifications at the same time. Manual validation is necessary and the right time necessary for this study is too demanding for this challenge.”

Participant’s Confidence in Analyzing PTM Data Before After 33

How difficult do you think this study was? What was your total analysis time for the entire project?

Based on this study, would you consider participating in future ABRF studies?

THANK YOU TO ALL STUDY PARTICIPANTS! Thank you! Questions? THANK YOU TO ALL STUDY PARTICIPANTS! iPRG Nuno Bandeira Robert Chalkley(chair) Matt Chambers Karl Clauser John Cottrell Eric Deutsch Eugene Kapp Henry Lam Hayes McDonald Tom Neubert (EB liaison) Ruixiang Sun Dataset Creation Chris Colangelo Anonymizer: Jeremy Carver, UCSD