1. Read Processing and Mapping: From Raw to Analysis-ready Reads
Ben Passarelli
Stem Cell Institute Genome Center
NGS Workshop
31 MAY 2013

2. From Raw to Analysis-ready Reads
Raw reads
Read assessment and prep
Mapping
Duplicate
Marking
Local realignment
Base quality recalibration
Analysis-ready
reads
Session Topics
Overview of high-throughput sequencing platforms
Understand read data formats and quality scores
Identify and fix some common read data problems
Find a genomic reference for mapping
Mapping reads to a reference genome
Understand alignment output
Sort, merge, index alignment for further analysis
Mark/eliminate duplicate reads
Locally realign at indels
Recalibrate base quality scores
How to get started

3. Sample to Raw Reads Library Construction QC and Quantification Sample
Preparation
Sequencing
Raw Reads

4. Sequence Data Instrument Output (FASTQ Format) Images (.tiff)
Illumina
MiSeq
Illumina
HiSeq
Ion PGM
Ion Proton
Pacific Biosciences RS
Images (.tiff)
Cluster intensity file (.cif)
Base call file (.bcl)
Standard flowgram file (.sff)
Movie
Trace (.trc.h5)
Pulse (.pls.h5)
Base (.bas.h5)
Sequence Data
(FASTQ Format)

5. Sequencing Platforms at a Glance

6. Solid Phase Amplification
V3 HiSeq
Sequencing Steps
Clusters are linearized
Sequencing primer annealed
All four dNTPs added at each cycle
Each with different **Fluorescent Tag**
Intensity of different tags  base call
Error Profile: substitutions
Library DNA binds to Oligos Immobilized on Glass Flowcell Surface

7. FASTQ Format (Illumina Example)
Flow Cell ID
Lane
Tile
Tile Coordinates
Barcode
Read Record
Header
@DJG84KN1:272:D17DBACXX:2:1101:12432:5554 1:N:0:AGTCAA
CAGGAGTCTTCGTACTGCTTCTCGGCCTCAGCCTGATCAGTCACACCGTT +
BCCFFFDFHHHHHIJJIJJJJJJJIJJJJJJJJJJIJJJJJJJJJIJJJJ
@DJG84KN1:272:D17DBACXX:2:1101:12454:5610 1:N:0:AG
AAAACTCTTACTACATCAGTATGGCTTTTAAAACCTCTGTTTGGAGCCAG
@DJG84KN1:272:D17DBACXX:2:1101:12438:5704 1:N:0:AG
CCTCCTGCTTAAAACCCAAAAGGTCAGAAGGATCGTGAGGCCCCGCTTTC
@DJG84KN1:272:D17DBACXX:2:1101:12340:5711 1:N:0:AG
GAAGATTTATAGGTAGAGGCGACAAACCTACCGAGCCTGGTGATAGCTGG
CCCFFFFFHHHHHGGIJJJIJJJJJJIJJIJJJJJGIJJJHIIJJJIJJJ
Read Bases
Separator
(with optional repeated header)
Read Quality Scores
NOTE: for paired-end runs, there is a second file
with one-to-one corresponding headers and reads

8. Base Call Quality: Phred Quality Scores
Phred* quality score Q with base-calling error probability P
Q = -10 log10P
* Name of first program to assign accurate base quality scores. From the Human Genome Project.
Q score
Probability of base error
Base confidence
Sanger-encoded
(Q Score + 33) ASCII character 10
0.1
90%
“+” 20
0.01
99%
“5” 30
0.001
99.9%
“?” 40
0.0001
99.99%
“I”
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL
| | | | | |
S - Sanger Phred+33 range: 0 to 40
I - Illumina 1.3+ Phred+64 range: 0 to 40
L - Illumina 1.8+ Phred+33 range: 0 to 41

9. Initial Read Assessment and Processing
Raw reads
Read assessment and prep
Mapping
Duplicate
Marking
Local realignment
Base quality recalibration
Analysis-ready
reads
Common problems that can affect analysis
Low confidence base calls
typically toward ends of reads
criteria vary by application
Presence of adapter sequence in reads
poor fragment size selection
protocol execution or artifacts
Over-abundant sequence duplicates
Library contamination

10. Quick Read Assessment: FastQC
Free Download
Download:
Tutorial :
Samples reads (200K default): fast, low resource use

11. Read Assessment Example (Cont’d)
Trim leading bases
(library artifact)
Trim for base quality or adapters
(run or library issue)

12. Read Assessment Example (Cont’d)
TruSeq Adapter, Index 9
5’ GATCGGAAGAGCACACGTCTGAACTCCAGTCACGATCAGATCTCGTATGCCGTCTTCTGCTTG

13. Comprehensive Read Assessment: Prinseq

14. Selected Tools to Process Reads
Fastx toolkit*
(partial list)
FASTQ Information: Chart Quality Statistics and Nucleotide Distribution
FASTQ Trimmer: Shortening FASTQ/FASTA reads (removing barcodes or noise).
FASTQ Clipper: Removing sequencing adapters
FASTQ Quality Filter: Filters sequences based on quality
FASTQ Quality Trimmer: Trims (cuts) sequences based on quality
FASTQ Masker: Masks nucleotides with 'N' (or other character) based on quality
*defaults to old Illumina fastq (ASCII offset 64). Use –Q33 option.
SepPrep
Adapter trimming
Merge overlapping paired-end read
Biopython
(for python programmers)
Especially useful for implementing custom/complex sequence analysis/manipulation
Galaxy
Great for beginners: upload data, point and click
Just about everything you’ll see in today’s presentations
SolexaQA2
Dynamic trimming
Length sorting (resembles read grouping of Prinseq)

15. Many Analysis Pipelines Start with Read Mapping

16. Read Mapping http://www.broadinstitute.org/igv/ Raw reads
Read assessment and prep
Mapping
Duplicate
Marking
Local realignment
Base quality recalibration
Analysis-ready
reads

17. Sequence References and Annotations
Comprehensive reference information
Comprehensive reference, annotation, and translation information
References and SNP information data by GATK
Human only
Pre-indexed references and gene annotations for Tuxedo suite
Human, Mouse, Rat , Cow, Dog, Chicken, Drosophila, C. elegans, Yeast

18. One or more sequences per file
Fasta Sequence Format
One or more sequences per file
“>” denotes beginning of sequence or contig
Subsequent lines up to the next “>” define sequence
Lowercase base denotes repeat masked base
Contig ID may have comments delimited by “|”
>chr1 …
TGGACTTGTGGCAGGAATgaaatccttagacctgtgctgtccaatatggt
agccaccaggcacatgcagccactgagcacttgaaatgtggatagtctga
attgagatgtgccataagtgtaaaatatgcaccaaatttcaaaggctaga
aaaaaagaatgtaaaatatcttattattttatattgattacgtgctaaaa
taaccatatttgggatatactggattttaaaaatatatcactaatttcat
>chr2
>chr3

19. Read Mapping Novoalign (3.0) SOAP3 (version 91) BWA (0.7.4) Bowtie2
Novoalign
(3.0)
SOAP3
(version 91)
BWA
(0.7.4)
Bowtie2
(2.1.0)
Tophat2
(2.0.8b)
STAR
(2.3.0e)
License
Commercial
GPL v3
Artistic
Mismatch allowed
up to 8
up to 3
user specified. max is function of read length and error rate
user specified
uses Bowtie2
Alignments reported per read
random/all/none
user selected
Gapped alignment
up to 7bp
1-3bp gap
yes
splice junctions
introns
Pair-end reads
Best alignment
highest alignment score
minimal number of mismatches
Trim bases
3’ end
3’ and 5’ end
Comments
At one time, best performance and alignment quality
Element of Broad’s “best practices” genotyping workflow
Smith-Waterman quality alignments, currently fastest
Currently most popular RNA-seq aligner
Very fast; uses memory to achieve performance

20. Read Mapping: BWA BWA Features Uses Burrows Wheeler Transform fast
modest memory footprint (<4GB)
Accurate
Tolerates base mismatches
increased sensitivity
reduces allele bias
Gapped alignment for both single- and paired-ended reads
Automatically adjusts parameters based on read lengths and error rates
Native BAM/SAM output (the de facto standard)
Large installed base, well-supported
Open-source (no charge)

21. Read Mapping: Bowtie 2 Bowtie2
Uses dynamic programming (edit distance scoring)
Eliminates need for realignment around indels
Can be tuned for different sequencing technologies
Multi-seed search - adjustable sensitivity
Input read length limited only by available memory
Fasta or Fastq input
Caveats
Longer input reads require much more memory
Trade-off parallelism with memory requirement
Dynamic Programming Illustration
Langmead B, Salzberg S. Fast gapped-read alignment with Bowtie 2, Nature Methods. 2012, 9:

22. SAM (BAM) Format Sequence Alignment/Map format Universal standard
Human-readable (SAM) and compact (BAM) forms
Structure
Header
version, sort order, reference sequences, read groups, program/processing history
Alignment records

23. SAM/BAM Format: Header
[benpass align_genotype]$ samtools view -H allY.recalibrated.merge.bam
@HD VN:1.0 GO:none SO:coordinate
@SQ SN:chrM LN:16571
@SQ SN:chr1 LN:
@SQ SN:chr2 LN:
@SQ SN:chr3 LN: …
@SQ SN:chr19 LN:
@SQ SN:chr20 LN:
@SQ SN:chr21 LN:
@SQ SN:chr22 LN:
@SQ SN:chrX LN:
@SQ SN:chrY LN:
@RG ID: PL:ILLUMINA LB:IL500 SM:
@RG ID:BsK010 PL:ILLUMINA LB:IL501 SM:BsK010-1
@RG ID:Bsk136 PL:ILLUMINA LB:IL502 SM:Bsk136-1
@RG ID:MAK001 PL:ILLUMINA LB:IL503 SM:MAK001-1
@RG ID:NG87 PL:ILLUMINA LB:IL504 SM:NG87-1
@RG ID:SDH023 PL:ILLUMINA LB:IL508 SM:SDH023
@PG ID:GATK IndelRealigner VN: gd091f72 CL:knownAlleles=[] targetIntervals=tmp.intervals.list LODThresholdForCleaning=5.0 consensusDeterminationModel=USE_READS entropyThreshold=0.15 maxReadsInMemory= maxIsizeForMovement=3000 maxPositionalMoveAllowed=200 maxConsensuses=30 maxReadsForConsensuses=120 maxReadsForRealignment=20000 noOriginalAlignmentTags=false nWayOut=null generate_nWayOut_md5s=false check_early=false noPGTag=false keepPGTags=false indelsFileForDebugging=null statisticsFileForDebugging=null SNPsFileForDebugging=null
@PG ID:bwa PN:bwa VN:0.6.2-r126
samtools to view bam
header
sort order
reference sequence names with lengths
read groups with platform, library and sample information
program (analysis) history

24. SAM/BAM Format: Alignment Records
[benpass align_genotype]$ samtools view allY.recalibrated.merge.bam
HW-ST605:127:B0568ABXX:2:1201:10933: chr M =
TCATTTTATGGCCCCTTCTTCCTATATCTGGTAGCTTTTAAATGATGACCATGTAGATAATCTTTATTGTCCCTCTTTCAGCAGACGGTATTTTCTTATGC
RG:Z:86-191
HW-ST605:127:B0568ABXX:3:1104:21059: chr M =
ATGGCCCCTTCTTCCTATATCTGGTAGCTTTTAAATGATGACCATGTAGATAATCTTTATTGTCCCTCTTTCAGCAGACGGTATTTTCTTATGCTACAGTA
RG:Z:SDH023
* Many fields after column 12 deleted (e.g., recalibrated base scores) have been deleted for improved readability 2 3 4 5 6 8 9 1 10 11

25. Preparing for Next Steps
Raw reads
Read assessment and prep
Mapping
Duplicate
Marking
Local realignment
Base quality recalibration
Analysis-ready
reads
Subsequent steps require sorted and indexed bams
Sort orders: karyotypic, lexicographical
Indexing improves analysis performance
Picard tools: fast, portable, free
Sort: SortSam.jar
Merge: MergeSamFiles.jar
Index: BuildBamIndex.jar
Order: sort, merge (optional), index

26. Duplicate Marking
Raw reads
Read assessment and prep
Mapping
Duplicate
Marking
Local realignment
Base quality recalibration
Analysis-ready
reads
$java -Xmx4g -jar <path to picard>/MarkDuplicates.jar \
INPUT=aligned.sorted.bam \
OUTPUT=aligned.sorted.dedup.bam \
VALIDATION_STRINGENCY=LENIENT \
METRICS_FILE=aligned.dedup.metrics.txt \
REMOVE_DUPLICATES=false \
ASSUME_SORTED=true

27. SAM/BAM Format: Alignment Records
[benpass align_genotype]$ samtools view allY.recalibrated.merge.bam
HW-ST605:127:B0568ABXX:2:1201:10933: chr M =
TCATTTTATGGCCCCTTCTTCCTATATCTGGTAGCTTTTAAATGATGACCATGTAGATAATCTTTATTGTCCCTCTTTCAGCAGACGGTATTTTCTTATGC
RG:Z:86-191

28. Local Realignment
Raw reads
Read assessment and prep
Mapping
Duplicate
Marking
Local realignment
Base quality recalibration
Analysis-ready
reads
BWT-based alignment is fast for matching reads to reference
Individual base alignments often sub-optimal at indels
Approach
Fast read mapping with BWT-based aligner
Realign reads at indel sites using gold standard (but much slower) Smith-Waterman algorithm
Benefits
Refines location of indels
Reduces erroneous SNP calls
Very high alignment accuracy in significantly less time, with fewer resources
1Smith, Temple F.; and Waterman, Michael S. (1981). "Identification of Common Molecular Subsequences". Journal of Molecular Biology 147: 195–197. doi: / (81) PMID

29. Post re-alignment at indels
Local Realignment
Raw BWA alignment
Post re-alignment at indels
DePristo MA, et al. A framework for variation discovery and genotyping using next-generation DNA sequencing data. Nat Genet May;43(5): PMID:

30. Base Quality Recalibration
Raw reads
Read assessment and prep
Mapping
Duplicate
Marking
Local realignment
Base quality recalibration
Analysis-ready
reads
STEP 1: Find covariates at non-dbSNP sites using:
Reported quality score
The position within the read
The preceding and current nucleotide (sequencer properties)
java -Xmx4g -jar GenomeAnalysisTK.jar \
-T BaseRecalibrator \
-I alignment.bam \
-R hg19/ucsc.hg19.fasta \
-knownSites hg19/dbsnp_135.hg19.vcf \
-o alignment.recal_data.grp
STEP 2: Generate BAM with recalibrated base scores:
-T PrintReads \
-BQSR alignment.recal_data.grp \
-o alignment.recalibrated.bam

31. Base Quality Recalibration (Cont’d)

32. Raw reads Analysis-ready reads Mapping Duplicate Marking
Read assessment and prep
Mapping
Duplicate
Marking
Local realignment
Base quality recalibration
Analysis-ready
reads

33. Is there an easier way to get started?!
Click on “Use Galaxy”

34. Getting Started

35. Raw reads Analysis-ready reads Mapping Duplicate Marking
Read assessment and prep
Mapping
Duplicate
Marking
Local realignment
Base quality recalibration
Analysis-ready
reads