PAGE: A Framework for Easy Parallelization of Genomic Applications 1 Mucahid Kutlu Gagan Agrawal Department of Computer Science and Engineering The Ohio State University IPDPS 2014, Phoenix, Arizona

Motivation The sequencing costs are decreasing IPDPS'14 2 *Adapted from genome.gov/sequencingcosts

Big data problem – 1000 Human Genome Project already produced 200 TB data – Parallel processing is inevitable! *Adapted from Motivation IPDPS'14 3

Typical Analysis on Genomic Data Single Nucleotide Polymorphism (SNP) calling IPDPS'14 4 Sequences Read-1 AGCG Read-2 GCGG Read-3 GCGTA Read-4 CGTTCC Alignment File-1 Reference AGCGTACC Sequences Read-1 AGAG Read-2 AGAGT Read-3 GAGT Read-4 GTTCC Alignment File-2 *Adapted from Wikipedia A single SNP may cause Mendelian disease! ✖✓ ✖

Outline Motivation Existing Solutions for Implementation Our Work Experimental Evaluation Conclusion IPDPS'14 5

Existing Solutions for Implementation Serial tools – SamTools, VCFTools, BedTools – File merging, sorting etc. – VarScan – SNP calling Parallel implementations – Turboblast, searching local alignments, – SEAL, read mapping and duplicate removal – Biodoop, statistical analysis Middleware Systems – Hadoop Not designed for specific needs of genetic data Limited programmability – Genome Analysis Tool Kit (GATK) Designed for genetic data processing Provides special data traversal patterns Limited parallelization for some of its tools IPDPS'14 6

Outline Motivation Existing Solutions for Implementation Our Work Experimental Evaluation Conclusion IPDPS'14 7

Our Goal We want to develop a middleware system – Specific for parallel genetic data processing – Allow parallelization of a variety of genetic algorithms – Be able to work with different popular genetic data formats – Allows use of existing programs IPDPS'14 8

Challenges Load Imbalance due to nature of genomic data – It is not just an array of A, G, C and T characters High overhead of tasks I/O contention IPDPS' Coverage Variance

Our Work PAGE: A Map-Reduce-like middleware for easy parallelization of genomic applications Mappers and reducers are executable programs – Allows us to exploit existing applications – No restriction on programming language IPDPS'14 10

File-m File-2 File-1 Map Reduce Region-1 Map Region-n Intra-dependent Processing IPDPS'14 11 O-1 1 O-1 n Output-1 Map Reduce Region-1 Map Region-n O-m 1 O-m n Output -m Each file is processed independently

Map O1O1 O1O1 OkOk OkOk OnOn OnOn Reduce Output Region-1 Input Files Map Region-k Map Region-n Inter-dependent Processing Each map task processes a particular region of ALL files IPDPS'14 12

What Can PAGE Parallelize? PAGE can parallelize all applications that have the following property M - Map task R, R 1 and R 2 are three regions such that R = concatenation of R 1 and R 2 M (R) = M(R 1 ) ⊕ M(R 2 ) where ⊕ is the reduction function IPDPS'14 13 R1R1 R2R2 R

Data Partitioning Data is NOT packaged into equal-size data blocks as in Hadoop – Each application has a different way of reading the data – Equal-size data block packaging ignores nucleotide base location information Genome structure is divided into regions and each map task is assigned for a region. – Takes account location information – The map task is responsible of accessing particular region of the input files It is a common feature for many genomic tools (GATK, SamTools) IPDPS'14 14

Genome Partition PAGE provides two data partitioning methods – By-locus partitioning: Chromosomes are divided into regions – By-chromosome partitioning: Chromosomes preserve their unity IPDPS'14 15 Chr-1 Chr-2 Chr-3 Chr-4 Chr-5 Chr-6 Chr-1 Chr-2 Chr-3 Chr-4 Chr-5 Chr-6

Task Scheduling Static Each processor is responsible of regions with equal length. All map tasks should finish before the execution of reduce tasks. Dynamic Map & reduce tasks are assigned by a master process Reduce tasks can start if there are enough available intermediate results. IPDPS'14 16 PAGE provides two types of scheduling schemes.

Applications Developed Using PAGE We parallelized 4 applications – VarScan: SNP detection – Realigner Target Creator: Detects insertion/deletions in alignment files – Indel Realigner: Applies local realignment to improve quality of alignment files – Unified Genotyper: SNP detection IPDPS'14 17

Sample Application Development with PAGE Serial execution command of VarScan Software – samtools mpileup –b file_list -f reference | java -jar VarScan.jar mpileup2snp To parallelize VarScan with PAGE, user needs to define: – Genome Partition: By-Locus – Scheduling Scheme: Dynamic (or Static) – Execution Model: Inter-dependent – Map command: samtools mpileup –b file_list -r regionloc -f reference | java -jar VarScan.jar mpileup2snp >outputloc – Reduction : cat bash shell command IPDPS'14 18

Outline Motivation Existing Solutions for Implementation Our Work Experimental Evaluation Conclusion IPDPS'14 19

Experiments Experimental Setup – In our cluster Each node has 12 GB memory 8 cores (2.53 GHz) – We obtained the data from 1000 Human Genome Project – We evaluated PAGE with 4 applications – We compared PAGE with Hadoop Streaming and GATK IPDPS'14 20

Comparison with GATK IPDPS'14 21 ScalabilityData Size Impact - Indel Realigner tool of GATK Data Size: 11 GB# of cores: x 9x

Comparison with GATK IPDPS'14 22 ScalabilityData Size Impact - Unified Genotyper tool of GATK 10.9x 12.8x Data Size: 34 GB# of cores: 128

ScalabilityData Size Impact - VarScan Application 6.9x 12.7x Comparison with Hadoop Streaming IPDPS'14 23 Data Size: 52 GB# of cores: 128

Summary of Experimental Results When the computing power increased by 16 times IPDPS'14 24 Indel Realigner Unified Genotyper VarScanRealigner Target Creator PAGE9x12.8x12.7x14.1x GATK3.3x10.9x-- Hadoop Streaming --6.9x-

Conclusion We developed a middleware – Easily parallelizes genomic applications – High applicability No restriction on programming language or data format Allows to use existing applications – Provides user to control the parallel execution while hiding the details Alternative scheduling schemes, execution models and data partitioning types – Good Scalability IPDPS'14 25

Thank you for listening … IPDPS'14 26 Questions