1. SALSASALSASALSASALSA Applying Twister for Scientific Applications NSF Cloud PI Workshop March 17, 2011 Judy Qiu http://salsahpc.indiana.edu School of Informatics and Computing Indiana University

2. Twister v0.9 March 15, 2011 New Infrastructure for Iterative MapReduce Programming SALSA Group Bingjing Zhang, Yang Ruan, Tak-Lon Wu, Judy Qiu, Adam Hughes, Geoffrey Fox, Applying Twister to Scientific Applications, Proceedings of IEEE CloudCom 2010 Conference, Indianapolis, November 30-December 3, 2010 Auto generation of partition files and configureMaps Auto configuration to setup Twister environment automatically on a cluster Concurrent file loading in Mapper configuration phase and file loading balancing Performance improvement (e.g. JVM Tuning) Scalability

3. Iteratively refining operation Typical MapReduce runtimes incur extremely high overheads – New maps/reducers/vertices in every iteration – File system based communication Long running tasks and faster communication in Twister enables it to perform close to MPI Time for 20 iterations K-Means Clustering map reduce Compute the distance to each data point from each cluster center and assign points to cluster centers Compute new cluster centers Compute new cluster centers User program

4. SALSASALSA Motivation Data Deluge MapReduce Classic Parallel Runtimes (MPI) Experiencing in many domains Data Centered, QoSEfficient and Proven techniques Input Output map Input map reduce Input map reduce iterations Pij Expand the Applicability of MapReduce to more classes of Applications Map-OnlyMapReduce Iterative MapReduce More Extensions

5. SALSASALSA A Programming Model for Iterative MapReduce Distributed data access In-memory MapReduce Distinction on static data and variable data (data flow vs. δ flow) Cacheable map/reduce tasks (long running tasks) Combine operation Support fast intermediate data transfers Reduce (Key, List ) Iterate Map(Key, Value) Combine (Map ) User Program Close() Configure() Static data Static data δ flow

6. Bare-metal Nodes Linux Virtual Machines Microsoft Dryad / TwisterApache Hadoop / Twister Data Mining Services in the Cloud Smith Waterman Dissimilarities, PhyloD Using DryadLINQ, Clustering, Multidimensional Scaling, Generative Topological Mapping, etc Xen, KVM SaaS Applications/ Workflow Cloud Platform Cloud Infrastructure Hardware Nimbus, Eucalyptus, OpenStack, OpenNebula Hypervisor/ Virtualization Windows Virtual Machines Linux Virtual Machines Windows Virtual Machines Apache PigLatin/Microsoft DryadLINQ/Google Sawzall Higher Level Languages Cloud Technologies and Their Applications

7. SALSASALSA MPI & Iterative MapReduce papers MapReduce on MPI Torsten Hoefler, Andrew Lumsdaine and Jack Dongarra, Towards Efficient MapReduce Using MPI, Recent Advances in Parallel Virtual Machine and Message Passing Interface Lecture Notes in Computer Science, 2009, Volume 5759/2009, 240-249 MPI with generalized MapReduce Jaliya Ekanayake, Hui Li, Bingjing Zhang, Thilina Gunarathne, Seung-Hee Bae, Judy Qiu, Geoffrey Fox Twister: A Runtime for Iterative MapReduce, Proceedings of the First International Workshop on MapReduce and its Applications of ACM HPDC 2010 conference, Chicago, Illinois, June 20-25, 2010 Grzegorz Malewicz, Matthew H. Austern, Aart J. C. Bik, James C. Dehnert, Ilan Horn, Naty Leiser, and Grzegorz Czajkowski Pregel: A System for Large-Scale Graph Processing, Proceedings of the 2010 international conference on Management of data Indianapolis, Indiana, USA Pages: 135-146 2010 Yingyi Bu, Bill Howe, Magdalena Balazinska, Michael D. Ernst HaLoop: Efficient Iterative Data Processing on Large Clusters, Proceedings of the VLDB Endowment, Vol. 3, No. 1, The 36th International Conference on Very Large Data Bases, September 1317, 2010, Singapore. Matei Zaharia, Mosharaf Chowdhury, Michael J. Franklin, Scott Shenker, Ion Stoica Spark: Cluster Computing with Working Sets poster at http://radlab.cs.berkeley.edu/w/upload/9/9c/Spark-retreat-poster-s10.pdf http://radlab.cs.berkeley.edu/w/upload/9/9c/Spark-retreat-poster-s10.pdf Russel Power, Jinyang Li, Piccolo: Building Fast, Distributed Programs with Partitioned Tables, OSDI 2010, Vancouver, BC, Canada

8. SALSASALSA Features of Existing Architectures Programming Model (SPMD) – MapReduce (Optionally “map-only”) – Focus on Single Step MapReduce computations (DryadLINQ supports more than one stage) Input and Output Handling – Distributed data access (HDFS in Hadoop, Sector in Sphere, and shared directories in Dryad) – Outputs normally goes to the distributed file systems Intermediate data – Transferred via file systems (Local disk-> HTTP -> local disk in Hadoop) – Easy to support fault tolerance – Considerably high latencies Fault Tolerance Google, Apache Hadoop, Sector/Sphere, Dryad/DryadLINQ (DAG based)

9. SALSASALSA Twister Architecture Scripts for file manipulations Twister daemon is a process, but Map/Reduce tasks are Java Threads (Hybrid approach) M M Local R R Twister Daemon Twister Daemon Map output goes directly to reducer Twister Daemon Twister Daemon Local Reduce output goes to local disk OR to Combiner 1 1 3 3 4 4 Read static data from local disk 1 1 B B B B B B B B Broker Connection Receive static data (1) OR Variable data (key,value) via the brokers (2) 4 4 2 2 Broker Network Twister Driver Main program

10. SALSASALSA Twister Programming Model configureMaps(..) Two configuration options : 1.Using local disks (only for maps) 2.Using pub-sub bus configureReduce(..) runMapReduce(..) while(condition){ } //end while updateCondition() close() User program’s process space Combine() operation Reduce() Map() Worker Nodes Communications/data transfers via the pub-sub broker network Iterations May send pairs directly Local Disk Cacheable map/reduce tasks

11. SALSASALSA Twister API 1.configureMaps(PartitionFile partitionFile) 2.configureMaps(Value[] values) 3.configureReduce(Value[] values) 4.runMapReduce() 5.runMapReduce(KeyValue[] keyValues) 6.runMapReduceBCast(Value value) 7.map(MapOutputCollector collector, Key key, Value val) 8.reduce(ReduceOutputCollector collector, Key key,List values) 9.combine(Map keyValues) 1.configureMaps(PartitionFile partitionFile) 2.configureMaps(Value[] values) 3.configureReduce(Value[] values) 4.runMapReduce() 5.runMapReduce(KeyValue[] keyValues) 6.runMapReduceBCast(Value value) 7.map(MapOutputCollector collector, Key key, Value val) 8.reduce(ReduceOutputCollector collector, Key key,List values) 9.combine(Map keyValues)

12. SALSASALSA Pagerank – An Iterative MapReduce Algorithm Well-known pagerank algorithm [1] Used ClueWeb09 [2] (1TB in size) from CMU Reuse of map tasks and faster communication pays off [1] Pagerank Algorithm, http://en.wikipedia.org/wiki/PageRankhttp://en.wikipedia.org/wiki/PageRank [2] ClueWeb09 Data Set, http://boston.lti.cs.cmu.edu/Data/clueweb09/http://boston.lti.cs.cmu.edu/Data/clueweb09/ M R Current Page ranks (Compressed) Partial Adjacency Matrix Partial Updates C Partially merged Updates Iterations

13. Overhead OpenMPI v Twister negative overhead due to cache http://futuregrid.org13

14. SALSASALSA Dimension Reduction Algorithms Multidimensional Scaling (MDS) [1] o Given the proximity information among points. o Optimization problem to find mapping in target dimension of the given data based on pairwise proximity information while minimize the objective function. o Objective functions: STRESS (1) or SSTRESS (2) o Only needs pairwise distances  ij between original points (typically not Euclidean) o d ij (X) is Euclidean distance between mapped (3D) points Generative Topographic Mapping (GTM) [2] o Find optimal K-representations for the given data (in 3D), known as K-cluster problem (NP-hard) o Original algorithm use EM method for optimization o Deterministic Annealing algorithm can be used for finding a global solution o Objective functions is to maximize log- likelihood: [1] I. Borg and P. J. Groenen. Modern Multidimensional Scaling: Theory and Applications. Springer, New York, NY, U.S.A., 2005. [2] C. Bishop, M. Svens´en, and C. Williams. GTM: The generative topographic mapping. Neural computation, 10(1):215–234, 1998.

15. SALSASALSA Multi-dimensional Scaling (EM) Maps high dimensional data to lower dimensions (typically 2D or 3D) SMACOF (Scaling by Majorizing of COmplicated Function)[1] [1] J. de Leeuw, "Applications of convex analysis to multidimensional scaling," Recent Developments in Statistics, pp. 133-145, 1977. While(condition) { = [A] [B] C = CalcStress( ) } While(condition) { = [A] [B] C = CalcStress( ) } While(condition) { = MapReduce1([B], ) = MapReduce2([A], ) C = MapReduce3( ) } While(condition) { = MapReduce1([B], ) = MapReduce2([A], ) C = MapReduce3( ) }

16. Next Generation Sequencing Pipeline on Cloud 16 Blast Pairwise Distance Calculation Pairwise Distance Calculation Dissimilarity Matrix N(N-1)/2 values Dissimilarity Matrix N(N-1)/2 values FASTA File N Sequences block Pairings MapReduce 123 Clustering Visualization Plotviz Visualization Plotviz 4 Visualization Plotviz Visualization Plotviz MDS Pairwise clustering Pairwise clustering MPI 4 5 Users submit their jobs to the pipeline and the results will be shown in a visualization tool. This chart illustrate a hybrid model with MapReduce and MPI. Twister will be an unified solution for the pipeline mode. The components are services and so is the whole pipeline. We could research on which stages of pipeline services are suitable for private or commercial Clouds.

17. SALSASALSA Scale-up Sequence Clustering Model with Twister Gene Sequences (N = 1 Million) Distance Matrix Interpolative MDS with Pairwise Distance Calculation Multi- Dimensional Scaling (MDS) Visualization 3D Plot Reference Sequence Set (M = 100K) N - M Sequence Set (900K) Select Reference Reference Coordinates x, y, z N - M Coordinates x, y, z Pairwise Alignment & Distance Calculation O(N 2 ) O(N 2 ) O(N 2 )

18. SALSASALSA Current Sequence Clustering Model with MPI Gene Sequences Pairwise Alignment & Distance Calculation Distance Matrix Pairwise Clustering Multi- Dimensional Scaling Visualization Cluster Indices Coordinates 3D Plot Smith-Waterman / Needleman-Wunsch with Kimura2 / Jukes-Cantor / Percent-Identity MPI.NET Implementation Chi-Square / Deterministic Annealing C# Desktop Application based on VTK * * Note. The implementations of Smith-Waterman and Needleman-Wunsch algorithms are from Microsoft Biology Foundation library

19. SALSASALSA Twister MDS Interpolation Performance Test

20. SALSASALSASALSASALSA University of Arkansas Indiana University University of California at Los Angeles Penn State Iowa Univ.Illinois at Chicago University of Minnesota Michigan State Notre Dame University of Texas at El Paso IBM Almaden Research Center Washington University San Diego Supercomputer Center University of Florida Johns Hopkins July 26-30, 2010 NCSA Summer School Workshop http://salsahpc.indiana.edu/tutorial 300+ Students learning about Twister & Hadoop MapReduce technologies, supported by FutureGrid.

22. SALSASALSA 22 http://salsahpc.indiana.edu/b534/

23. SALSASALSA 23

24.  MapReduce and MPI are SPMD programming model  Twister extends the MapReduce to iterative algorithms  Dataming in the Cloud (Data Analysis in the Cloud)  Several iterative algorithms we have implemented  K-Means Clustering  Pagerank  Matrix Multiplication  Breadth First Search  Multi Dimensional Scaling (MDS)  Integrating a distributed file system  Integrating with a high performance messaging system  Programming with side effects yet support fault tolerance Summary

25. SALSASALSA MapReduceRoles4Azure Will have prototype Twister4Azure by May 2011 Several iterative algorithms we have implemented

26. SALSASALSA 26 Twister for Azure Map 1 Map 2 Map n Map Workers Red 1 Red 2 Red n Reduce Workers In Memory Data Cache Task Monitoring Role Monitoring Worker Role MapID…….Status Map Task Table MapID…….Status Job Bulleting Board Scheduling Queue

27. SALSASALSA Sequence Assembly Performance

28. SALSASALSA Acknowledgements to: SALSA HPC Group Indiana University http://salsahpc.indiana.edu