Combining the strengths of UMIST and The Victoria University of Manchester Matthew Livesey, Hemanth John Jose and Yongping Men COMP60611 – Patterns of parallel programming Task parallelism Turning a serial program into a parallel one by understanding dependencies between the tasks in the program Traditionally tasks were mapped statically, but dynamic mapping algorithms may improve performance. It may be possible to re-organise a serial program to reduce dependencies, but need to be careful that the re-organisation does not impact the performance of the algorithm Scheduling algorithms which focus on the critical path perform best. 01 a = b + c; 02 d = e + f ; 03 z = a + d ;

Combining the strengths of UMIST and The Victoria University of Manchester Matthew Livesey, Hemanth John Jose and Yongping Men COMP60611 – Patterns of parallel programming Pipeline parallelism A stream of data items is processed one after another by a sequence of threads. Figure 4 from Pautasso and Alonso 2006 [9]

Combining the strengths of UMIST and The Victoria University of Manchester Matthew Livesey, Hemanth John Jose and Yongping Men COMP60611 – Patterns of parallel programming Data parallelism General purpose programming on Graphics processors (GPGPU). Exploiting the parallel power of graphics chips to execute data parallel algorithms. E.g. OpenCL Allows parallel programs to run on whatever (supported) hardware is available E.g. Multi-core CPU and GPU Provides bindings in many popular languages Cross platform, open source Map Reduce Based on a paper from Google. Designed for processing large data sets in parallel Data is processed in parallel by “mappers”, and the results recombined with “reducers” Distributed parallelism across heterogeneous, commodity hardware Requires the problem at hand to fit the map reduce pattern Combining patterns It is possible to combine patterns, for example a task graph approach may guide the order in which pipeline steps are applied. However data and task parallelism are two different ways of looking at a parallel problem. They may be applied heirarchically or sequentially but represent two distinct approaches

Combining the strengths of UMIST and The Victoria University of Manchester Matthew Livesey, Hemanth John Jose and Yongping Men COMP60611 – Patterns of parallel programming Presentation and report available at: References: 1. Michael D McCool (Intel). Structured Parallel Programming with Deterministic Patterns. To appear in HotPar ’10, 2 nd USENIX Workshop on Hot Topics in Parallelism, June 2010, Berkeley CA. Accessed from patterns/ patterns/ 2. Jeffrey Dean and Sanjay Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. OSDI'04: Sixth Symposium on Operating System Design and Implementation, San Francisco, CA, December, The Apache Software Foundation. Hadoop Map Reduce Tutorial [Accessed 15 th October 2010] Available from 4. David Tarditi et al. Accelerator: Using Data Parallelism to Program GPUs for General-Purpose Uses Microsoft Research Nickolls et al. Scalable Parallel Programming. ACM Queue Volume 6, Issue 2 (March/April 2008) Pages: Aaftab Munshi, OpenCL, Parallel Computing on the GPU and CPU, SIGGRAPH 2008: Beyond Programmable Shading (presentation) 7. A. Grama et al. Introduction to Parallel Computing. 2 nd Ed. Addison-Wesley, Thomas Rauber and Gudula R¨unger l. Parallel Programming For Multicore and Cluster systems, 1st edition, Spriner, Pages 280, Pautasso, C. and Alonso, G. Parallel Computing Patterns for Grid Workflows, Workflows in Support of Large-Scale Science, WORKS '06 Workshop on, June C. Charles Law,William J. Schroeder, Kenneth M. Martin and Joshua Temkin. A multi-threaded streaming pipeline architecture for large structured data sets, VIS '99: Proceedings of the conference on Visualization '99: celebrating ten years,page , R. Cytron, M. Hindy and W. Hsiehz. Automatic Generation of DAG Parallelism. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation, June 1989, SIGPLAN Notices, Volume 24, Number Y. Kwok and I. Ahmad. Benchmarking and Comparison of the Task Graph Scheduling Algorithms. Journal of Parallel and Distributed Computing, 59, pages (1999)