Dilemma of Parallel Programming Xinhua Lin ( 林新华 ) HPC Lab of 17 th Oct 2011

Disclaimers I am not funded by CRAY Slides marked with Chapel logo are taken from Brad Chamberlain’s talk ‘The Mother of All Chapel Talks’, with permission from himself Funny pictures are from Internet

About me and HPC Lab in SJTU Directing HPC Lab Co-translator of PPP Co-founder of HMPP CoC for AP&Japan As MS HPC Invitation Support For HPC Center of SJTU Hold SJTU HPC Seminar monthly

Three Challenges for ParaProg in multi/many core era Revolution V.S. Evolution Low level V.S. High level – Performance V.S. Programmable Performance V.S. Performance Portability For more detail: Paper Version: Special issue for HPC and Cloud, Sep 2011 Online Version:

Outline Right Level to expose Parallel ParaProg languages Reviews Multiresolution and Chapel

Right Level to Expose Parallel

Can we stop water/parallel ? Hardware ISA OS Library Language

Performance V.S. Programmable Target Machine MPI OpenMP pthreads Expose Implementing Mechanisms “Why is everything so tedious?” Target Machine ZPL HPF Higher-Level Abstractions “Why don’t I have more control?” Low Level High Level

ParaProg Education Tired of teaching yet another specific lang. – MPI for Cluster – OpenMP for SMP then Multi-core CPU – CUDA for GPU, and now OpenCL – More on the way… Had to explain concepts by different tools – Single lang. to explain them all? Similar in OS education – Production OS: Linux, Unix and Window – OS only for education: Minix

ParaProg languages Reviews

Hybrid Programming Model MPI is insufficient in multi/many core era – OpenMP for multi-core – CUDA/OpenCL for many-core* So called Hybrid Programming was invented as a temporary solution, workable but ugly – MPI+OpenMP for Multi-core cluster – MPI+CUDA/OpenCL for GPU cluster like Tianhe-1A Similar idea used in CUDA for thread and thread-block, OpenCL for work-item and work- group * We will wait and see how OpenMP works on Intel MIC

ParaProg from different ways Low Level (expose implementation mechanism ) – MPI, CUDA and OpenCL – OpenMP High Level – PGAS: CAF, UPC and Tianuim – Global View: NESL, ZPL – APGAS: Chapel, X10 Directive Based – HMPP, PGI, CRAY-directive

Mulutiesolution and Chapel

What is Mulutiesolution? Structure the language in a layered manner, permitting it to be used at multiple levels as required/desired – support high-level features and automation for convenience – provide the ability to drop down to lower, more manual levels – use appropriate separation of concerns to keep these layers clean Distributions Data parallelism Task Parallelism Locality Control Target Machine Base Language language concepts

Where Chapel was born: HPCS HPCS: High Productivity Computing Systems (DARPA et al.) – Goal: Raise productivity of high-end computing users by 10  – Productivity = Performance + Programmability + Portability + Robustness Phase II: Cray, IBM, Sun (July 2003 – June 2006) – Evaluated the entire system architecture’s impact on productivity… processors, memory, network, I/O, OS, runtime, compilers, tools, … …and new languages: Cray: Chapel IBM: X10 Sun: Fortress Phase III: Cray, IBM (July 2006 – 2010) – Implement the systems and technologies resulting from phase II – (Sun also continues work on Fortress, without HPCS funding)

Global-view V.S. Fragmented Problem: “Apply 3-pt stencil to vector” global-view = + ( )/2)/2 fragmented = + = + = )/2+ ( ((

Global-view V.S. SPMD Code Global-View def main() { var n: int = 1000; var a, b: [1..n] real; forall i in 2..n-1 { b(i) = (a(i-1) + a(i+1))/2; } SPMD def main() { var n: int = 1000; var locN: int = n/numProcs; var a, b: [0..locN+1] real; if (iHaveRightNeighbor) { send(right, a(locN)); recv(right, a(locN+1)); } if (iHaveLeftNeighbor) { send(left, a(1)); recv(left, a(0)); } forall i in 1..locN { b(i) = (a(i-1) + a(i+1))/2; }

Chapel Overview A design principle for HPC – “Support the general case, optimize for the common case” Data Parallel (ZPL) + Task Parallel(CRAY MTA) + Script Lang. Latest version is available in as OSS: Distributions Data parallelism Task Parallelism Locality Control Target Machine Base Language language concepts

Chapel example: Heat Transfer A: 1.0 n n   4 repeat until max change < 

Chapel Code For Heat Transfer

Chapel as Minix in ParaProg If I were to offer a ParaProg class, I’d want to teach about: – data parallelism – task parallelism – concurrency – synchronization – locality/affinity – deadlock, livelock, and other pitfalls – performance tuning – …

Conclusion—Major Points Programmable and Performance are always the dilemma of ParaProg Multiresolution sounds perfect in theory but not mature enough for production However, Chapel could be used as Minix in ParaProg

Q&A