Parallel Distributed Programming Introduction

Course Plan (1)  Introduction to parallel problem solving programming models standard APIs  A model of parallel/distributed computation state transition system  Proving correctness of parallel/distributed algorithms Proof by hand Proof by tools (model checker)

Course Plan (2)  Calculating global state of distributed computation consistent snapshot, checkpointing  Shared memory consistency models sequential/relaxed consistency emulating shared memory  Fault tolerance impossibility of consensus in asynchronous message passing

Parallel Programming Models  Message passing, shared memory, multithreading, object-oriented, etc. Different in what are done by the system and what are left to programmers Communication, task mapping, memory management, etc.  Popular libraries MPI, Pthreads, OpenMP, Java, PVM

Very Very Brief Introductions to Important Concepts

Model of Computation  To rigorously discuss correctness/incorrectness of a program, a formal model is necessary  A formal model is a state transition graph A program describes all possible state transitions of an individual process The entire system is a composition of many such processes and thus modeled by a large state transition graph

; ; -----; Compose N processes + communication channels State transition of a process State transition of the system local state global state

Proving the Correctness  Proving the correctness of a program boils down to proving some desired properties of the state transition graph derived from the program  E.g., Do all state satisfy X + Y  1? Does the program always terminate?

Model Checking Tools  State transition graph is usually large, but it is in theory possible to mechanize proofs of many properties as long as the graph is finite  Model checkers are tools based on this idea, with many ideas to deal with large graphs

Calculating Global State  Capturing global state of a distributed computation is not easy  Many algorithms need calculate/detect a property of the global state termination deadlock etc.  There is a general algorithm to calculate global state of a computation  Moreover, it is a general method for fault tolerance by rollback

Sequential Consistency  Sequential consistency (SC) is a formal definition of what one usually calls “shared memory”  Again, a formal model is required to reason about the correctness of programs  Moreover, it is required to discuss whether a protocol implementing shared memory actually guarantees SC

Relaxed Consistency  SC turns out costly to implement many communication optimizations break SC  Modern multiprocessors thus do not guarantee SC, but instead some of relaxed consistency models

Software Shared Memory  A memory is said “shared” as long as it behaves according to a shared memory consistency model (SC or not)  It is therefore possible to implement shared memory in software

Fault Tolerance  There are many types of faults (crash, Byzantine, etc.), which are modeled differently  In pure asynchronous models, many simple problems turn out impossible

 1/15 ふじた はやつ  10/18 加藤 ( 数理 ), …  11/29 吉本 ( 電子情報 ) ，吉田  12/6 初田, 豊島  10/25 今竹，堀田  11/22 林