Example: Sorting on Distributed Computing Environment Apr 20,

About this presentation Example for starting preparation of the "Enshu" part of this class Give a talk about the problem attached to you according to the theme of the day. Does not show complete presentation. Just shows the points to be studied, explained and solved. 2

Sorting Large Number of Data Data size > Memory size of single computer Ex. 1~100trillion integer numbers Distributed Parallel Sort: Distribute the data into multiple computers on a network and sort. ⇒ Use multiple computational power ⇒ Requires communication among computers 3

Computational Infrastructures Case 1: PCs in a computer room Use all of the PCs on holidays or in midnights ~100 PCs (200~400GB of memory in total) Case 2: Supercomputers in Japan Enable "Ultra Large Scale Computation" by using supercomputers all over Japan 10~20 supercomputers Speed: 10TFLOPS ~ 100TFLOPS / each Memory: 10TB ~ 100TB / each 4

Network Infrastructures Case 1: Ethernet Switch Bandwidth: 100Mbps ~ 1Gbps Latency: 0.05~0.1msec Case 2: SINET3 (Academic Network in Japan) Backbone Bandwidth: 10~40Gbps Bandwidth per computer: ~10Gbps Latency: 10~100msec depends on the length of physical networks 5

Bandwidth? Latency? Bandwidth: Available speed of data transfer (bit/sec) on the network Latency: Minimum time required for each data transfer Estimation of the cost for a data transfer: T = L + S / B L: Latency, B: Bandwidth, S: Data size (bit) 6

System Infrastructure Case 1: Network environment can be "Reliable", since no other user is using the system. Implementation of the program will be easier by installing MPI(Message Passing Interface). Case 2: Network environment may be "Unreliable" since many users share the network routes. Usage of MPI is difficult, since the environment is "Heterogeneous" Different architectures and OSs 7

Implementation on Internet Everything can be built on "Application Layer" Choose a protocol for internet: TCP or UDP Case 1: UDP (or MPI over UDP) Case 2: TCP Choose a parallel algorithm of sorting Parallel Algorithm: algorithm for solving a problem by dividing it into multiple tasks and running them concurrently 8

"Layers" of networks OSI model: divides facilities of network devices into 7 layers Application Layer Presentation Layer Session Layer Transport Layer Network Layer Data Link Layer Physical Layer 9

TCP or UDP TCP (Transmission Control Protocol) Guarantees the completion of data transfer. Slow but reliable. UDP(User Datagram Protocol) No guarantee about data transfer. Fast but unreliable. Sorting requires every data to be correctly transferred. ⇒ TCP is preferred. On reliable networks such as Case 1, UDP can be used. MPI is an interface over UDP (or TCP). Guarantees data transfer even over UDP. 10

Detailed Implementation of Softing Program Implementation of parallel algorithm: Cost of computation? Cost of communication? Requirements of Memory? Policies for distributing computation and data affects the performance. 11

Characteristics of each case Case 1: Low latency and narrow bandwidth Total amount of computational power and memory is small No need for load-balancing Case 2: High latency and wide bandwidth (Possibly) Large amount of computational power and memory Requires load-balancing according to the computational power of each machine. 12

Implementation for Case 1. Distribute same amount of computation and data on each computer Consider the number of PCs to be used: Communication cost increases according to the number of PCs If the target data is large enough, it will achieve sufficient speedup by parallelization even with 100PCs. 13

Implementation for Case 2 The amount of computation and data depends on the relative performance of each computer. Accurate analysis of the performance of each machine and network is important. It must be difficult to obtain sufficient effect of parallelization with large number of nodes. Performance degradation by load unbalance and communication cost. 14

To complete the presentation of your solution Detailed information about the infrastructure. Detailed information of implementation: Parallel Algorithm? Policy for distributing data and computation? Estimate computation and communication time, and find the optimal distribution. How to distribute the target data? and how to gather the results? Management of multiple jobs. Standardization of the solution Relationship with the future networks 15

Exercise Find existing parallel algorithms For example: Sorting Note: Algorithms for "distributed memory parallel" computing environment = Each computer has its own memory => Requires explicit communication Consider how to implement them on computers connected with Internet. 16