1. 平成30年度 後期 月曜 第3時限（13：00－14：30） 吉永 努（ＵＥＣ) yoshinaga@uec.ac.jp
計算機ネットワーク特論
平成30年度　後期
月曜　第3時限（13：00－14：30）
吉永　努（ＵＥＣ)
IN 30

2. 内 容 分散・並列処理計算機における相互結合ネットワークとその上でのメッセージ・ルーティング技法などについて学ぶ
資料　
appendix_f.pdf (P.118, 2MB)
TA:
IN 30

3. References
T. M. Pinkston and J. Duato: Interconnection Networks, Appendix E in Computer Architecture: A Quantitative Approach, 4th Edition, Morgan Kaufmann publishers (2006).
5th Edition, Morgan Kaufmann publishers (2011).
J. Duato, S. Yalamanchili, L. Ni: Interconnection Networks - an Engineering Approach-, 第2版, Morgan Kaufmann publishers (2003)
富田眞治：　並列コンピュータ、昭晃堂（1996）
W.D. Dally, B. Towles: Principles and Practices of Interconnection Networks, Morgan Kaufmann publishers (2003)
IN 30

4. What is an interconnection Network?
It is a programmable system that transports data between terminals, such as processors and memory.
It is programmable in the sense that it makes different connections at different points.
It is a system because it is composed of many components: buffers, channels, switches, and controls that works together to deliver data.
IN 30

5. Interconnection Network (1/2)P P P M M M
Multicomputer
IN 30

6. Interconnection Network (2/2)P P P
Interconnection Network M M M
UMA type shared memory multiprocessor
It is also called dance-hall architecture.
IN 30

7. Trend
Its performance is increasing with processor performance at a rate of 50% per year.
Communication is a limiting factor in the performance of many modern systems.
Buses have been unable to keep up with the bandwidth demand, and point-to-point interconnection networks are rapidly taking over.
IN 30

8. HPC Interconnect share (%)
2018/06
2017/06
2016/06
10G Ethernet
34.2
39.0
35.4
Infiniband FDR†
14.4
23.4
28.0
Aries interconnect ††
9.2
8.6
7.6
† Fourteen Data Rate ( 14Gbps/link )
††Cray Inc.
IN 30

9. Examples of HPC cluster
Processors
Accelerator
Interconnect
Summit
IBM Power
ORNL
2018
IBM Power 9
(20 cores)
3 GHz×2
×4608 nodes
NVIDIA Tesla V100
×4 / node
Mellanox EDR
Infiniband
Fat tree
Tsubame 3.0
Tokyo Tech.
2017
Xeon E V4
(14 cores)
2.4GHz×2
×540 nodes
NVIDIA Tesla P100
Intel
Omni-Path (400Gbps) ×2
IN 30

10. 5-level hierarchical Sunway interconnect
Examples of MPPs
Node
Topology
#core
Rmax
K computer
@RIKEN
Fujitsu
2011
SPARC64 VIIIfx
2 GHz
(16 GFlops×
8-core)
6D mesh/
3D torus
Tofu interconnect
80K-node x 8-core
= 640K-core
10.51 PFlops
7,890 KW
Sunway TaihuLight
2016
SW26010
1.45 GHz
(11.6 GFlops×
260-core)
5-level hierarchical Sunway　interconnect
40,960 nodes
125.4 PFlops
15,371 KW
IN 30

11. Other Networks of Supercomputers
Cray X50 (2016): dragonfly, Aries interconnect
Titan/Cray XK7 (2012): 3D torus, proprietary Gemini interconnect
Sequoia/BG Q (2011): 5D torus, proprietary IBM SeaStar
Pleiades / NASA (2011): partial 11D hypercube topology with IB QDR/DDR
Fujitsu PRIMERGY CX1640 (2016):
Fat-Tree, Intel Omni-Path
Earth Simulator2 / NEC SX-9E (2009):
Fat-Tree (64GB/s/cpu, 8-CPU/node, 160 nodes)
IN 30

12. Architecture vs. software
memory
programming
UMA
(SMP)
shared
OpenMP
NUMA
(MPP)
distributed
(not shared)
MPI
(Message Passing Interface)
IN 30

13. Network Design (1/3) Performance: latency and throughput (bandwidth)
Scalability: #processors vs. network, memory, I/O bandwidth
Incremental expandability:
small to maximum size
Partitionability: netwrok may be partitioned for several users
IN 30

14. Network Design (2/3)
Simplicity: simple design, higher clock frequency, easy to use
Distance span: smaller system is preferred for noise and cable delay, etc.
Physical constraints: packaging (pin count), wiring(wire length), and maintenance (power consumption) should meet physical limitation.
IN 30

15. Network Design (3/3)
Reliability: fault tolerant, reliable communication, hot swap
Expected workload: robust performance over a wade range of traffic conditions.
Cost: trade-offs between cost and performance.
IN 30

16. Classifiction of Interconnection Networks
Shared-Medium Networks
Local area networks (ethernet, token ring)
Backplane bus (e.g. SUN Gigaplane)
Direct Networks (router-based)
mesh, torus, hypercube, tree, … etc.
Indirect Networks (switch-based)
Hybrid Networks
IN 30

17. Shared-Medium Networks (LAN)
Arbitration that determines the mastership of the shared-medium network to resolve network access is needed.
The most well-known protocol is carrier-sense multiple access with collision detection (CSMA/CD).
Token bus and token ring pass a token from the owner which has the right to access the bus/ring and resolve nondeterministic waiting time.
IN 30

18. Shared-Medium Networks (Backplane bus)
It is commonly used to interconnect processor(s) and memory modules to provide SMP (Symmetrical Memory Processor) architecture.
It is realized by printed lines on a circuit board by discrete wiring.
Gigaplane in SUN Enterprise x000 server(1996): 2.6GB/s, 256 bits data, 42 bits address, 83.8MHz clock.
IN 30

19. Direct (static) Networks
Consists of a set of nodes.
Each node is directly connected to a subset of other nodes in the network.
Examples:
2D mesh (intel Paragon), 3D mesh (MIT J-Mahine)
2D torus (Fujitsu AP3000), 3D torus (Cray T3D, T3E)
Hypercube (CM1, CM2, nCUBE)
IN 30

20. Mesh topology
node 2D 3D
IN 30

21. Torus topology 2D
(4-ary 2-cube) 3D
(3-ary 3-cube)
IN 30

22. Hypercube (binary n-cube)4D
(2-ary 4-cube)
IN 30

23. tree
x tree
Binary tree
fat tree
IN 30

24. Hierarchical topology (1/2)
Pyramid
(Hierarchical 2D mesh)
Hierarchical ring
IN 30

25. Hierarchical topology (2/2)
Cube-connected cycles
RDT
(Recursive Diagonal Torus)
IN 30

26. Hypermesh (spaninng-bus hypercube)
Single or multiple buses
IN 30

27. Base-m n-cube (hyper-crossbar)
770
777
070
077
707
000
007
8x8 crossbar
Base-8 3-cube (Toshiba Prodigy)
IN 30

28. Diameter and degrees (1/2)
2D　mesh
2D torus
3D torus
binary n-cube
#node N Ｎ
Ｎ = 2n
Diameter
２√N √N
log N
degree ４ ６ ３
IN 30

29. Diameter and degrees (2/2)
Base-m n-cube
CCC
Binary tree
ring
#node
N = mn
Ｎ = n2n Ｎ
Diameter
logm N
3n/2
2log N
N/2
degree 3 2 ３
IN 30