1. 平成29年度 後期 月曜 第3時限（13：00－14：30） 吉永 努（ＵＥＣ) yosinaga@is.uec.ac.jp
計算機ネットワーク特論
平成29年度　後期
月曜　第3時限（13：00－14：30）
吉永　努（ＵＥＣ)
IN 29

2. 内 容 分散・並列処理計算機における相互結合ネットワークとその上でのメッセージ・ルーティング技法などについて学ぶ
資料　
appendix_f.pdf (P.118, 2MB)
TA:
IN 29

3. References
T. M. Pinkston and J. Duato: Interconnection Networks, Appendix E in Computer Architecture: A Quantitative Approach, 4th Edition, Morgan Kaufmann publishers (2006).
5th Edition, Morgan Kaufmann publishers (2011).
J. Duato, S. Yalamanchili, L. Ni: Interconnection Networks - an Engineering Approach-, 第2版, Morgan Kaufmann publishers (2003)
富田眞治：　並列コンピュータ、昭晃堂（1996）
W.D. Dally, B. Towles: Principles and Practices of Interconnection Networks, Morgan Kaufmann publishers (2003)
IN 29

4. What is an interconnection Network?
It is a programmable system that transports data between terminals, such as processors and memory.
It is programmable in the sense that it makes different connections at different points.
It is a system because it is composed of many components: buffers, channels, switches, and controls that works together to deliver data.
IN 29

5. Interconnection Network (1/2)P P P M M M
Multicomputer
IN 29

6. Interconnection Network (2/2)P P P
Interconnection Network M M M
UMA type shared memory multiprocessor
It is also called dance-hall architecture.
IN 29

7. Trend
Its performance is increasing with processor performance at a rate of 50% per year.
Communication is a limiting factor in the performance of many modern systems.
Buses have been unable to keep up with the bandwidth demand, and point-to-point interconnection networks are rapidly taking over.
IN 29

8. Computer Classifications (%)
2017/06
2016/06
2015/06
MPP
13.6
13.8
13.2
Cluster
86.4
86.2
86.8
Others
0.0
share of the TOP500 June, 2017 – June, 2015
IN 29

9. Infiniband QDR (40Gbps) ×2
Examples of clusters
Processors
Accelerator
Interconnect
Tianhe-2
(天河2号）China
2013
Intel Xeon
E C
2.2 GHz×2 ×16K
Xeon Phi 31S1P (57 cores)×3
×16K
TH Express-2
(proprietary)
Fat tree
Tsubame 2.5
Tokyo Tech.
Xeon X5670
2.93GHz×2
×1,408
NVIDIA Kepler K20x
×3×1,048
Infiniband QDR (40Gbps) ×2
IN 29

10. 5-level hierarchical Sunway interconnect
Examples of MPPs
Node
Topology
#core
Rmax
K computer
@RIKEN
Fujitsu
2011
SPARC64 VIIIfx
2 GHz
(16 GFlops×
8-core)
6D mesh/
3D torus
Tofu interconnect
80K-node x 8-core
= 640K-core
10.51 PFlops
7,890 KW
Sunway TaihuLight
2016
SW26010
1.45 GHz
(11.6 GFlops×
260-core)
5-level hierarchical Sunway　interconnect
40,960 nodes
125.4 PFlops
15,371 KW
IN 29

11. Other Networks of Supercomputers
Cray X50 (2016): dragonfly, Aries interconnect
Titan/Cray XK7 (2012): 3D torus, proprietary Gemini interconnect
Sequoia/BG Q (2011): 5D torus, proprietary IBM SeaStar
Pleiades / NASA (2011): partial 11D hypercube topology with IB QDR/DDR
Fujitsu PRIMERGY CX1640 (2016):
Fat-Tree, Intel Omni-Path
Earth Simulator2 / NEC SX-9E (2009):
Fat-Tree (64GB/s/cpu, 8-CPU/node, 160 nodes)
IN 29

12. Architecture vs. software
memory
programming
UMA
(SMP)
shared
OpenMP
NUMA
(MPP)
distributed
(not shared)
MPI
(Message Passing Interface)
IN 29

13. Network Design (1/3) Performance: latency and throughput (bandwidth)
Scalability: #processors vs. network, memory, I/O bandwidth
Incremental expandability:
small to maximum size
Partitionability: netwrok may be partitioned for several users
IN 29

14. Network Design (2/3)
Simplicity: simple design, higher clock frequency, easy to use
Distance span: smaller system is preferred for noise and cable delay, etc.
Physical constraints: packaging (pin count), wiring(wire length), and maintenance (power consumption) should meet physical limitation.
IN 29

15. Network Design (3/3)
Reliability: fault tolerant, reliable communication, hot swap
Expected workload: robust performance over a wade range of traffic conditions.
Cost: trade-offs between cost and performance.
IN 29

16. Classifiction of Interconnection Networks
Shared-Medium Networks
Local area networks (ethernet, token ring)
Backplane bus (e.g. SUN Gigaplane)
Direct Networks (router-based)
mesh, torus, hypercube, tree, … etc.
Indirect Networks (switch-based)
Hybrid Networks
IN 29

17. Shared-Medium Networks (LAN)
Arbitration that determines the mastership of the shared-medium network to resolve network access is needed.
The most well-known protocol is carrier-sense multiple access with collision detection (CSMA/CD).
Token bus and token ring pass a token from the owner which has the right to access the bus/ring and resolve nondeterministic waiting time.
IN 29

18. Shared-Medium Networks (Backplane bus)
It is commonly used to interconnect processor(s) and memory modules to provide SMP (Symmetrical Memory Processor) architecture.
It is realized by printed lines on a circuit board by discrete wiring.
Gigaplane in SUN Enterprise x000 server(1996): 2.6GB/s, 256 bits data, 42 bits address, 83.8MHz clock.
IN 29

19. Direct (static) Networks
Consists of a set of nodes.
Each node is directly connected to a subset of other nodes in the network.
Examples:
2D mesh (intel Paragon), 3D mesh (MIT J-Mahine)
2D torus (Fujitsu AP3000), 3D torus (Cray T3D, T3E)
Hypercube (CM1, CM2, nCUBE)
IN 29

20. Mesh topology
node 2D 3D
IN 29

21. Torus topology 2D
(4-ary 2-cube) 3D
(3-ary 3-cube)
IN 29

22. Hypercube (binary n-cube)4D
(2-ary 4-cube)
IN 29

23. tree
x tree
Binary tree
fat tree
IN 29

24. Hierarchical topology (1/2)
Pyramid
(Hierarchical 2D mesh)
Hierarchical ring
IN 29

25. Hierarchical topology (2/2)
Cube-connected cycles
RDT
(Recursive Diagonal Torus)
IN 29

26. Hypermesh (spaninng-bus hypercube)
Single or multiple buses
IN 29

27. Base-m n-cube (hyper-crossbar)
770
777
070
077
707
000
007
8x8 crossbar
Base-8 3-cube (Toshiba Prodigy)
IN 29

28. Diameter and degrees (1/2)
2D　mesh
2D torus
3D torus
binary n-cube
#node N Ｎ
Ｎ = 2n
Diameter
２√N √N
log N
degree ４ ６ ３
IN 29

29. Diameter and degrees (2/2)
Base-m n-cube
CCC
Binary tree
ring
#node
N = mn
Ｎ = n2n Ｎ
Diameter
logm N
3n/2
2log N
N/2
degree 3 2 ３
IN 29