1. 平成25年度 後期 火曜 第２時限（１０：４０－１２：１０） 吉永 努（ＵＥＣ) yosinaga@is.uec.ac.jp
ネットワークコンピューティング論Ⅱ
平成25年度　後期
火曜　第２時限（１０：４０－１２：１０）
吉永　努（ＵＥＣ)
ＮＣ論２

2. 内 容 分散・並列処理計算機における相互結合ネットワークとその上でのメッセージ・ルーティング技法などについて学ぶ
資料　
(253 slides, 13MB)
(P.118, 2MB)
TA: 重信
ＮＣ論２

3. References
T. M. Pinkston and J. Duato: Interconnection Networks, Appendix E in Computer Architecture: A Quantitative Approach, 4th Edition, Morgan Kaufmann publishers (2006).
5th Edition, Morgan Kaufmann publishers (2011).
J. Duato, S. Yalamanchili, L. Ni: Interconnection Networks - an Engineering Approach-, 第2版, Morgan Kaufmann publishers (2003)
富田眞治：　並列コンピュータ、昭晃堂（1996）
W.D. Dally, B. Towles: Principles and Practices of Interconnection Networks, Morgan Kaufmann publishers (2003)

4. What is an interconnection Network?
It is a programmable system that transports data between terminals, such as processors and memory.
It is programmable in the sense that it makes different connections at different points.
It is a system because it is composed of many components: buffers, channels, switches, and controls that works together to deliver data.
ＮＣ論２

5. Interconnection Network (1/2)P P P M M M
Multicomputer
ＮＣ論２

6. Interconnection Network (2/2)P P P
Interconnection Network M M M
UMA type shared memory multiprocessor
It is also called dance-hall architecture.
ＮＣ論２

7. Trend
Its performance is increasing with processor performance at a rate of 50% per year.
Communication is a limiting factor in the performance of many modern systems.
Buses have been unable to keep up with the bandwidth demand, and point-to-point interconnection networks are rapidly taking over.
ＮＣ論２

8. Computer Classifications (%)
2013/06
2012/06
2011/06
MPP
16.6
18.6
17.4
Cluster
83.4
81.4
82.2
Others
0.0
0.4
share of the TOP500 June, 2013 – June, 2011
ＮＣ論２

9. Infiniband QDR (40Gbps) ×2
Examples of clusters
Processors
Accelerator
Interconnect
Tianhe-2
(天河2号）China
2013
Intel Xeon
E C
2.2 GHz×2 ×16K
Xeon Phi 31S1P (57 cores)×3
×16K
TH Express-2
(proprietary)
Fat tree
Tsubame 2.5
Tokyo Tech.
Xeon X5670
2.93GHz×2
×1,408
NVIDIA Kepler K20x
×3×1,048
Infiniband QDR (40Gbps) ×2
ＮＣ論２

10. Examples of MPPs #core Rmax Node Topology K computer @RIKEN Fujitsu
2011
SPARC64 VIIIfx
2 GHz
(16 GFlops×
8 cores)
6D mesh/
3D torus
Tofu interconnect
80K-node x 8-core
= 640K-core
10.51 PFlops
7,890 KW
Cray XK7
2012
AMD Opteron 16C 2.2 GHz
+ NVIDIA K20x
Gemini interconnect
18,688 nodes
(200 Cabinets)
27.11 PFlops
8,209 KW
ＮＣ論２

11. Other Networks of Supercomputers
Sequoia (2011): 5D torus, proprietary IBM SeaStar
Pleiades / NASA (2011): partial 11D hypercube topology with IB QDR/DDR
Red Sky/ Sandia National Lab. (2010):
3D torus (12 bristled node) with IB QDR switches
IBM Roadrunner (2009): fat-tree with IB DDR
Earth Simulator2 / NEC SX-9E (2009):
Fat-Tree (64GB/s/cpu, 8-CPU/node, 160 nodes)
IBM Blue Gene/L (2004): 3D torus proprietary
(64 x 32 x 32 = 64K nodes)
ＮＣ論２

12. Architecture vs. software
memory
programming
UMA
(SMP)
shared
OpenMP
NUMA
(MPP)
distributed
(not shared)
MPI
(Message Passing Interface)
ＮＣ論２

13. Network Design (1/3) Performance: latency and throughput (bandwidth)
Scalability: #processors vs. network, memory, I/O bandwidth
Incremental expandability:
small to maximum size
Partitionability: netwrok may be partitioned for several users
ＮＣ論２

14. Network Design (2/3)
Simplicity: simple design, higher clock frequency, easy to use
Distance span: smaller system is preferred for noise and cable delay, etc.
Physical constraints: packaging (pin count), wiring(wire length), and maintenance (power consumption) should meet physical limitation.
ＮＣ論２

15. Network Design (3/3)
Reliability: fault tolerant, reliable communication, hot swap
Expected workload: robust performance over a wade range of traffic conditions.
Cost: trade-offs between cost and performance.
ＮＣ論２

16. Classifiction of Interconnection Networks
Shared-Medium Networks
Local area networks (ethernet, token ring)
Backplane bus (e.g. SUN Gigaplane)
Direct Networks (router-based)
mesh, torus, hypercube, tree, … etc.
Indirect Networks (switch-based)
Hybrid Networks
ＮＣ論２

17. Shared-Medium Networks (LAN)
Arbitration that determines the mastership of the shared-medium network to resolve network access is needed.
The most well-known protocol is carrier-sense multiple access with collision detection (CSMA/CD).
Token bus and token ring pass a token from the owner which has the right to access the bus/ring and resolve nondeterministic waiting time.
ＮＣ論２

18. Shared-Medium Networks (Backplane bus)
It is commonly used to interconnect processor(s) and memory modules to provide SMP (Symmetrical Memory Processor) architecture.
It is realized by printed lines on a circuit board by discrete wiring.
Gigaplane in SUN Enterprise x000 server(1996): 2.6GB/s, 256 bits data, 42 bits address, 83.8MHz clock.
ＮＣ論２

19. Direct (static) Networks
Consists of a set of nodes.
Each node is directly connected to a subset of other nodes in the network.
Examples:
2D mesh (intel Paragon), 3D mesh (MIT J-Mahine)
2D torus (Fujitsu AP3000), 3D torus (Cray T3D, T3E)
Hypercube (CM1, CM2, nCUBE)
ＮＣ論２

20. Mesh topology
node 2D 3D
ＮＣ論２

21. Torus topology 2D
(4-ary 2-cube) 3D
(3-ary 3-cube)
ＮＣ論２

22. Hypercube (binary n-cube)4D
(2-ary 4-cube)
ＮＣ論２

23. tree
x tree
Binary tree
fat tree
ＮＣ論２

24. Hierarchical topology (1/2)
Pyramid
(Hierarchical 2D mesh)
Hierarchical ring
ＮＣ論２

25. Hierarchical topology (2/2)
Cube-connected cycles
RDT
(Recursive Diagonal Torus)
ＮＣ論２

26. Hypermesh (spaninng-bus hypercube)
Single or multiple buses
ＮＣ論２

27. Base-m n-cube (hyper-crossbar)
770
777
070
077
707
000
007
8x8 crossbar
Base-8 3-cube (Toshiba Prodigy)
ＮＣ論２

28. Diameter and degrees (1/2)
2D　mesh
2D torus
3D torus
binary n-cube
#node N Ｎ
Ｎ = 2n
Diameter
２√N √N
log N
degree ４ ６ ３
ＮＣ論２

29. Diameter and degrees (2/2)
Base-m n-cube
CCC
Binary tree
ring
#node
N = mn
Ｎ = n2n Ｎ
Diameter
logm N
3n/2
2log N
N/2
degree 3 2 ３
ＮＣ論２