Data Center Network Architectures Jellyfish vs. Fat-tree Weiye Hu ELEN6909 Network Algorithms and Dynamics May 13th 2016 1 1 1 1 1 1 1 1 1 1

Data Centers Many applications must exchange information with remote nodes to proceed with their local computation. -- MapReduce -- Applications running on cluster-based file systems -- Web search engine -- Parallel applications The principle bottleneck in large-scale data centers is often inter-node communication bandwidth.

Typical Architecture Today Core Aggregation Edge What do we want? A host can communicate with any other host at high bandwidth (e.g. the full bandwidth of its local network interface).

How to improve inter-node bandwidth? Solutions Use specialized hardware and communication protocols Use high-performance switches -- Very expensive -- The aggregate bandwidth may still be very low Use new network topologies that meet the following goals: -- Scalable interconnection bandwidth -- Economies of scale -- Backward compatibility

Fat-tree & Jellyfish Fat-tree (tree-like topology) Jellyfish (random graph)

Fat-tree

Fat-tree Topology Core Aggregation Edge k-ary tree (here k = 4)

Fat-tree Topology k-ary tree Core (here k = 4) Aggregation Edge k-port switches k pods, each containing two layers of k/2 switches Each switch in the lower layer is connected to k/2 hosts k2/4 core switches Each core switch has one port connected to each of k pods Supports k3/4 hosts (determined by k)

Why fat-tree can deliver high bandwidth? traditional topology fat-tree BW 1 2 4 8 16 BW 16

Fat-tree vs. Traditional Topology Advantages Fat-tree achieves lower cost while delivering high bandwidth. e.g. when supports 27,648 hosts at full bandwidth -- Fat-tree (k = 48): costs $8.64M -- traditional solution: costs $37M Fat-tree requires no changes to end hosts. -- is fully TCP/IP compatible Fat-tree requires only a little modifications to the switches.

Fat-tree vs. Traditional Topology

Incremental Expansion What is incremental expansion? Adding servers and capacity incrementally to the data center. What do we want? The topology allows construction of arbitrary-sized networks. Add new components to the existing topology without replacing or upgrading existing components.

Does fat-tree allow incremental expansion? Does fat-tree allow construction of arbitrary-sized networks? No! The number of hosts a fat-tree can support is determined by k, i.e., the port-count of the switches it uses. e.g., when k = 48, the fat-tree can only be built at size 27,648. (k3/4 = 483/4 = 27,648) Does fat-tree allow adding new components without replacing or upgrading existing components? Even small incremental expansion would require replacing every existing switch to maintain the fat-tree structure.

Structure hinders incremental expansion Fat-tree The entire structure is completely determined by the port-count of the switches available. Adding servers while preserving the structural properties would require replacing a large number of network elements. Structure hinders incremental expansion. Other topologies have similar problems. How to solve this problem? Use random graph!

Jellyfish

a degree-bounded random graph Jellyfish Topology Jellyfish a degree-bounded random graph

Jellyfish Topology Each switch uses some ports to connect to other switches, and uses the remaining ports for end hosts. It doesn't require every switch have the same port-count. If there are N switches in total, and all switches are identical: -- k-port switch -- r ports for other switches -- (k – r) ports for hosts then the network can supports N * (k – r) hosts.

Jellyfish Topology How to construct a jellyfish? Simply pick a random pair of switches with free ports (for the switch-pairs are not already neighbors), join them with a link. Repeat until no further links can be added. How to add a new switch (i.e., incremental expansion)? Randomly remove an existing link (x, y). Add link (p1, x) and (p2, y), where p1 and p2 are two ports on the new switch.

Jellyfish Properties Flexible. Incrementally constructed Jellyfish has the same capacity as Jellyfish built from scratch. Large throughput. -- achieves 91% of the theoretical optimal throughput -- leaves little room for improvement Low mean path length. Highly failure resilient. -- maintains its structure in the face of link or node failures -- a random graph with a few failures is just another random graph of slightly smaller size

Jellyfish vs. Fat-tree

Jellyfish vs. Fat-tree – Throughput Jellyfish can support 27% more servers at full capacity than a fat- tree with the same switching equipment (i.e., with the same cost).

Jellyfish vs. Fat-tree – Path Length With 686 servers, >99.5% of source-destination pairs in Jellyfish can be reached in <6 hops, while the corresponding number is only 7.5% in the fat-tree.

Jellyfish vs. Fat-tree – Failure Resilience Both topologies are highly resilient to failures. But the throughput per server decreases more gracefully for Jellyfish than for a same- equipment fat-tree as the percentage of failed links increases.

Jellyfish vs. Fat-tree – Flexibility Arbitrary-sized Networks: Jellyfish: allows construction of arbitrary-size networks √ Fat-tree: the size is constrained by the port-count of switches × Incremental Expandability: Jellyfish: can be expanded easily, and the desirable properties are maintained: high bandwidth and short paths at low cost √ Fat-tree: structure hinders incremental expansion ×

Routing & Congestion Control Jellyfish Fat-tree Routing k-shortest-paths ECMP (equal cost multi- path routing) Congestion Control MPTCP (multipath TCP )

References [1] Singla A, Hong C Y, Popa L, et al. Jellyfish: Networking data centers randomly[C]//Presented as part of the 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI 12). 2012: 225-238. [2] Al-Fares M, Loukissas A, Vahdat A. A scalable, commodity data center network architecture[J]. ACM SIGCOMM Computer Communication Review, 2008, 38(4): 63-74.

Thanks!