1. Data Center Network Architectures
Jellyfish vs. Fat-tree
Weiye Hu
ELEN6909 Network Algorithms and Dynamics
May 13th 2016 1 1 1 1 1 1 1 1 1 1

2. Data Centers
Many applications must exchange information with remote nodes to proceed with their local computation.
-- MapReduce
-- Applications running on cluster-based file systems
-- Web search engine
-- Parallel applications
The principle bottleneck in large-scale data centers is often inter-node communication bandwidth.

3. Typical Architecture Today
Core
Aggregation
Edge
What do we want?
A host can communicate with any other host at high bandwidth (e.g. the full bandwidth of its local network interface).

4. How to improve inter-node bandwidth?
Solutions
Use specialized hardware and communication protocols
Use high-performance switches
-- Very expensive
-- The aggregate bandwidth may still be very low
Use new network topologies that meet the following goals:
-- Scalable interconnection bandwidth
-- Economies of scale
-- Backward compatibility

5. Fat-tree & Jellyfish Fat-tree (tree-like topology) Jellyfish
(random graph)

6. Fat-tree

7. Fat-tree Topology
Core
Aggregation
Edge
k-ary tree
(here k = 4)

8. Fat-tree Topology k-ary tree Core (here k = 4) Aggregation Edge
k-port switches
k pods, each containing two layers of k/2 switches
Each switch in the lower layer is connected to k/2 hosts
k2/4 core switches
Each core switch has one port connected to each of k pods
Supports k3/4 hosts (determined by k)

9. Why fat-tree can deliver high bandwidth?
traditional topology
fat-tree BW 1 2 4 8 16 BW 16

10. Fat-tree vs. Traditional Topology
Advantages
Fat-tree achieves lower cost while delivering high bandwidth.
e.g. when supports 27,648 hosts at full bandwidth
-- Fat-tree (k = 48): costs $8.64M
-- traditional solution: costs $37M
Fat-tree requires no changes to end hosts.
-- is fully TCP/IP compatible
Fat-tree requires only a little modifications to the switches.

11. Fat-tree vs. Traditional Topology

12. Incremental Expansion
What is incremental expansion?
Adding servers and capacity incrementally to the data center.
What do we want?
The topology allows construction of arbitrary-sized networks.
Add new components to the existing topology without replacing or upgrading existing components.

13. Does fat-tree allow incremental expansion?
Does fat-tree allow construction of arbitrary-sized networks?
No!
The number of hosts a fat-tree can support is determined by k, i.e., the port-count of the switches it uses.
e.g., when k = 48, the fat-tree can only be built at size 27,648.
(k3/4 = 483/4 = 27,648)
Does fat-tree allow adding new components without replacing or upgrading existing components?
Even small incremental expansion would require replacing every existing switch to maintain the fat-tree structure.

14. Structure hinders incremental expansion
Fat-tree
The entire structure is completely determined by the port-count of the switches available.
Adding servers while preserving the structural properties would require replacing a large number of network elements.
Structure hinders incremental expansion.
Other topologies have similar problems.
How to solve this problem?
Use random graph!

15. Jellyfish

16. a degree-bounded random graph
Jellyfish Topology
Jellyfish
a degree-bounded random graph

17. Jellyfish Topology
Each switch uses some ports to connect to other switches, and uses the remaining ports for end hosts.
It doesn't require every switch have the same port-count.
If there are N switches in total, and all
switches are identical:
-- k-port switch
-- r ports for other switches
-- (k – r) ports for hosts
then the network can supports N * (k – r) hosts.

18. Jellyfish Topology How to construct a jellyfish?
Simply pick a random pair of switches with free ports (for the switch-pairs are not already neighbors), join them with a link.
Repeat until no further links can be added.
How to add a new switch (i.e., incremental expansion)?
Randomly remove an existing link (x, y).
Add link (p1, x) and (p2, y), where p1 and p2 are two ports on the new switch.

19. Jellyfish Properties Flexible.
Incrementally constructed Jellyfish has the same capacity as Jellyfish built from scratch.
Large throughput.
-- achieves 91% of the theoretical optimal throughput
-- leaves little room for improvement
Low mean path length.
Highly failure resilient.
-- maintains its structure in the face of link or node failures
-- a random graph with a few failures is just another random graph of slightly smaller size

20. Jellyfish vs. Fat-tree

21. Jellyfish vs. Fat-tree – Throughput
Jellyfish can support 27% more servers at full capacity than a fat- tree with the same switching equipment (i.e., with the same cost).

22. Jellyfish vs. Fat-tree – Path Length
With 686 servers, >99.5% of source-destination pairs in Jellyfish can be reached in <6 hops, while the corresponding number is only 7.5% in the fat-tree.

23. Jellyfish vs. Fat-tree – Failure Resilience
Both topologies are highly resilient to failures. But the throughput per server decreases more gracefully for Jellyfish than for a same- equipment fat-tree as the percentage of failed links increases.

24. Jellyfish vs. Fat-tree – Flexibility
Arbitrary-sized Networks:
Jellyfish: allows construction of arbitrary-size networks √
Fat-tree: the size is constrained by the port-count of switches ×
Incremental Expandability:
Jellyfish: can be expanded easily, and the desirable properties are maintained: high bandwidth and short paths at low cost √
Fat-tree: structure hinders incremental expansion ×

25. Routing & Congestion Control
Jellyfish
Fat-tree
Routing
k-shortest-paths
ECMP
(equal cost multi-
path routing)
Congestion Control
MPTCP
(multipath TCP )

26. References
[1] Singla A, Hong C Y, Popa L, et al. Jellyfish: Networking data centers randomly[C]//Presented as part of the 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI 12). 2012:
[2] Al-Fares M, Loukissas A, Vahdat A. A scalable, commodity data center network architecture[J]. ACM SIGCOMM Computer Communication Review, 2008, 38(4):

27. Thanks!