1. Acknowledgement: slides include content from Hedera and MP-TCP authors
CS434/534: Topics in Network Systems Cloud Data Centers Transport: MP-TCP, DC Transport Issues Yang (Richard) Yang Computer Science Department Yale University 208A Watson
Acknowledgement: slides include content from Hedera and MP-TCP authors

2. Outline Admin and recap Cloud data center (CDC) network infrastructure
Background, high-level goals
Traditional CDC vs the one-big switch abstraction
VL2 design and implementation: L2 semantics, VLB/ECMP
Load-aware centralized load balancing (Hedera)
Distributed load balancing by end hosts: MP-TCP
DC transport
issues

3. Admin PS1 help by Jensen () Office hours on projects
Thursday: 1:30-3:30 pm

4. Recap: ECMP/VLB Hashing Collision ProblemD2 S4 D1 D4 D3

5. Recap: Centralized Scheduling of Flow Routing
Estimate
Flow Demands
Place
Flows
Detect
Large Flows
Detect large flows
Estimate flow demands
Place flows
Global first fit
Simulated annealing

6. Recap: Distributed Scheduling using Endhosts
Instead of a central scheduler, end hosts distributedly compute flow rates using TCP
Two separate paths
Logically a single pool =
In a circuit-switched network, there is a dedicated channel for each flow. It’s rigid and inflexible: when one flow is silent, the other flow can’t fill in. Packet switching gives you much more flexibility (whether you use it for ATM virtual circuits, or for full-blown IP).
Multipath brings flexibility of the same sort. Pic 3 is rigid and inflexible, Pic 4 is flexible.
In the case of packet switching, you need to be careful about how the flows share the link. Pic 1 circuits made it easy – they give strict isolation between flows. But with Pic 2 packet switching, you need a control plane. It could be with ATM and admission control. Or it could be TCP congestion control, which says how end-systems should adapt their rates so that the network shares its capacity fairly.
What sort of control plane do we need, to ensure that a multipath network works well?

7. Recap: Simple Model Driving TCP CC Design
User 1 x1
d = sum xi > Xgoal?
sum xi
User 2 x2 xn
User n
Flows observe congestion signal d, and locally take actions to adjust rates.

8. Recap: Distributed Linear Control
Considers the simplest class of control strategy to achieve fairness and efficiency

9. AIMD: State Transition Tracex2
fairness line: x1=x2
efficiency line: x1+x2=C
overload
underload x0 x1

10. Recap: Mapping from Model to Protocol
Window Based Mapping
Assume window size is cwnd segments, each with MSS bytes, round-trip time (RTT):
Rate x =
cwnd * MSS
RTT
Bytes/sec
MSS: Minimum Segment Size

11. Discussion
How can a sender know congestion?

12. Approach 1: End Hosts Consider Loss as Congestion
Packets 1 2 3 4 5 6 7
Acknowledgements (waiting seq#) 2 3 4 4 4 4
Pros and Cons of endhosts using loss as congestion?
Assume loss => cong

13. Approach 2: Network Feedback (ECN: Explicit Congestion Notification)
Sender reduces rate if ECN received.
Pros and Cons of ECN?
Sender 1
Receiver bounces marker back to sender in ACK msg
Receiver
Network marks ECN Mark (1 bit) on pkt according to local condition, e.g., queue length > K
Sender 2

14. TCP/Reno Full Alg Initially: cwnd = 1;
ssthresh = infinite (e.g., 64K);
For each newly ACKed segment:
if (cwnd < ssthresh) // slow start: MI
cwnd = cwnd + 1;
else
// congestion avoidance; AI
cwnd += 1/cwnd;
Triple-duplicate ACKs or ECN in RTT:
// MD
cwnd = ssthresh = cwnd/2;
Timeout:
ssthresh = cwnd/2; // reset
(if already timed out, double timeout value; this is called exponential backoff)

15. Recap: EW-TCP for Multiple TCP Sharing Bottleneck
A multipath TCP flow with two subflows
Regular TCP
Goal: The subflows of each flow share each bottleneck w/ other flows fairly.
Mechanism:
Each subflow uses basic TCP with an increase parameter a (instead of 1) per RTT
If n subflows share a bottleneck w/ a TC flow, choose a = 1/n2.
𝑚𝑒𝑎𝑛 𝑊= 𝑎 2(1−𝑝) 𝑝 ≈ 𝑎 2/𝑝

16. Discussion
Issues of EW-TCP

17. Equal vs Non-Equal Share Example

18. Equal vs Non-Equal Share Example
What is the throughput of each flow, if each subflow shares its path with others equally?
12Mb/s
8Mb/s
12Mb/s
8Mb/s
8Mb/s
12Mb/s

19. Equal vs Non-Equal Share Example
If each flow split its traffic 2:1 ...
12Mb/s
9Mb/s
12Mb/s
9Mb/s
9Mb/s
12Mb/s

20. Equal vs Non-Equal Share Example
If each flow split its traffic ∞:1 ...
12Mb/s
12Mb/s
12Mb/s
12Mb/s
12Mb/s
12Mb/s
Equal share may not be efficient: not even Pareto Optimal !

21. Outline Admin and recap Cloud data center (CDC) networks
Background, high-level goal
Traditional CDC vs the one-big switch abstraction
VL2 design and implementation: L2 semantics, VLB/ECMP
Load-aware centralized load balancing (Hedera)
Distributed load balancing by end hosts: MP-TCP
Intuition
EW-TCP
COUPLED

22. Design Option 2: Total Coupled TCP
Each ACK on subflow r, increase the window wr by 1/wtotal
Each loss on subflow r, decrease the window wr by wtotal/2
Q: what is the behavior, say two subflows with w1 and w2 respectively?

23. Discussion
Theory: MPTCP should send all its traffic on its least-congested (lowest-loss-rate) paths.
Kelly+Voice 2005; Han, Towsley et al. 2006
Issues of sending all traffic to least congested (lowest loss rate) path?

24. Outline Admin and recap Cloud data center (CDC) networks
Background, high-level goal
Traditional CDC vs the one-big switch abstraction
VL2 design and implementation: L2 semantics, VLB/ECMP
Load-aware centralized load balancing (Hedera)
Distributed load balancing by end hosts: MP-TCP
Intuition
EW-TCP
COUPLED TCP
SEMI-COUPLED TCP

25. Design Option 3: Semi Coupled TCP
Each ACK on subflow r, increase the window wr by a/wtotal
Each loss on subflow r, decrease the window wr by wr/2
Q: what is the equilibrium sharing of bw?

26. Semi-Coupled TCP 𝑚𝑒𝑎𝑛 Δ𝑊𝑟= 1−𝑝𝑟 𝑎 𝑊𝑡 +𝑝𝑟(− 𝑊𝑟 2 ) = 0
𝑚𝑒𝑎𝑛 Δ𝑊𝑟= 1−𝑝𝑟 𝑎 𝑊𝑡 +𝑝𝑟(− 𝑊𝑟 2 ) = 0
1−𝑝𝑟 𝑎 𝑊𝑡 =𝑝𝑟 𝑊𝑟 2
𝑊𝑟= 1−𝑝𝑟 2𝑎 𝑝𝑟𝑊𝑡 ≈ 2𝑎 𝑝𝑟𝑊𝑡
𝑊𝑡 = 2𝑎 𝑊𝑡 1/𝑝𝑟
𝑊𝑡= 2𝑎 1/𝑝𝑟
𝑊𝑟 = 2a 1/𝑝𝑟 1/𝑝𝑟

27. Outline Admin and recap Cloud data center (CDC) network infrastructure
Background, high-level goal
Traditional CDC vs the one-big switch abstraction
VL2 design and implementation: L2 semantics, VLB/ECMP
Load-aware centralized load balancing (Hedera)
Distributed load balancing by end hosts: MP-TCP
Intuition
Formal definition

28. Formal Requirement
So far we focus on window size (w) and ignore flows w/ different RTTs, but flow rate is w/RTT.
Formal requirement:
𝑟 ∈𝑅 𝑤𝑟 𝑅𝑇𝑇𝑟 ≥ max 𝑟 ∈𝑅 𝑤𝑟 𝑇𝐶𝑃 𝑅𝑇𝑇𝑟

29. Design Option 4
Each ACK on subflow r, increase the window wr by a/wtotal
Each loss on subflow r, decrease the window wr by wr/2
min(
, 1/wr)

30. Deriving a 𝑟 ∈𝑅 𝑤𝑟 𝑅𝑇𝑇𝑟 ≥ max 𝑟 ∈𝑅 𝑤𝑟 𝑇𝐶𝑃 𝑅𝑇𝑇𝑟
1−𝑝𝑟 min 𝑎 𝑤 𝑡𝑜𝑡𝑎𝑙 , 1 𝑤 𝑟 =𝑝𝑟 𝑤 𝑟 2
min 𝑎 𝑤 𝑡𝑜𝑡𝑎𝑙 𝑤 𝑟 , 1 𝑤 𝑟2 =𝑝𝑟 1 2(1−𝑝𝑟)
min 𝑎 𝑤 𝑡𝑜𝑡𝑎𝑙 𝑤 𝑟 , 1 𝑤 𝑟2 ≈ 𝑝𝑟 2
𝑤𝑟 𝑡𝑐𝑝= 2/𝑝𝑟
min 𝑎 𝑤 𝑡𝑜𝑡𝑎𝑙 𝑤 𝑟 , 1 𝑤 𝑟2 = 1 𝑤 𝑟𝑡𝑐𝑝2
max 𝑤 𝑡𝑜𝑡𝑎𝑙 𝑤 𝑟 𝑎 , 𝑤 𝑟 = 𝑤 𝑟𝑡𝑐𝑝
max 𝑤 𝑡𝑜𝑡𝑎𝑙 𝑤 𝑟 𝑎 𝑅𝑇𝑇𝑟2 , 𝑤 𝑟 𝑅𝑇𝑇𝑟 = 𝑤𝑟 𝑡𝑐𝑝 𝑅𝑇𝑇𝑟

31. Deriving a 𝑟 ∈𝑅 𝑤𝑟 𝑅𝑇𝑇𝑟 ≥ max 𝑟 ∈𝑅 𝑤𝑟 𝑇𝐶𝑃 𝑅𝑇𝑇𝑟

32. Final MP-TCP Alg

33. MPTCP Evaluation Throughput (% of optimal) FatTree, 128 nodes
Flow rank
Simulations of FatTree, 100Mb/s links, permutation traffic matrix, one flow per host, TCP+ECMP versus MPTCP.

34. Hedera first-fit heuristic
MPTCP vs Hedera
Throughput [% of optimal]
Simulation of FatTree with 128 hosts.
Permutation traffic matrix
Closed-loop flow arrivals (one flow finishes, another starts)
Flow size distributions from VL2 dataset
MPTCP
Hedera first-fit heuristic

35. MPTCP at Different Load
Ratio of throughputs, MPTCP/TCP
Simulation of a FatTree-like topology with 512 nodes, but with 4 hosts for every up-link from a top-of-rack switch, i.e. the core is oversubscribed 4:1.
Permutation TM: each host sends to one other, each host receives from one other
Random TM: each host sends to one other, each host may receive from any number
Connections per host
At low loads, there are few collisions, and NICs are saturated, so TCP ≈ MPTCP
At high loads, the core is severely congested, and TCP can fully exploit all the core links, so TCP ≈ MPTCP
When the core is “right-provisioned”, i.e. just saturated, MPTCP > TCP

36. MPTCP at Different Load
Underloaded
Sweet Spot
Overloaded

37. A Puzzle: cwnd and Rate of a TCP Session
Question: cwnd fluctuates widely (i.e., cut to half); how can the sending rate stay relatively smooth?
Question: Where does loss rate come from?

38. TCP/Reno Queueing Dynamics
cwnd
filling buffer TD
bottleneck
bandwidth
ssthresh
draining buffer
Time
congestion
avoidance
If the buffer at the bottleneck is large enough, the buffer is never empty (not idle), during the cut-to-half to “grow-back” process.

39. Discussion
If the buffer size at the bottleneck link is very small, what is the link utilization?
cwnd TD
bottleneck
bandwidth
ssthresh
Time
congestion
avoidance

40. Exercise: Small Buffer
Assume
BW: 10 G
RTT: 100 ms
Packet: 1250 bytes
BDP (full window size): 100,000 packets
A loss can cut window size from 100,000 to 50,000 packets
To fully grow back
Need 50,000 RTTs => 5000 seconds, 1.4 hours

41. Discussion What you like about MP-TCP?
What you do not like about MP-TCP?

42. Outline Admin and recap Cloud data center (CDC) network infrastructure
Background, high-level goals
Traditional CDC vs the one-big switch abstraction
VL2 design and implementation: L2 semantics, VLB/ECMP
Load-aware centralized load balancing (Hedera)
Distributed load balancing by end hosts: MP-TCP
DC transport
issues

43. Big Picture
INTERNET
Servers
Fabric

44. Diverse workload sharing the same infrastructure
Big Picture
INTERNET
Fabric
Diverse workload sharing the same infrastructure
Specifically, while some of the traffic in data center networks is sent across the Internet, the majority of data center traffic is between the servers within the data center and never leaves the data center.
This is because
Servers
web
app
cache
data-base
map-reduce
HPC
monitoring

45. Different Types of Workloads
Mice & Elephants
Short messages
(e.g., query, coordination)
Large flows
(e.g., data update, backup)
Low Latency
High Throughput
So the ultimate goal of data center transport is to complete these internal transactions or “flows” as quickly as possible.
What makes this challenging is that the flows are actually quite diverse. In particular, there are…
And these different flows require different things to complete quickly. For the short flows, we need to provide low latency…

46. Implication: Mice Mix w/ Elephants

47. Discussion
cwnd TD
bottleneck
bandwidth
ssthresh
Time
congestion
avoidance
What is potential issue of TCP for such mixed mice-elephants traffic?