1. COS 561: Advanced Computer Networks
Multipath TCP
Jennifer Rexford
Fall 2017 (TTh 1:30-2:50 in CS 105)
COS 561: Advanced Computer Networks

2. Multipath Mobile user High-end servers Data centers
WiFi and cellular at the same time
High-end servers
Multiple Ethernet cards
Data centers
Rich topologies with many paths
Benefits of multipath
Higher throughput
Failover from one path to another
Seamless mobility

3. Multipath TCP Protocol

4. Working With Unmodified Apps
Present the same socket API and expectations
Identified by the “five tuple” (IP address, port #, protocol)
From

5. Working With Unmodified Hosts
Establish the TCP connection in the normal way
Create a socket to a single remote IP address/port
And then add more subflows, if possible A B
SYN
SYN ACK
Each host tells its Initial Sequence Number (ISN) to the other host.
ACK
Data
Data

6. Negotiating MPTCP Capability
How do hosts know they both speak MPTCP?
During the 3-way SYN/SYN-ACK/ACK handshake
If SYN-ACK doesn’t contain MP_CAPABLE
Don’t try to add any subflows!

7. Adding Subflows, Idealized
How to associate a new subflow with the connection?
Use a token generated from original subflow set-up
How to start using the new subflow?
Simply start sending packets with new IP/port pairs
… and associate them with the existing connection
How could two end-points learn about extra IP addresses for establishing new subflows?
Implicitly: one end-point establishes a new subflow, to already-known address(es) at the other end-point

8. Sequence Numbers Challenges across subflows
Out-of-order packets due to RTT differences
Access networks that rewrite sequence numbers
Middleboxes upset by discontinuous TCP byte stream
Need to retransmit lost packets on a different subflow
Two levels of sequence numbers
Sequence numbers per subflow
Sequence numbers for the entire connection
Enables
Efficient detection of loss on each subflow
Retransmission of lost packet on a different subflow

9. Receive Buffer Space Each TCP connection has a receive buffer
Buffer space to store incoming data
… until it is read by the application
TCP flow control
Receiver advertises the available buffer space
… using the “receive window”
Should each subflow have its own receive window?
Starvation of some subflows in a connection?
Fairness relative to other TCP connections?
Fragmentation of the available buffer space?
Instead, use a common receive window

10. Fairness and Efficiency in Multipath Congestion Control
Slides from Damon Wischik

11. Goal #1: Fairness at Shared Bottlenecks
To be fair, Multipath TCP should take as much capacity as TCP at a bottleneck link, no matter how many paths it is using.
A multipath TCP flow with two subflows
Regular TCP
This is the very first thing that comes to mind with multipath TCP, and it’s something that many other people have solved in different ways.
This is just a warm-up... Design Goal 3 is a much “richer” generalization of this goal, which accommodates different topologies, different RTTs. So there’s no point giving an evaluation here.

12. Goal #2: Use Efficient Paths
12Mb/s
12Mb/s
12Mb/s
Each flow has a choice of a 1-hop and a 2-hop path. How should split its traffic?
I’m thinking of the paths as given. MPTCP has the choice of how to split its traffic over those given paths.

13. Use Efficient Paths If each flow split its traffic 1:1 ... 12Mb/s

14. Use Efficient Paths If each flow split its traffic 2:1 ... 12Mb/s

15. Use Efficient Paths
12Mb/s
12Mb/s
Better: Each connection on a one-hop path Each connection should send all traffic on the least-congested paths
12Mb/s
12Mb/s
12Mb/s
12Mb/s

16. Use Efficient Paths
Better: Each connection on a one-hop path Each connection should send all traffic on the least-congested paths But keep some traffic on the alternate paths as a probe
12Mb/s
12Mb/s
12Mb/s
12Mb/s
12Mb/s
12Mb/s

17. Goal #3: Be Fair Compared to TCP
Least-congested paths may not be best!
Due to differences in round-trip time
Two paths
WiFi: high loss, low RTT
Cellular: low loss, high RTT
Using the least-congested path
Choose the cellular path, due to low loss
But, the RTT is high
So throughput is low!

18. Be Fair Compared to TCP
To be fair, Multipath TCP should give a connection at least as much throughput as it would get with a single-path TCP on the best of its paths.
Ensure incentive for deploying MPTCP
A Multipath TCP should take no more capacity on any path (or collection of paths) than if it was a single-path TCP flow using the best of those paths.
Do no harm!

19. Achieving These Goals Regular TCP MPTCP
Maintain a congestion window w
On an ACK, increase by 1/w (increase 1 per window)
On a loss, decrease by w/2
MPTCP
Maintain a congestion window per path wr
On an ACK on path r, increase wr
On a loss on path r, decrease by wr/2
How much to increase wr on an ACK??
If r is the only path at that bottleneck, increase by 1/wr

20. If Multiple Paths Share Bottleneck?
Don’t take any more bandwidth on a link than the best of the TCP paths would
But, where might the bottlenecks be?
Multiple paths might share the same bottleneck
So, consider all possible subsets of the paths
Set R of paths
Subset S of R that includes path r
E.g., consider path 3
Suppose paths 1, 3, and 4 share a bottleneck
… but, path 2 does not
Then, we care about S = {1,3,4}

21. Achieving These Goals What is the best of these subflows achieving?
Path s is achieving throughput of ws/RTTs
So best path is getting maxs(ws/RTTs)
What total bandwidth are these subflows getting?
Across all subflows sharing that bottleneck
Sum over s in S of ws/RTTs
Consider the ratio of the two
Increase by less if many subflows are sharing
And pick the results for the set S with min ratio
To account for the most paths sharing a bottleneck

22. Incremental Deployment Challenges of Middleboxes

23. Middleboxes In-network services, e.g., Interaction with TCP Firewall
Network address translator
Transparent proxy
Intrusion detection system
Interaction with TCP
Change IP addresses and port numbers
Change TCP initial sequence number
Remove TCP options
Dividing large block of data into smaller packets
Expect to see all packets of the connection
Etc.

24. Negotiating MPTCP Capability
What if middleboxes strip the TCP option?
On the SYN? On the SYN-ACK?
Include capability on the ACK of the SYN-ACK?
What if the ACK is lost?
Carry on all subsequent packets
What if the middlebox drops SYN packets with unfamiliar options?
Sender can retransmit lost SYN without the option
… and fall back to regular TCP behavior

25. Challenges: NAT Network Address Translators (NAT) NAT1
Problem: NAT changes the IP address and port number
How to identify a connection?
Using a token established during connection set-up
How to establish new subflows?
Allow one end-point to tell another about its addresses
WiFi
NAT1
NAT2
LTE

26. Challenges: Security Security How to bootstrap security?
Malicious parties creating subflows
To highjack (part of) the connection
How to bootstrap security?
Include a random key during connection set-up
… and use it to verify authenticity of new subflows
How to identify the connection on new subflows?
A token generated from the key
How to authenticate the addition of subflows?
Exchanging nonces and computing message authentication codes using the keys

27. Use of Multipath TCP in iOS 7
Multipath TCP in iOS 7 (fall 2013)
Primary TCP connection over WiFi
Backup TCP connection over cellular data
Failover
If WiFi becomes unavailable…
… iOS 7 will use the cellular data connection
For destinations controlled by Apple
E.g., Siri
See

28. Discussion

29. Backup Slides: Review of TCP Protocol

30. Establishing a TCP Connection
SYN
SYN ACK
Each host tells its Initial Sequence Number (ISN) to the other host.
ACK
Data
Data
Three-way handshake to establish connection
Host A sends a SYN (open) to the host B
Host B returns a SYN acknowledgment (SYN ACK)
Host A sends an ACK to acknowledge the SYN ACK

31. Initial Sequence Number (ISN)
Sequence number for the very first byte
E.g., Why not a de facto ISN of 0?
Practical issue: reuse of port numbers
Port numbers must (eventually) get used again
… and an old packet may still be in flight
… and associated with the new connection
Security issue: adversary injecting packets
Adversary may try to inject packets in a connection
… by guessing the Initial Sequence Number
… to send counterfeit packets to the receiving host
… e.g., counterfeit packets that reset the connection
Some firewalls change the ISN to further randomize

32. Step 1: A’s Initial SYN Packet
A’s port
B’s port
A’s Initial Sequence Number
Flags:
SYN
FIN
RST
PSH
URG
ACK
Acknowledgment 20
Flags
Advertised window
Checksum
Urgent pointer
Options (variable)
A tells B it wants to open a connection…

33. Step 2: B’s SYN-ACK Packet
B’s port
A’s port
B’s Initial Sequence Number
Flags:
SYN
FIN
RST
PSH
URG
ACK
A’s ISN plus 1 20
Flags
Advertised window
Checksum
Urgent pointer
Options (variable)
B tells A it accepts, and is ready to hear the next byte…
… upon receiving this packet, A can start sending data

34. Step 3: A’s ACK of the SYN-ACK
A’s port
B’s port
Sequence number
Flags:
SYN
FIN
RST
PSH
URG
ACK
B’s ISN plus 1 20
Flags
Advertised window
Checksum
Urgent pointer
Options (variable)
A tells B it is okay to start sending
… upon receiving this packet, B can start sending data

35. Sequence number = 1st byte
Host A
ISN (initial sequence number)
Byte 81
Sequence number = 1st byte
TCP Data
TCP Data
Host B

36. TCP Header Data Source port Destination port Sequence number Flags:
SYN
FIN
RST
PSH
URG
ACK
Acknowledgment
HdrLen
Flags
Advertised window
Checksum
Urgent pointer
Options (variable)
Data

37. Receive Buffering: Flow Control
Receive window size
Amount that can be sent without acknowledgment
Receiver must be able to store this amount of data
Receiver tells the sender the window
Tells the sender the amount of free space left
Window Size
Data ACK’d
Outstanding
Un-ack’d data
Data OK
to send
Data not OK
to send yet

38. TCP Header: Receive Window
Source port
Destination port
Sequence number
Flags:
SYN
FIN
RST
PSH
URG
ACK
Acknowledgment
HdrLen
Flags
Advertised window
Checksum
Urgent pointer
Options (variable)
Data

39. Tearing Down the ConnectionB
ACK
SYN
SYN ACK
Data
ACK
FIN
ACK
FIN
ACK A
time
Closing (each end of) the connection
Finish (FIN) to close and receive remaining bytes
And other host sends a FIN ACK to acknowledge
Reset (RST) to close and not receive remaining bytes

40. Extending TCP: TCP Options
TCP header
Ten mandatory fields
Optional extension field (usually during handshake)
Examples
Maximum segment size (MSS)
Window scaling
Support for Selected ACKs
Unknown options
Ignored by receiving host
Routers and TCP options
Should ignore them, passing them through unchanged
But, some middleboxes: (i) strip TCP options from some packets or (ii) drop packets with TCP options