Foreleser: Carsten Griwodz Congestion Control Foreleser: Carsten Griwodz Email: griff@ifi.uio.no 26. Apr. 2006 1

Congestion and Flow Control Opposite objectives End-system Optimize its own throughput Possibly at the expense of other end-systems Opposite objectives Network Optimize overall throughput Transmission rate adjustment Two different problems Receiver capacity Network capacity Cannot be distinguished easily at all places Should be differentiated Internal congestion Flow control Congestion control Network Small receiver capacity Large receiver capacity Packet loss

Congestion 2 problem areas Possible approach to avoid both bottlenecks Receiver capacity Approached by flow control Network capacity Approached by congestion control Possible approach to avoid both bottlenecks Receiver capacity: “actual window”, credit window Network capacity: “congestion window” Valid send window = min(actual window, congestion window) Terms Traffic All packets from all sources Traffic class All packets from all sources with a common distinguishing property, e.g. priority

Congestion Persistent congestion Transient congestion Router stays congested for a long time Excessive traffic offered Transient congestion Congestion occurs for a while Router is temporarily overloaded Often due to burstiness Burstiness Average rate r Burst size b (#packets that appear at the same time) Token bucket model

Congestion Reasons for congestion, among others perfect Reasons for congestion, among others Incoming traffic overloads outgoing lines Router too slow for routing algorithms Too little buffer space in router When too much traffic is offered Congestion sets in Performance degrades sharply maximum transmission capacity of the subnet desirable packets delivered congested packets sent by application Congestion tends to amplify itself Network layer: unreliable service Router simply drops packet due to congestion Transport layer: reliable service Packet is retransmitted Congestion => More delays at end-systems Higher delays => Retransmissions Retransmissions => Additional traffic

Congestion Control General methods of resolution Strategies Increase capacity Decrease traffic Strategies Repair When congestion is noticed Explicit feedback (packets are sent from the point of congestion) Implicit feedback (source assumes that congestion occurred due to other effects) Methods: drop packets, choke packets, hop-by-hop choke packets, fair queuing,... Avoid Before congestion happens Initiate countermeasures at the sender Initiate countermeasures at the receiver Methods: leaky bucket, token bucket, isarithmic congestion control, reservation, …

Repair Principle No resource reservation Necessary steps Congestion detected Introduce appropriate procedures for reduction

Repair by Packet dropping Principle At each intermediate system Queue length is tested Incoming packet is dropped if it cannot be buffered We may not wait until the queue is entirely full To provide Connectionless service No preparations necessary Connection-oriented service Buffer packet until reception has been acknowledged

Repair by Packet dropping Output lines Input Assigning buffers to queues at output lines 1. Maximum number of buffers per output line Packet may be dropped although there are free lines 2. Minimal number of buffers per output line Sequences to same output line (“bursts”) lead to drops 3. Dynamic buffer assignment Unused lines are starved

Repair by Packet dropping 4. Content-related dropping: relevance Relevance of data connection as a whole or every packet from one end system to another end system Examples Favor IPv6 packets with flow id 0x4b5 over all others Favor packets of TCP connection (65.246.255.51,80,129.240.69.49,53051) over all others Relevance of a traffic class Favor ICMP packets over IP packets Favor HTTP traffic (all TCP packets with source port 80) over FTP traffic Favor packets from 65.246.0.0/16 over all others

Repair by Packet dropping Properties Very simple But Retransmitted packets waste bandwidth Packet has to be sent 1 / (1 - p) times before it is accepted (p ... probability that packet will be dropped) Optimization necessary to reduce the waste of bandwidth Dropping packets that have not gotten that far yet e.g. Choke packets

Repair by Choke Packets Principle Reduce traffic during congestion by telling source to slow down Procedure for router Each outgoing line has one variable Utilization u ( 0≤u≤1 ) Calculating u: Router checks the line usage f periodically (f is 0 or 1) u = a * u + ( 1 - a ) * f 0 ≤ a ≤ 1 determines to what extent "history" is taken into account u > threshold: line changes to condition "warning“ Send choke packet to source (indicating destination) Tag packet (to avoid further choke packets from down stream router) & forward it

Repair by Choke Packets Principle Reduce traffic during congestion by telling source to slow down Procedure for source Source receives the choke packet Reduces the data traffic to the destination in question by x1% Source recognizes 2 phases (gate time so that the algorithm can take effect) Ignore: source ignores further Choke packets until timeout Listen: source listens if more Choke packets are arriving yes: further reduction by X2%; go to Ignore phase no: increase the data traffic

Repair by Choke Packets Hop-by-Hop Choke Packets Principle Reaction to Choke packets already at router (not only at end system) Plain Choke packets Hop-by-hop Choke packets B C B C A D A D E F E F A heavy flow is established Congestion is noticed at D A Choke packet is sent to A The flow is reduced at A The flow is reduced at D A heavy flow is established Congestion is noticed at D A Choke packet is sent to A The flow is reduced at F The flow is reduced at D

Repair by Choke Packets Variation u > threshold: line changes to condition "warning“ Procedure for router Do not send choke packet to source (indicating destination) Tag packet (to avoid further choke packets from down stream router) & forward it Procedure at receiver Send choke packet to sender Other variations Varying choke packets depending on state of congestion Warning Acute warning For u instead of utilization Queue length ....

Repair by Choke Packets Properties Effective procedure But Possibly many choke packets in the network Even if Choke bits may be included in the data at the senders to minimize reflux End systems can (but do not have to) adjust the traffic Choke packets take time to reach source Transient congestion may have passed when the source reacts Oscillations Several end systems reduce speed because of choke packets Seeing no more choke packets, all increase speed again

Repair with Fair Queuing Background End-system adapting to traffic (e.g. by Choke-Packet algorithm) should not be disadvantaged Principle On each outgoing line each end-system receives its own queue Packet sending based on Round-Robin (always one packet of each queue (sender)) Enhancement "Fair Queuing with Byte-by-Byte Round Robin“ Adapt Round-Robin to packet length But weighting is not taken into account Enhancement "Weighted Fair Queuing“ Favoring (statistically) certain traffic Criteria variants In relation to VPs (virtual paths) Service specific (individual quality of service) etc.

Congestion Avoidance 26. Apr. 2006 18

Avoidance Principle Policies at various layers can affect congestion Appropriate communication system behavior and design Policies at various layers can affect congestion Data link layer Flow control Acknowledgements Error treatment / retransmission / FEC Network layer Datagram (more complex) vs. virtual circuit (more procedures available) Packet queueing and scheduling in router Packet dropping in router (including packet lifetime) Selected route Transport layer Basically the same as for the data link layer But some issues are harder (determining timeout interval)

Avoidance by Traffic Shaping Motivation Congestion is often caused by bursts Bursts are relieved by smoothing the traffic (at the price of a delay) Original packet arrival Smoothed stream Peak rate time Procedure Negotiate the traffic contract beforehand (e.g., flow specification) The traffic is shaped by sender Average rate and Burstiness Applied In ATM In the Internet (“DiffServ” - Differentiated Services)

Traffic Shaping with Leaky Bucket Principle Continuous outflow Congestion corresponds to data loss Described by Packet rate Queue length Implementation Easy if packet length stays constant (like ATM cells) Implementation with limited buffers Symbolic: bucket with outflow per time Output lines Input

Traffic Shaping with Token Bucket Principle Permit a certain amount of data to flow off for a certain amount of time Controlled by "tokens“ Number of tokens limited Implementation Add tokens periodically Until maximum has been reached) Remove token Depending on the length of the packet (byte counter) Comparison Leaky Bucket Max. constant rate (at any point in time) Token Bucket Permits a limited burst

Traffic Shaping with Token Bucket Principle Permit a certain amount of data to flow off for a certain amount of time Controlled by "tokens“ Number of tokens limited Number of queued packets limited Implementation Add tokens periodically Until maximum has been reached) Remove token Depending on the length of the packet (byte counter) Comparison Leaky Bucket Max. constant rate (at any point in time) Token Bucket Permits a limited burst packet burst

Traffic Shaping with Token Bucket Principle Permit a certain amount of data to flow off for a certain amount of time Controlled by "tokens“ Number of tokens limited Number of queued packets limited Implementation Add tokens periodically Until maximum has been reached) Remove token Depending on the length of the packet (byte counter) Comparison Leaky Bucket Max. constant rate (at any point in time) Token Bucket Permits a limited burst

Avoidance by Reservation: Admission Control Principle Prerequisite: virtual circuits Reserving the necessary resources (incl. buffers) during connect If buffer or other resources not available Alternative path Desired connection refused Example Network layer may adjust routing based on congestion When the actual connect occurs

Avoidance by Reservation Admission Control sender Sender oriented Sender (initiates reservation) Must know target addresses (participants) Not scalable Good security 1. reserve data flow 2. reserve 3. reserve receiver

Avoidance by Reservation Admission Control sender Receiver oriented Receive (initiates reservation) Needs advertisement before reservation Must know “flow” addresses Sender Need not to know receivers More scalable Insecure 3. reserve data flow 2. reserve 1. reserve receiver

Avoidance by Reservation Admission Control sender Combination? Start sender oriented reservation 1. reserve data flow 2. reserve reserve from nearest router 3. reserve receiver

Avoidance by Buffer Reservation Principle Buffer reservation Implementation variant: Stop-and-Wait protocol One buffer per router and connection (simplex, VC=virtual circuit) Implementation variant: Sliding Window protocol m buffers per router and (simplex-) connection Properties Congestion not possible Buffers remain reserved, Even if there is no data transmission for some periods Usually only with applications that require low delay & high bandwidth rsvd for conn 1 unreserved buffers 1 2 3 1 2 3

Avoidance by Isarithmic Congestion Control Principle Limiting the number of packets in the network by assigning "permits“ Amount of "permits" in the network A "permit" is required for sending When sending: "permit" is destroyed When receiving: "permit" is generated Problems Parts of the network may be overloaded Equal distribution of the "permits" is difficult Additional bandwidth for the transfer of "permits" necessary Bad for transmitting large data amounts (e.g. file transfer) Loss of "permits" hard to detect

Avoidance: combined approaches Controlled load Traffic in the controlled load class experiences the network as empty Approach Allocate few buffers for this class on each router Use admission control for these few buffers Reservation is in packets/second (or Token Bucket specification) Router knows its transmission speed Router knows the number of packets it can store Strictly prioritize traffic in a controlled load class Effect Controlled load traffic is hardly ever dropped Overtakes other traffic RFC 2211

Avoidance: combined approaches Expedited forwarding Very similar to controlled load A differentiated services PHB (per-hop-behavior) Approach Set aside few buffers for this class on each router Police the traffic Shape or mark the traffic Only at senders, or at some routers Strictly prioritize traffic in a controlled load class Effect Shapers drop excessive traffic EF traffic is hardly ever dropped Overtakes other traffic RFC 2211 Version IHL DS Identification Total length D M Fragment offset Time to live Protocol Destination Address Source address Header checksum 1

Internet Congestion Control TCP Congestion Control 26. Apr. 2006 33

TCP Congestion Control TCP limits sending rate as a function of perceived network congestion Little traffic – increase sending rate Much traffic – reduce sending rate TCP’s congestion algorithm has four major “components”: Additive-increase Multiplicative-decrease (together AIMD algorithm) Slow-start Reaction to timeout events

TCP Congestion Control sender receiver Initially, the CONGESTION WINDOW is 1 MSS (message segment size) round 1 16 8 4 2 1 sent packets per round (congestion window) Then, the size increases by 1 for each received ACK until a threshold is reached or an ACK is missing round 2 round 3 round 4 time

TCP Congestion Control Normally, the threshold is 64K sent packets per round (congestion window) time 40 20 10 5 80 15 30 25 35 75 55 45 50 65 60 70 Loosing a single packet (TCP Tahoe): threshold drops to half CONGESTION WINDOW CONGESTION WINDOW back to 1 16 8 4 2 1 Loosing a single packet (TCP Reno): if notified by timeout: like TCP Tahoe if notified by fast retransmit: threshold drops to half CONGESTION WINDOW CONGESTION WINDOW back to new threshold

Congestion Control: Example Parameters Maximum segment size = 1024 bytes Initial congestion window = 64 KB Areas Previously (transm. nr. = 0) Congestion window before timeout: 64 KB Threshold after timeout: 32 KB Congestion window after timeout: 1 KB Exponential area “slow start” Linear area “congestion avoidance” Note Altways observe receiver’s window size, i.e. use minimum of Congestion window Actual send window (receiver status)

Additive Increase Objective Method: congestion window To avoid immediate network overload After just recent overload has finished After timeout At the start of a data transmission Method: congestion window Defined/initiated at connection set-up Initial max. window size = 1 segment Threshold (based on previous max. window size) Max. size of segment to be transferred To be doubled as long as 1. segment size below a certain threshold and 2. all ACKs arrive in time (i.e. correct segment transfer) (each burst acknowledged, i.e. successfully transmitted, doubles congestion window) To be linearly increased 1. segment size above a certain threshold and

AIMD Threshold Assumption Use: on missing acknowledgements Adaptive Parameter in addition to the actual and the congestion window Assumption Threshold, i.e. adaptation to the network: “sensible window size” Use: on missing acknowledgements Threshold is set to half of current congestion window Congestion window is reduced Implementation- and situation-dependant: to 1 or to new threshold Use slow start of congestion window is below threshold Use: on timeout Congestion window is reset to one maximum segment Use slow start to determine what the network can handle Exponential growth stops when threshold is hit From there congestion window grows linearly (1 segment) on successful transmission

TCP Congestion Control Some parameters 65.536 byte max. per segment IP recommended value TTL interval 2 min Optimization for low throughput rate Problem 1 byte data requires 162 byte incl. ACK (if, at any given time, it shows up just by itself) Algorithm Acknowledgment delayed by 500 msec because of window adaptation Comment Often part of TCP implementation

TCP Congestion Control TCP assumes that every loss is an indication for congestion Not always true Packets may be discarded because of bit errors Low bit error rates Optical fiber Copper cable under normal conditions Mobile phone channels (link layer retransmission) High bit errors rates Modem cables Copper cable in settings with high background noise HAM radio (IP over radio) TCP variations exist

TCP Congestion Control TCP congestion control is based on the notion that the network is a “black box” Congestion indicated by a loss Sufficient for best-effort applications, but losses might severely hurt traffic like audio and video streams  congestion indication better enabling features like quality adaptation Approaches Use ACK rate rather than losses for bandwidth estimation Example: TCP Westwood Use active queue management to detect congestion

The TCP family TCP Tahoe Oldest TCP variation still around Assumes that every packets loss is due to congestion If there is at least one loss during one RTT Reduction of threshold to half of the congestion window Congestion window back to 1 Slow start Only the missing packet is retransmitted Efficient handling of single losses Inefficient handling of multiple losses by RTT “Fast Retransmit” One packet is sent for every arriving packet If there is a gap, acknowledge the last packet before the gap again Sender interprets 3 ACKs for the same packet as a loss event

The TCP family TCP Reno Next oldest variation still around In case of Fast Retransmit, continue with “Fast Recovery” Reduction of the threshold to half the congestion window Congestion window to the new threshold Do not enter slow start Increase the congestion window for every ACK, even if it is a duplicate ACK An arriving ACK means that a packet has arrived Although the sender doesn’t know which

The TCP family Problem: Packets get dropped in bursts First approach When queues get full and routers drop packets, all senders require time to notice the problem and back off Multiple losses are more likely than Tahoe and Reno assume Particularly bad for Reno First approach TCP New Reno Most frequently used today Sender-only modification to TCP Reno Define duration of Fast Recovery Note the highest sequence number sent when fast recovery was entered Stay in fast recovery until that sequence number is ACK’d For duplicate ACKs received during fast recovery Don’t reduce the congestion window or threshold Retransmit the lost packet Do not extend the fast recovery period Increase the congestion window by 1 (as for good ACKs)

The TCP family Problem: Assumes that every loss is an indication for congestion But loss may be due to bit errors Severity: growing because of wireless links, satellite links and new bit-error sources at multi-gigabit speeds Approach TCP SACK TCP Sack (Selective ACK) Becoming wide-spread Modification at both sides Reports in addition to standard cumulative ACK Successfully received blocks of packets behind a gap Always including the packets that triggered the ACK in first block Uses a TCP option There can’t be more than 40 bytes TCP options per packet SACK requires 2+8*n Most likely used with 10 bytes timestamp option (PAWS option) Therefore no more than 3 blocks Distinguish: single loss, multiple loss, actual duplicates

The TCP family Bottleneck router Bandwidth-delay product Term used under the assumption that exactly one router between sender and receiver is responsible for packet loss Approaches look for Capacity of the bottleneck router Available bandwidth at the bottleneck router Bandwidth-delay product 1 Gbit/s, 40msec  41Mbit can be “in flight” . D a t A c k n o w l e d g m s t=0 t=11.2 μsec t=20 msec t=40 msec

The TCP family Problem: A loss event reduces congestion window but not sending rate Tahoe, Reno, New Reno and SACK will send a new packet for every incoming ACKs until congestion window is used up They send in bursts Bottleck router’s queue grows and shrinks Oscillating RTTs, packet drops Approach TCP FACK TCP FACK (Forward ACK) Quite rare Sender-only modification In Fast Recovery Sender uses a timer to spread packets evenly over one RTT

The TCP family Problems Determines available bandwidth by probing Increase the transmission rate until there is a loss event Severity: blocks advancement of the sliding window until the packet is retransmitted Climbs very slowly to highest bandwidth utilization when bandwidth is high The threshold that ends slow start is fixed to 64k in original TCP Severity: inhibits high bandwidth communication, esp. communication among supercomputers Does not use more than 75% of the available bandwidth Sawtooth pattern Severity: resource waste, intensive jitter in high bandwidth environments Can’t use high bandwidth if delay is high due to limited credit window size The problem of “high bandwidth delay products” Impossible to have more than 216 bytes unacknowledged Severity: most severe for satellite communication and communication among supercomputers; problem increases linearly with network speeds, and network speed grows potentially

The TCP family TCP Vegas Rare Sender-only modification Tries to fix all of the 3 problems above Model Congestion build-up can be detected RTT gets longer Throughput drops

The TCP family TCP Vegas TCP Vegas is a congestion avoidance approach Congestion window maintenance In each RTT, time-stamp one packet Between two marked packets, count bytes sent and bytes ACK’d If throughput lower than expected congestion window – 1 If throughput higher than expected congestion window + 1 Retransmission handling Course-grained 500ms timer for original retransmission behavior Maintain a fine-grained timer for every packet in flight Retransmit if on the first duplicate ACK if the RTT is exceeded according to the fine-grained timer Slow start handling Grow congestion window only every second RTT in slow start Don’t leave slow start until congestion is detected TCP Vegas is a congestion avoidance approach

Internet Congestion Control TCP Congestion Avoidance 26. Apr. 2006 52

Random Early Detection (RED) Random Early Detection (discard/drop) (RED) uses active queue management Drops packet in an intermediate node based on average queue length exceeding a threshold TCP receiver reports loss in ACK Sender applies multiple decrease Idea Congestion should be attacked as early as possible Some transport protocols (e.g., TCP) react to lost packets by rate reduction

Random Early Detection (RED) Router drops some packet before congestion significant (i.e., early) Gives time to react Dropping starts when moving avg. of queue length exceeds threshold Small bursts pass through unharmed Only affects sustained overloads Packet drop probability is a function of mean queue length Prevents severe reaction to mild overload RED improves performance of a network of cooperating TCP sources No bias against bursty sources Controls queue length regardless of endpoint cooperation

Early Congestion Notification (ECN) Early Congestion Notification (ECN) - RFC 2481 an end-to-end congestion avoidance mechanism Implemented in routers and supported by end-systems Not multimedia-specific, but very TCP-specific Two IP header bits used ECT - ECN Capable Transport, set by sender CE - Congestion Experienced, may be set by router Extends RED if packet has ECT bit set ECN node sets CE bit TCP receiver sets ECN bit in ACK sender applies multiple decrease (AIMD) else Act like RED

Early Congestion Notification (ECN) Effects Congestion is not oscillating - RED & ECN ECN-packets are never lost on uncongested links Receiving an ECN mark means TCP window decrease No packet loss No retransmission

Endpoint Admission Control Motivation Let end-systems test whether a desired throughput can be supported In case of success, start transmission Applicability Only for some kinds of traffic (traffic classes) Inelastic flows Requires exclusive use of some resources for this traffic Assumes short queues in that traffic class Send probes at desired rate Routers can mark or drop probes Probe packets can have separate queues or use main queue

Endpoint Admission Control Thrashing and Slow Start Probing Thrashing Many endpoints probe concurrently Probes interfere with each other and all deduce insufficient bandwidth Bandwidth is underutilized Slow start probing Probe for small bandwidth Probe for twice the amount of bandwidth … Until desired speed is reached Start sending

XCP: eXplicit Control Protocol Provide feedback initialize Round-trip-time Desired congestion window Feedback update IP header XCP TCP header Payload

XCP: eXplicit Control Protocol Congestion Controller Goal: Match input traffic to link capacity Compute an average RTT for all connections Looks at queue Combined traffic changes by   ~ Spare Bandwidth  ~ - Queue Size sendable per RTT So,  =  Spare -  Queue Fairness Controller Goal: Divide  between flows to converge to fairness Looks at a state in XCP header If  > 0  Divide  equally between flows If  < 0  Divide  between flows proportionally to their current rates

TCP Friendliness 26. Apr. 2006 61

TCP Friendliness - TCP Compatible A TCP connection’s throughput is bounded wmax - maximum retransmission window size RTT - round-trip time Congestion windows size changes AIMD (additive increase, multiple decrease) algorithm TCP is said to be fair Streams that share a path will reach an equal share A protocol is TCP-friendly if Colloquial It long-term average throughput is not bigger than TCP’s Formal Its arrival rate is at most some constant over the square root of the packet loss rate

TCP Friendliness AIMD algorithm A TCP connection’s throughput is bounded wmax - maximum retransmission window size RTT - round-trip time The TCP send rate limit is In case of at least one loss in an RTT In case of no loss , Congestion windows size changes AIMD algorithm additive increase, multiple decrease That’s not generally true Bigger RTT higher loss probability per RTT slower recovery Disadvantage for long-distance traffic TCP is said to be fair Streams that share a path will reach an equal share

TCP Friendliness A protocol is TCP-friendly if Colloquial: It long-term average throughput is not bigger than TCP’s Formal: Its arrival rate is at most some constant over the square root of the packet loss rate P – packet loss rate C – constant value Rr – packet arrival rate The AIMD algorithm with a≠1/2 and b≠1 is still TCP-friendly, if the rule is not violated TCP-friendly protocols may - if the rule is not violated - Probe for available bandwidth faster than TCP Adapt to bandwidth changes more slowly than TCP Use different equations or statistics, i.e., not AIMD Not use slow start (i.e., don’t start with w=0)

Datagram Congestion Control Protocol (DCCP) Under development http://www.ietf.org/html.charters/dccp-charter.html Transport Protocol Offers unreliable delivery Low overhead like UDP Applications using UDP can easily change to this new protocol Accommodates different congestion control mechanisms Congestion Control IDs (CCIDs) Add congestion control schemes on the fly Choose a congestion control scheme TCP-friendly Rate Control (TFRC) is included Half-Connection Data Packets sent in one direction

Datagram Congestion Control Protocol (DCCP) Congestion control is plugable One proposal is TCP-Friendly Rate Control (TFRC) Equation-based TCP-friendly congestion control Receiver sends rate estimate and loss event history Sender uses models of SACK TCP to compute send rate Steady state TCP send rate Loss probability Retransmission timeout Number of packets ack’d by one ACK