Distributed Virtual-Time Scheduling in Rings (DVSR) Chun-Hung Chen National Taipei University of Technology

Outlines RPR Recall Problems in RPR Ring Ingress Aggregated with Spatial Reuse Fairness (RIAS) Distributed Virtual-Time Scheduling in Rings (DVSR) Simulation Results Conclusions

RPR Recall RPR stands for Resilient Packet Ring, which is in IEEE Draft State Dual rings structure with Destination strip mechanism

Traffic is classified in three classes: Class A (A0 or A1), Class B (CIR or EIR), Class C Class A (A0 or A1), Class B (CIR or EIR), Class C When congested, the station will compute its approximation fair rate by Dividing the available bandwidth between all upstream stations that are currently sending frames through this station Dividing the available bandwidth between all upstream stations that are currently sending frames through this station Using its own current add rate Using its own current add rate

Two operation mode Conservative Mode Conservative Mode Congested station will wait a FRTT to send a new fair rate if it is still in congestion Aggressive Mode Aggressive Mode Congested station sends new fair rate in every 100μs if it is still in congestion

Problems in RPR Single Rate Controller Per-destination rate controller is optional Per-destination rate controller is optional Permanent Oscillation With Unbalanced Constant-Rate Traffic Inputs Unbalanced traffic will trigger severe and permanent oscillations Unbalanced traffic will trigger severe and permanent oscillations Computed add_rate or Capacity/Active_Stations do not reflect the true situation Computed add_rate or Capacity/Active_Stations do not reflect the true situation Throughput Loss Utilization degrades due to oscillation Utilization degrades due to oscillation AM & CM Convergence Slow convergence time Slow convergence time

Transit traffic has priority over ingress “ station ” traffic Each node measures my_rate of ingress traffic If a node is congested: send my_rate upstream send my_rate upstream upstream nodes throttle to my_rate upstream nodes throttle to my_rate my_rate allow_rate …… congested Throttle traffic and propagate rate upstream Throttle traffic After approximately 500 iterations, converges to the fair rates in this case

The Problem with Darwin my_rate is NOT the ring-wide fair rate Example of permanent oscillation and throughput degradation in Darwin:

Modeling RPR Oscillations (Analytical and Simulation Results) Conservative Mode Aggressive Mode Model accurately matches simulation

RIAS Ring Ingress Aggregated with Spatial Reuse Fairness Define the level of traffic granularity for fairness determination at a link as an ingress-aggregated (IA) flow Define the level of traffic granularity for fairness determination at a link as an ingress-aggregated (IA) flow Ensure maximal spatial reuse subject to the first constraint Ensure maximal spatial reuse subject to the first constraint Steps of RIAS Allocate bandwidth on each link locally fair according to an ingress aggregated granularity (IA traffic) Allocate bandwidth on each link locally fair according to an ingress aggregated granularity (IA traffic) Refine bandwidth allocation for each IA flow according to its egress point and bottlenecks Refine bandwidth allocation for each IA flow according to its egress point and bottlenecks Reclaim unused bandwidth fairly by iterating Reclaim unused bandwidth fairly by iterating Highly Similar to Max-Min Flow Control

Comparison Proportional Fair Allocation Penalizes flows farther away from the destination Penalizes flows farther away from the destination Important for TCP in the Internet (rate decrease with RTT) Important for TCP in the Internet (rate decrease with RTT) Fairness with Ingress-Egress flow granularity Incorrectly rewards nodes for spreading out traffic to many destination versus all to hub node Incorrectly rewards nodes for spreading out traffic to many destination versus all to hub node

Illustration of RIAS Fair (1/3) Parking Lot 4 flows each receive rate ¼ 4 flows each receive rate ¼ 1/4

Illustration of RIAS Fair (2/3) Parallel Parking Lot Each flow receives rate ¼ on downstream link Each flow receives rate ¼ on downstream link Left 1-hop flow fully reclaims excess bandwidth (RIAS) Left 1-hop flow fully reclaims excess bandwidth (RIAS) 1/4 3/4

1/4 3/4 1/2 1/4 1/2 Upstream Parallel Parking Lot Key points: Key points: Flow granularity for fairness Spatial reuse Illustration of RIAS Fair (3/3)

Proportional Fair “ Proportional fairness ” Penalizes flows farther away from the hub Penalizes flows farther away from the hub Important for TCP in the Internet (rate decreases with RTT) Important for TCP in the Internet (rate decreases with RTT) TCP/GigE approximates this in the parking lot TCP/GigE approximates this in the parking lot Variants of all of these have been discussed and proposed in the RPR standard meetings

Ingress-Egress Flow Granularity Fairness with Ingress-Egress flow granularity Incorrectly rewards nodes for spreading out traffic to many destinations vs. all to hub node Incorrectly rewards nodes for spreading out traffic to many destinations vs. all to hub node Wrong flow granularity counts 6 flows and gives rate 1/6 Wrong flow granularity counts 6 flows and gives rate 1/6 (RIAS-fair: all green flows together get ¼ vs ½ ) (RIAS-fair: all green flows together get ¼ vs ½ )

DVSR Nodes construct a proxy of virtual time at the ingress-aggregated flow granularity Using per-ingress byte counts Using per-ingress byte counts The proxy is a lower bound on virtual time temporally aggregated over time and spatially aggregated over traffic flows sharing the same ingress point (IA flows)

Distributed Fair Bandwidth Allocation Remote Fair Queuing Control of upstream rate controllers via use of ingress-aggregated virtual time as a congestion message received from downstream nodes Control of upstream rate controllers via use of ingress-aggregated virtual time as a congestion message received from downstream nodes Conceptually an ideal GPS processor Conceptually an ideal GPS processor Delayed and Temporally Aggregated Control Information Proxy of Virtual Time Proxy of Virtual Time Multinode RIAS Fairness Three Steps to approximate RIAS Three Steps to approximate RIAS

Remote Fair Queuing: Single Resource Illustration Control of upstream rate controllers via downstream virtual time progression True fair queueing replaced with rate controllers + multiplexer Note: no packets queued in mux when  = 0

Example Link capacity = 1 pkt/sec T = 10 pkt transmission times b = 0.8 (fraction of time busy)  > 0 Controller set at t for rates in [t-T- , t-  ] Controller set at t for rates in [t-T- , t-  ] Limiter value = 0.8

Step I: Local Fairness Label nodes 1, …, N and links 1, …, N-1 r ij is the traffic demand between nodes i and j at a particular time instant r i n is the Ingress Aggregated traffic from ingress node i at link n r i n = ∑ j>n r ij r i n = ∑ j>n r ij The locally fair allocation on link n is R i n = max_min i (C,r 1 n,r 2 n, …,r i n, …, r n n )

Footnote on max_min What is max_min i ( )? The “ textbook ” definition of (locally) fair The “ textbook ” definition of (locally) fair Would be achieved by fair queueing if fair queueing was performed on ingress aggregates Would be achieved by fair queueing if fair queueing was performed on ingress aggregates Can write down the exact computation [BerGal92,p527] Can write down the exact computation [BerGal92,p527] Maximizing the network use allocated to the sessions with the minimum allocation Maximizing the network use allocated to the sessions with the minimum allocation

Step II: Ingress Fairly Sub-allocates Per-link Bandwidths R ij n = max_min j (R i n,r i,n+1,r i,n+2,…,r i,j,…,r i,N ) Ingress has bandwidth R i n on link n and divides it fairly among flows traversing n End-to-End rate is the bottleneck rate r i,j = min n R ij n, n=i, i+1,…,j-1

Step III: Iterate There may be further bandwidth available for spatial reuse –Due to multiple congestion points Iterate process such that all excess capacity is fairly reclaimed Set new capacity to all unallocated capacity C n =C n -∑ ij R ij n Go to Step I

DVSR Protocol Scheduling of Station versus Transit Packets FIFO queue FIFO queue Class A is not taken in consideration Class A is not taken in consideration Feedback Signal Computation Feedback Signal Transmission Control message is N bytes while there exist N stations Control message is N bytes while there exist N stations Each station i writes its value at i bytes Each station i writes its value at i bytes Rate Limit Computation Suballocate its per-link fair rates to the flows with different egress nodes Suballocate its per-link fair rates to the flows with different egress nodes

DVSR Protocol Scheduling FIFO (or SP) FIFO (or SP) Computation of feedback signal Byte count for each ingress node - lower bound of virtual time Byte count for each ingress node - lower bound of virtual time Order such that Order such that l 1 ≤ l 2 ≤ … ≤ l k l 1 ≤ l 2 ≤ … ≤ l k

Analysis of DVSR Fairness Bound Lemma 1 Lemma 1 A node-backlogged flow in DVSR can be under-throttled by at most (1-(1/N))CT Lemma 2 Lemma 2 A node-backlogged flow in DVSR can be over-throttled by at most (1-(1/N))CT Lemma 3 Lemma 3 The service difference during any interval for two flows i and j with infinite demand is bounded by 2(C-(1/N)C)T under DVSR

Simulations Results Fairness and Spatial Reuse Fairness in the Parking Lot Fairness in the Parking Lot Performance Isolation for TCP Traffic Performance Isolation for TCP Traffic RIAS versus Proportional Fairness for TCP Traffic RIAS versus Proportional Fairness for TCP Traffic Spatial Reuse in the Parallel Parking Lot Spatial Reuse in the Parallel Parking Lot Convergence Time Comparison

Fairness in the Parking Lot Four constant-rate UDP flows sending at 622 Mbps DVSR provides RIAS fair shares GigE does not

Spatial Reuse in the Parallel Parking Lot DVSR is within  1% of RIAS fair rates GigE favors downstream flows & cannot achieve spatial reuse Darwin achieves only if using “ multi-choke ” option CBR UDP flows sending at the link capacity

Upstream Parallel Parking Lot (Results in Unbalanced Traffic Even with Balanced Inputs) Darwin oscillation range is 0.25 to 0.75 and throughput loss is 14% Many other scenarios can result in traffic imbalances and throughput losses DVSR within 0.1% of RIAS Darwin Behavior

RIAS vs. Proportional Fairness for TCP Traffic Each flow =1 TCP micro flow (ftp/TCP Reno) Rate within  1% of RIAS fair rates for 1 TCP micro-flow GigE tends to provide “ proportional fair ” rates

Convergence Time in the Parking Lot l CBR UDP flows with rate 0.4 (248.8Mbps) l Flow(1,5), (2,5), (3,5), (4,5) begin transmission at times 0.0, 0.1, 0.2, and 0.3 seconds respectively l Convergence time 0.2 msec for DVSR, 50 msec for Darwin l Richer feedback signal allows faster convergence DVSR Gandalf

Inter-Node Performance Isolation of TCP/UDP Traffic Flow (1,5) TCP micro-flows Others are CBR UDP flows with rate 0.3 More TCP micro-flows – DVSR able to achieve RIAS fairness Darwin performance unknown (MAC sim incompatible with TCP)

Conclusions Link capacity does not be considered in RPR Do my_rate and forward_rate in RPR fit the bandwidth allocation? DVSR approximate RIAS quicker than RPR RPR may have better performance if feedback mechanism is modified

Reference V. Gambiroza, P. Yuan, B. Balzano, Y. Liu, S.Sheafor, “Design, Analysis, and Implementation of DVSR: A Fair High- Performance Protocol for Packet Rings”, IEEE/ACM Transactions on Networking, Feb F. Davik, M.Yilmaz, S. Gjessing, N. Uzun, “IEEE Resilient Packet Ring Tutorial”, IEEE Communicaion Magazine, Mar