1. Bandwidth on the Internet Mat Ford ISOC Standards & Technology ISOC Advisory Council Meeting, Hiroshima, Japan Sunday November 8 th 2009

2. Takeaways, upfront Gross bandwidth growth more than catered for by new capacity No Internet ‘brownouts’ anytime soon Broadband creating new realities for access nets More bandwidth provisioning isn’t the answer Additional capacity is quickly consumed (by design) Interests of content providers, ISPs and users not always well-aligned We need more data IETF has several streams of relevant work ISOC has a panel on this topic during IETF76 2009-11-08Bandwidth on the Internet, ISOC AC Meeting2

3. Bandwidth, defined A term that describes the amount of information that can be passed through a communications channel in a given amount of time; that is, the capacity of the channel. The bandwidth is usually expressed in 'bits per second’… Or bytes per second, leading to lots of confusion Used for expressing Capacity of a channel or interface (e.g. 100Mbps ethernet i/f) Rate of information transfer (e.g. downloading at 3.2KB/s) 2009-11-083Bandwidth on the Internet, ISOC AC Meeting

4. Internet is sharing Early objections to feasibility of packet-switching centred on the need for very large packet buffers to handle uncontrolled loads from end hosts Leonard Kleinrock credited with demonstrating the theoretical feasibility of packet-switched communications 2009-11-08Bandwidth on the Internet, ISOC AC Meeting4 Internet protocols have their origin in the desire to share computers in diverse locations Extension of the OS-paradigm of shared resources

5. Rules of the road Resource sharing Central to notions of Internet’s value and success Enables interconnection of diverse applications Over heterogeneous networking media with diverse speeds Sharing creates potential for demand to outstrip supply Necessitates a sharing mechanism This is congestion control 2009-11-08Bandwidth on the Internet, ISOC AC Meeting5

6. How much bandwidth? Cisco Forecasts 44 Exabytes per Month of IP Traffic in 2012 2009-11-08Bandwidth on the Internet, ISOC AC Meeting6

7. Some rough consensus? 2009-11-08Bandwidth on the Internet, ISOC AC Meeting7 Kenjiro Cho et al. see ~40% growth per annum since 2005, for IX peaks

8. What about capacity? 2009-11-08Bandwidth on the Internet, ISOC AC Meeting8 Telegeography: Capacity growing by 50% per annum

9. 2009-11-08Bandwidth on the Internet, ISOC AC Meeting9

10. 2009-11-08Bandwidth on the Internet, ISOC AC Meeting10

11. 2009-11-08Bandwidth on the Internet, ISOC AC Meeting11

12. 2009-11-08Bandwidth on the Internet, ISOC AC Meeting12

13. 2009-11-08Bandwidth on the Internet, ISOC AC Meeting13

14. 2009-11-08Bandwidth on the Internet, ISOC AC Meeting14

15. 2009-11-08Bandwidth on the Internet, ISOC AC Meeting15

16. Congestion 2009-11-08Bandwidth on the Internet, ISOC AC Meeting16

17. Definitions of congestion Arrival rate > service rate (λ > μ) Congestion = arrival rate/service rate (p=λ/μ) In the presence of queues, congestion only present when packets are dropped? This is TCP’s definition of congestion For network operators, congestion is usually measured over longer timescales, e.g. 15mins For economists, congestion occurs when increased use would impose a cost on existing users In this case congestion can occur before queues start to build or packets are dropped 2009-11-08Bandwidth on the Internet, ISOC AC Meeting17

18. Evolution of congestion control Congestion control originally handled by ICMP Widespread incidents of congestion collapse in mid-1980s Early TCP implementations had very bad retransmission behavior that pushed the entire network into a state where most packets were lost and the resultant throughput was negligible. Addition of slow start, congestion avoidance, fast retransmit and fast recovery algorithms to TCP 2009-11-08Bandwidth on the Internet, ISOC AC Meeting18 Image source: Internet Protocol Journal, Vol 9, No. 2

19. Standards evolution RFC 793 ‘Transmission Control Protocol’, STD 7 (Sept. 1981) Doesn’t specify any congestion control mechanism Congestion control handled by ICMP Source Quench RFC 1122 ‘Requirements for Internet Hosts – Communication Layers’ (October 1989) Mandates implementation of a congestion control algorithm RFC 2581 ‘TCP Congestion Control’ (April 1999) Defines the accepted mechanism Required for the avoidance of congestion collapse Flow-rate fairness 2009-11-08Bandwidth on the Internet, ISOC AC Meeting19 Ref: RFC 4614 ‘A Roadmap for TCP Specification Documents’

20. TCP Implemented on end hosts Regulates sending rate for a single flow Doesn’t observe sending rate over time, or whether there are multiple flows from a single end host Doesn’t regulate aggregate demand Doesn’t know where congestion occurred, when it occurs TCP will keep getting faster unless All data is sent Congestion experienced End system maxed out 2009-11-08Bandwidth on the Internet, ISOC AC Meeting20

21. TCP in action 2009-11-08Bandwidth on the Internet, ISOC AC Meeting21 Animation credit: Guido Appenzeller

22. TCP isn’t enough In his original paper, Van Jacobson noted some limitations While algorithms at the transport endpoints can insure the network capacity isn’t exceeded, they cannot insure fair sharing of that capacity. Only in gateways, at the convergence of flows, is there enough information to control sharing and fair allocation. Thus, we view the gateway ‘congestion detection’ algorithm as the next big step.’ Whether or not multiple flows share common bottleneck links isn’t clear from the edge – only net ops have this view 2009-11-08Bandwidth on the Internet, ISOC AC Meeting22

23. Cable vs. DSL architectures 2009-11-0823Bandwidth on the Internet, ISOC AC Meeting

24. voluntarily polite algorithms in endpoints pushes until congested equalises rates of data flows a game of chicken – taking all and holding your ground pays or start more ‘TCP-fair’ flows than anyone else (Web: x2, p2p: x5-100) or for much more data than others (video streaming or p2p file-sharing x200) net effect of both (p2p: x1,000-20,000 higher traffic intensity) 24 TCP-fairness bandwidth 2 bandwidth 1 capacity time (VoIP, VoD) unresponsive flow 3 Slide credit: Bob Briscoe 2009-11-08Bandwidth on the Internet, ISOC AC Meeting

25. In defence of flow-rate fairness Floyd and Allman (2008) ‘We do not, however, claim that flow-rate fairness is necessarily an optimal fairness goal or resource allocation mechanism for simple best-effort traffic. Simple best-effort traffic and flow-rate fairness are in general not about optimality, but instead are about a low-overhead service (best-effort traffic) along with a rough, simple fairness model (flow-rate fairness).’ 2009-11-08Bandwidth on the Internet, ISOC AC Meeting25

26. What’s different this time? Congestion collapse isn’t new Happened in the 80s Predicted again in the 90s (Metcalfe’s gigalapse) More predictions swirling around now Growing diversity of uses/users Increasing traffic volumes Increasing economic value Emergence of public policy debates 2009-11-08Bandwidth on the Internet, ISOC AC Meeting26

27. Impact of all-you-can-eat broadband 2009-11-08Bandwidth on the Internet, ISOC AC Meeting27 source: Ellacoya 2007 (now Arbor Networks) Source: Cho et al., Proceedings of ACM CoNEXT2008

28. A changed landscape Services that are now dominant consumers of bandwidth were simply not viable at pre-broadband access data rates Natural suppression of user demand Congestion is at the edge Huge growth of mass-market broadband since 2000 ‘a seismic shift in the nature of the congestion problem’ Bandwidth hungry apps and multiple users per connection Raised expectations Access nets now the dominant constraint on achievable throughput Changing diurnal patterns of consumption Growing symmetry of traffic patterns 2009-11-08Bandwidth on the Internet, ISOC AC Meeting28

29. 29 ISPs forced to subvert TCP 1.equal bottleneck flow rates (TCP) ? 2.access rate shared between active users, but weighted by fee (weighed fair queuing, WFQ) ? 3.volume caps tiered by fee ? 4.heaviest applications of heaviest users throttled at peak times by deep packet inspection (DPI) ? 29 bit-rate time bit-rate time bit-rate time bit-rate time Slide credit: Bob Briscoe 2009-11-08Bandwidth on the Internet, ISOC AC Meeting

30. Congestion isn’t a bug, it’s a feature Want transmissions to occur as fast as possible Lack of congestion implies network could be more efficient Techniques are available to enable the use of congestion signals to control flow rates without lost packets/retransmissions 2009-11-08Bandwidth on the Internet, ISOC AC Meeting30 time bit- rate time bit- rate  Graphic credit: Bob Briscoe

31. 31 light usage can go much faster hardly affects completion time of heavy usage doesn’t have to shift into night BitTorrent & Microsoft have protocols to do this but... punished by #2, #3 & #4 NOTE: weighted sharing doesn't imply differentiated network service just weighted aggressiveness of end-system's rate response to congestion bit-rate time bit-rate time bit-rate time 1. TCP 4. deep packet inspection (DPI) weighted TCP sharing bit-rate time 2. (weighted) fair queuing bit-rate time 3. volume caps Better alternatives Slide credit: Bob Briscoe 2009-11-08Bandwidth on the Internet, ISOC AC Meeting

32. 32 Becoming impossible to deploy a new use of the Internet Must negotiate arbitrary blocks and throttles en route Two confusable motives Fairer cost sharing Competitive advantage to own services How to deconfuse? how to encourage fairer cost sharing? Make cost of usage transparent Fixing Internet technology should avoid need for legislation 32 Slide credit: Bob Briscoe Limiting the horizon 2009-11-08Bandwidth on the Internet, ISOC AC Meeting

33. BBC iPlayer example Original version used P2P to minimise BBC wholesale bandwidth costs Latest update, P2P technology disabled Disliked by end-users Disliked by ISPs BBC claim wholesale bandwidth costs have fallen iPlayer Desktop warns users to beware ISP bandwidth caps and charges 2009-11-0833Bandwidth on the Internet, ISOC AC Meeting

34. Wrapping up 2009-11-08Bandwidth on the Internet, ISOC AC Meeting34 Recent studies conclude P2P declining Being replaced with video and other web2.0 content Network capacity growing faster than traffic volumes Interests of content providers, ISPs and users not always well-aligned Fair management of congestion is a whole network issue

35. We need more data http://mitas.csail.mit.edu/ MINTS CAIDA M-Lab Transparency wrt congestion management policies of net ops is a good start Mismanagement of congestion is stifling application innovation Want to avoid imposition of regulation that enshrines TCP flow-rate fairness in law – there is ample scope for innovation in congestion management mechanisms at both sub-second and sub-month timescales. 2009-11-08Bandwidth on the Internet, ISOC AC Meeting35

36. Network operators play a key role Determine supply of network resources Limit demand (access link capacity) Is there a role for network operators at shorter time scales? 2009-11-08Bandwidth on the Internet, ISOC AC Meeting36

37. An inflection point? Can the Internet community move to a new notion of what constitutes ‘fair’ resource allocation on the Internet? TCP has ‘incumbent’s advantage’ If it ain’t broke, don’t fix it! Has the environment changed so significantly that it is broke? Who should have responsibility for prioritising traffic during overload conditions? How can the answer to that question be made to work in practice? 2009-11-08Bandwidth on the Internet, ISOC AC Meeting37

38. Relevant IETF work Conex BoF Exposing expected congestion along the forwarding path of the Internet Ledbat WG Congestion control algorithm for scavenger service Alto WG Protocols for better-than-random peer selection MultipathTCP WG Simultaneously use multiple paths in a single TCP session Homegate BoF Getting uniform set of requirements to aid deployment 2009-11-08Bandwidth on the Internet, ISOC AC Meeting38

39. ISOC Panel Internet Bandwidth Growth: Dealing with Reality 11.45am – 12.45pm, Tuesday 10 th November Light lunch provided Orchid West Room, ANA Crowne Plaza Hotel Panellists Leslie Daigle (Moderator) Kenjiro Cho, IIJ Richard Woundy, Comcast Danny McPherson, Arbor Networks Lars Eggert, Nokia Research Center, IETF Transport AD Audiocast & report will be available 2009-11-08Bandwidth on the Internet, ISOC AC Meeting39

40. References The Evolution of Internet Congestion Steven Bauer, David Clark, William Lehr (MIT) Internet Cost Transparency: Mending Value Chain Incentives Bob Briscoe (BT) ATLAS Internet Observatory 2009 Annual Report C. Labovitz et al. (Arbor Networks, Inc.) Observing Slow Crustal Movement in Residential User Traffic Cho et al. (IIJ) 2009-11-08Bandwidth on the Internet, ISOC AC Meeting40