Presentation is loading. Please wait.

Presentation is loading. Please wait.

SAC 2002-08-07 Campus Bandwidth Management Solutions1 Campus Bandwidth Management Solutions ken lindahl UC Berkeley SAC 7 Aug.

Similar presentations

Presentation on theme: "SAC 2002-08-07 Campus Bandwidth Management Solutions1 Campus Bandwidth Management Solutions ken lindahl UC Berkeley SAC 7 Aug."— Presentation transcript:

1 SAC Campus Bandwidth Management Solutions1 Campus Bandwidth Management Solutions ken lindahl UC Berkeley SAC 7 Aug 2002 Copyright ken lindahl This work is the intellectual property of the author. Permission is granted for this material to be shared for non- commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

2 SAC Campus Bandwidth Management Solutions2 where i’m coming from technical lead for UC Berkeley’s external connectivity for the past 7+ years, including: CalREN-2 and Internet2, commodity Internet service and private peerings. this talk is very much shaped by my experiences at Berkeley, probably 95%, with the other 5% coming from conversations and collaborations with network engineers from other academic institutions. in particular, Berkeley uses Packeteer PacketShapers; all of my experience is with PacketShapers. –i think it’s important to keep that in mind as you listen to this talk.

3 SAC Campus Bandwidth Management Solutions3 “solutions” there’s no “one size fits all” solution to managing campus bandwidth. in fact, it’s probably cleaner to say there’s no “solution,” just tools, techniques, strategies… if you’re not sure whether something makes sense for your campus, please ask. i’ll take questions or comments at any time.

4 SAC Campus Bandwidth Management Solutions4 why manage campus bandwidth? cost –generally, we’re talking about ISP costs, which typically are usage-sensitive. - Berkeley’s ISP traffic for Oct June 2002 (with rate-limiting) - cost:about $300 per Mbps (95 th %-ile) per month about $300,000 Jul 2001-Jun 2002

5 SAC Campus Bandwidth Management Solutions5 why manage campus bandwidth ? (2) cost performance –i.e. congestion control/avoidance. –minimize negative effects of “less desirable” traffic (music, video, software sharing) on “more desirable” data (research, learning, business operations); i.e. “protect mission-critical applications.” –generally, we’re talking about ISP links, since it’s generally easier and more effective to add bandwidth on interior links. –and we’re also talking about cost, since if cost were not an issue, it would be easier and more effective to add bandwidth to ISP links.

6 SAC Campus Bandwidth Management Solutions6 why manage campus bandwidth ? (3) cost performance policy –some campuses have policies requiring some form of traffic shaping, e.g. blocking peer-to-peer applications because they are used primarily for exchange of copyrighted materials. –Berkeley does not have such a policy.

7 SAC Campus Bandwidth Management Solutions7 why manage campus bandwidth ? (4) cost performance policy other reasons? (audience)

8 SAC Campus Bandwidth Management Solutions8 “you can’t go fast only on Internet2” “While the focus of I2 is properly on Abilene and related high performance networks initiatives, what one does to enable high network throughput for I2 purposes has an impact on commodity Internet transit traffic levels, too -- you can’t go fast only on Internet2. “… Pretty much everybody has engineered and deployed fast campus networks that enable high throughput to Internet2 -- and to the commodity Internet. Those networks usually encompass not only faculty offices and classrooms, but also student residence halls (aka “the dorms”). And students are using what’s been built…” - Joe St Sauver, University of Oregon at NLANR/Internet2 Joint Techs Workshop, Jan 28, 2002

9 SAC Campus Bandwidth Management Solutions9 “you can’t go fast only on Internet2” (2) “Many sites may have noticed surprising traffic levels associated with dorm users of peer to peer applications such as Kazaa/ Morpheus/FastTrack, (running on 1214/ TCP) although Gnutella clients (such as BearShare/LimeWire) are also often seen. “Typically, when such applications begin to be used in an unshaped residence hall environment, they routinely use all or most of the available commodity bandwidth.” - Joe St Sauver, University of Oregon at NLANR/Internet2 Joint Techs Workshop, Jan 28, 2002

10 SAC Campus Bandwidth Management Solutions10 terminology i kept finding myself typing “dedicated traffic-shaping device,” or “specialized traffic-shaping device,” or “traffic-shaping appliance,”... instead, let me just say “shaper” to mean a network appliance dedicated to traffic-shaping, specialized for that purpose. also, traffic-shaping means any intentional effort to forward packets in a manner other than FIFO, as fast as they arrive. –including rate-limiting, filtering, QoS/CoS, …

11 SAC Campus Bandwidth Management Solutions11 where to manage campus bandwidth traffic-shape where the problem is: usually at or near campus border router. border router external networks (ISPs, GigaPoP, Abilene, etc.) RRR Campus network traffic-shape here or here

12 SAC Campus Bandwidth Management Solutions12 where to manage – outside border router border router external networks (ISPs, GigaPoP, Abilene, etc.) RRR Campus network pros: +uses shaper. +carries only external traffic; cannot interfere with internal traffic. cons: –uses shaper. –may require WAN interface (ATM, POS, T-3, T-1); less flexible and more expensive than ethernet interface.

13 SAC Campus Bandwidth Management Solutions13 where to manage – on border router border router external networks (ISPs, GigaPoP, Abilene, etc.) RRR Campus network pros: +does not require a shaper. +carries only external traffic; cannot interfere with internal traffic. cons: –router-based traffic-shaping generally more limited, less effective than dedicated traffic- shaping devices. –puts additional CPU load on router, possibly introducing performance loss. in a few minutes, i’m going to try to convince you that traffic-shaping on routers is not a good idea.

14 SAC Campus Bandwidth Management Solutions14 where to manage – inside border router border router external networks (ISPs, GigaPoP, Abilene, etc.) RRR Campus network pros: +uses a shaper. +may use more flexible and less expensive ethernet interfaces. +allows “partioning” of campus network, and the use of multiple shapers. cons: –uses a shaper. –carries some internal traffic: can interfere with internal traffic. –greater bandwidth on managed link (internal traffic), may requires more capable (more expensive) shaper.

15 SAC Campus Bandwidth Management Solutions15 the Internet2 problem Internet2 costs are usage-insensitive, and Internet2 was designed to “go fast”…  we don’t want to rate-limit Internet2 traffic. many campuses get Internet service from their gigaPoP, often over a single circuit. i.e.: usage-sensitive and usage-insensitive traffic share a single link.  difficult to manage just one and not the other when they’re commingled on a single link. “you can’t go fast only on Internet2”

16 SAC Campus Bandwidth Management Solutions16 Berkeley traffic-shaping, version 1 our initial attempts at rate-limiting were against the residence halls: –approx Mbps ethernet drops (two per room); –100 Mbps routed backbone; –in Fall 1999, ResHalls’ Internet bandwidth soared from Mbps to Mbps, 24x7; –not centrally funded, ResHalls must pay for network service, including Internet service. good environment for testing new technologies –single 100 Mbps point of connection to rest-of-campus (RoC) network; –very simple policy: rate-limit all traffic to 15 Mbps (to start), each direction.

17 SAC Campus Bandwidth Management Solutions17 Berkeley traffic-shaping, version 1 (2) analyzed one week’s worth of netflow records to identify the 2400 most heavily used (by bandwidth) Internet2 routes. –at that time, packetshaper supported max 5000 classes. –these accounted for 99.2% of Internet2 traffic during the week (Jan 2000). created {in,out}bound classes based on those prefixes, plus campus routes, with no rate-limit; applied 15 Mbps rate-limit to remaining traffic. –campus traffic and approx 99.2% of Internet2 traffic was excluded from the rate limit.

18 SAC Campus Bandwidth Management Solutions18 Berkeley traffic-shaping, version 1 (3) it worked, but… –never found time to keep the exempt classes up to date; –the number of Internet2 routes increased over time; –so did the numbers of other rate-insensitive prefixes. so, class definitions based on usage-insensitive destinations didn’t scale well; and, we still had to worry about data to/from the rest of campus ( R o C ). btw, most ResHall traffic was to/from the commodity Internet; performance was bad and the residents let us know about it.

19 SAC Campus Bandwidth Management Solutions19 Berkeley traffic-shaping, version 2 most of the UC campuses needed to manage ISP bandwidth to contain costs, so we came up with a design to deliver traffic to/from ISP over a separate circuit from our Internet2 traffic. connected the ISP circuit to a second border router and engineered the routing so that only traffic to/from ISP passed through the shapers. added a second shaper for R o C-ISP traffic. –at that time, the combined needs of RoC (60 Mbps) and reshalls (30 Mbps) practically exceeded capacity of current PacketShapers.

20 SAC Campus Bandwidth Management Solutions20 UCB border topology RRR inr-000 CalREN2, Internet2, & commodity peerings (rate-independent costs) Commodity transit service (ISP) (rate-dependent costs) reshall shaper RRR i nr- 201i nr- 202 inr-new-666 RoC shaper Rest-of-campus (RoC) network ResHall network EBGP IBGP peers i nr- 205 EBGP border routers

21 SAC Campus Bandwidth Management Solutions21 UCB: inward route announcements RRR inr-000 CalREN2, Internet2, & commodity peerings (rate-independent costs) Commodity transit service (ISP) (rate-dependent costs) reshall shaper RRR i nr- 201i nr- 202 inr-new-666 RoC shaper Rest-of-campus (RoC) network ResHall network i nr- 205 default route I2 routes

22 SAC Campus Bandwidth Management Solutions22 RRR inr-000 CalREN2, Internet2, & commodity peerings (rate-independent costs) Commodity transit service (ISP) (rate-dependent costs) reshall shaper RRR i nr- 201i nr- 202 inr-new-666 RoC shaper Rest-of-campus (RoC) network ResHall network i nr- 205 UCB: outbound data paths

23 SAC Campus Bandwidth Management Solutions23 UCB: outward route announcements RRR inr-000 CalREN2, Internet2, & commodity peerings (rate-independent costs) Commodity transit service (ISP) (rate-dependent costs) reshall shaper RRR i nr- 201i nr- 202 inr-new-666 RoC shaper Rest-of-campus (RoC) network ResHall network i nr- 205 RoC routes reshall routes RoC+reshall routes reshall routes

24 SAC Campus Bandwidth Management Solutions24 RRR inr-000 CalREN2, Internet2, & commodity peerings (rate-independent costs) Commodity transit service (ISP) (rate-dependent costs) reshall shaper RRR i nr- 201i nr- 202 inr-new-666 RoC shaper Rest-of-campus (RoC) network ResHall network i nr- 205 UCB: inbound data paths

25 SAC Campus Bandwidth Management Solutions25 RoC traffic oct 2001-jul 2002 solid graph is data from campus to ISP line graph is data to campus from ISP one day averages before March 1: rate-limited at 70 Mbps, including March 1-June 1: rate-limited at 90 Mbps, with 20Mbps guaranteed to server after June 1: rate limited at 70 Mbps, without

26 SAC Campus Bandwidth Management Solutions26 RoC traffic 5-15 jul 2002 solid graph is data from campus to ISP line graph is data to campus from ISP 30 minute averages rate-limited at 70 Mbps (raised to 90 15jul2002)

27 SAC Campus Bandwidth Management Solutions27 ResHall traffic oct 2001-jul 2002 solid graph is data from campus to ISP line graph is data to campus from ISP one day averages rate-limited at 40 Mbps (likely to be increased for Fall semester)

28 SAC Campus Bandwidth Management Solutions28 ResHall traffic jul 2002 solid graph is data from campus to ISP line graph is data to campus from ISP 30 minute averages rate-limited at 40 Mbps

29 SAC Campus Bandwidth Management Solutions29 total ISP traffic oct 2001-jul 2002 line graph is data from campus to ISP solid graph is data to campus from ISP one day averages

30 SAC Campus Bandwidth Management Solutions30 total ISP traffic oct 2001-jul 2002 CalREN2 routing broken CalREN2 routing fixed Christmas break Spring break (prim. Reshalls) Reshalls decant for summer starts using different ISP

31 SAC Campus Bandwidth Management Solutions31 traffic-shaping at Berkeley today all ISP traffic is shaped (mostly, just rate-limited). only ISP traffic is shaped; other traffic doesn’t even go through the PacketShapers. costs: –PacketShapers, and staff-time to manage them. –additional routers, and staff-time to manage them. –routing complexity. –even greater complexity inside CalREN-2 (gigaPoP). –separation of ISP traffic and usage-insensitive traffic must be carried throughout the gigaPoP. –routing is very complex and stuff happens (routing anomalies). –the CalREN-2 NOC is very good, still, i think we tested their tolerance for network over-engineering.

32 SAC Campus Bandwidth Management Solutions32 under the hood: underlying mechanisms token bucket queuing TCP manipulation flows

33 SAC Campus Bandwidth Management Solutions33 token bucket one of the oldest rate-limiting techniques. –tokens are put into the “bucket” at a fixed rate, which equals the desired average bandwidth. –a packet is forwarded if the bucket contains a sufficient number of tokens: one token for every byte in the packet. those tokens are then removed from the bucket. –if the bucket contains an insufficient number of tokens, the packet is discarded. –the “size” of the bucket determines the maximum burst size. discarding packets is pretty harmful for some applications: audio, video, telnet, ssh

34 SAC Campus Bandwidth Management Solutions34 token bucket (2) discarding packets is pretty harmful for some applications: –audio, video, telnet, ssh token bucket drops packets indiscriminantly. –modern shapers strive to not drop packets, –and when they must drop packets, to do it with some intelligence. Cisco’s CAR is a token bucket mechanism. –some folks did some research to see how to set the CAR parameters to minimize the pain; scales only to about 12 Mbps; above that, the parameters can’t be set correctly. –

35 SAC Campus Bandwidth Management Solutions35 queuing on ingress, packets are placed into one or more queues. queue(s) are serviced according to the desired traffic-shaping characteristics for each queue. –“servicing queues” means removing packets from the queues, sending them out the egress interface.

36 SAC Campus Bandwidth Management Solutions36 queuing (2) introduce delay in order to limit bandwidth: if delivering the next packet would cause the bandwidth to go too high, simply leave the packet on the queue for a short period. –while a packet is being transmitted, instantaneous bandwidth is 100% of link speed. –while no packet is being transmitted, the instantaneous bandwidth is 0. –by increasing the amount of time when no packet is being transmitted, the average bandwidth is reduced.

37 SAC Campus Bandwidth Management Solutions37 queuing (3) problem: if there’s too much demand, queues will fill and subsequent packets will be dropped rather than queued. –dropped packets generally will be retransmitted, which ironically means increased bandwidth on the ingress link. –users can detect packet loss, and will perceive it as a network problem. –this is why most shapers have lots of RAM: they need large buffers to avoid queue overflow. –some shaper’s will temporarily exceed the configured maximum bandwidth to avoid packet loss.

38 SAC Campus Bandwidth Management Solutions38 queuing (4) problem: holding packets on the queue increases the latency on the link. –users can detect latency (ping, traceroute, telnet, ssh) and will perceive it as a network problem. –increased latency leads to longer-lived sessions (e.g. , ftp, http…); servers will need to hold sessions open for longer periods, and will refuse new connections if the number of session reaches the maximum number allowed. at the same time, the efficiency of the server is reduced, and other problems can occur, e.g. disk space exhaustion.

39 SAC Campus Bandwidth Management Solutions39 queuing (5) problem: holding packets on the queue can introduce jitter: variation in the latency for packets in a single stream. –very painful for keystroke-driven applications such as telnet, ssh. –can dramatically reduce quality of audio and video sessions.

40 SAC Campus Bandwidth Management Solutions40 queuing (6) packets can placed onto different queues, with different servicing rules. –different priorities: service queue A first, service queue B only if A is empty, service queue C only if A and B are empty, … –low priority queues can be “starved.” –must be careful not to break routing, DNS, etc. by accidentally giving these packets a low priority! –different bandwidth limits for the different queues, e.g. assign peer-to-peer applications to a queue with a low bandwidth limit. –preserves other bandwidth for “more worthy” applications.

41 SAC Campus Bandwidth Management Solutions41 TCP manipulation TCP uses a connection paradigm, with a “sliding window” mechanism that enables the sender to increase bandwidth if there is capacity available or decrease bandwidth if there is congestion. this mechanism can be manipulated to regulate the bandwidth used by the sender. –TCP connections are always bi-directional – both end- hosts can send and receive data – however, most connections have one end acting as sender and the other end as receiver. –sender sends data segments and the receiver ACK s data segments as they are received. –several things determine how many segments can be “in-flight” (sent but not ACK ed) at any moment.

42 SAC Campus Bandwidth Management Solutions42 TCP manipulation (2) RWIN – receiver window size, tells sender how much data the receiver is able to receive. –shaper can rewrite RWIN field in ACKs from receiver: reducing the RWIN value will cause the sender to slow down, since there can be fewer segments in-flight. delayed ACK s – cause the sender to over-estimate the number of in-flight segments; sender will slow down to let receiver to “catch up.” –also, sender uses ACK s to estimate round-trip time ( RTT ), and retransmits lost packets based on RTT. shaper can manipulate RTT to prevent the sender from undesirable retransmissions.

43 SAC Campus Bandwidth Management Solutions43 TCP manipulation (3) works only with TCP. –applying non-TCP packets is likely to have unintended effects. –works best with relatively long-lived, bursty TCP connections. TCP manipulation can break some applications. –e.g. IPSEC VPNs –don’t understand exactly how/why this is breaking, so finding other sensitive applications is trial-and-error.

44 SAC Campus Bandwidth Management Solutions44 flows a flow is a sequence of packets with the same (src IP, dst IP, protocol, src port, dst port) info. a flow starts when the first packet is seen, and ends after a period during which no packet is seen. a flow is uni-directional. flow-based mechanisms provide finer-grained control, can prevent some of the problems described previously. –in effect, flow-based mechanisms act as though each flow has it’s own queue. –but flow setup and teardown can burden an under- powered shaper.

45 SAC Campus Bandwidth Management Solutions45 how are these mechanisms used for traffic-shaping? classification partitions policies

46 SAC Campus Bandwidth Management Solutions46 classification specify classes (of packets) to which traffic- shaping mechanisms will be applied. packet attributes on which classes can be specified include: –source and/or destination IP address or address range (i.e. network prefixes); –can be used to providing different service levels to different campus groups (departments, workgroups, research units…) –protocol: TCP or UDP or ICMP or other –TCP or UDP ports –need to take care to avoid mismanaging traffic. E.g. KaZaA uses port 1214/TCP; we might severely constrain traffic using 1214/TCP. however, when a client initiates a connection to a server, the client chooses a random port > 1024, the server uses a well-known port < so, 1214/TCP might be used by a client contacting a web server (80/TCP).

47 SAC Campus Bandwidth Management Solutions47 classification (2) –application content in packet data –shapers can look deeper than the IP/TCP/UDP/ICMP headers, looking for specific content that identifies the application. –avoids the misidentification problem of using the TCP or UDP port; defeats “port hopping” applications. –ToS bits in IP header (CoS/QoS) –802.1Q VLAN label –MPLS VPN label class criteria can be used in combination. classes can be grouped to form a larger class. tag info, that would be set elsewhere, on another device.

48 SAC Campus Bandwidth Management Solutions48 partitions fractional allocations of available bandwidth 30 Mbps 15 Mbps 100 Mbps 75 Mbps –drawing shows a 100 Mbps link, rate-limited at 75 Mbps total bandwidth, using 4 partitions: –75 Mbps partition for all traffic, enclosing –30 Mbps partition for class A traffic –30 Mbps partition for class B traffic –15 Mbps partition for class C traffic

49 SAC Campus Bandwidth Management Solutions49 partitions (2) partitions manage the aggregate bandwidth of all flows in a class (or group of classes); all the flows in the class are managed as a group. partitions can be nested (hierarchical). are either fixed or burstable: –fixed partitions don’t change size; may lead to unused available bandwidth. –burstable partitions can “grow” to use available bandwidth; specify minimum partition size (guaranteed bandwidth) and maximum partition size (burst size).

50 SAC Campus Bandwidth Management Solutions50 partitions (3) caution: complicated schemes can have unintended effects. example: –intent: rate-limit total campus traffic at 70 Mbps, including with the additional constraint that not go above 30 Mbps. –configuration: 70 Mbps partition for all traffic, containing 0-30 (min-max) partition for –result: got preferred access to bandwidth, up to 30 Mbps. other traffic was effectively constrained to 40 Mbps.

51 SAC Campus Bandwidth Management Solutions51 policies manage bandwidth utilization of individual flows, rather than aggregate bandwidth. policies can be applied to classes with partitions, providing management of individual flows within the aggregate bandwidth management of the enclosing partition. two kinds of policies perform bandwidth management: priority policies and rate policies. three other kinds of policies affect bandwidth only indirectly: discard policies, no admission policies, and ignore policies.

52 SAC Campus Bandwidth Management Solutions52 priority policies establish priority for flows without specifying any particular bandwidth. manage competion for limited bandwidth: packets with higher priority get sent before packets with lower priority. –when bandwidth is scarce, lower priority packets will be held on queues longer, will get less bandwidth. –packets could get held for very long times, like forever; need to be careful to avoid “starving” applications. –e.g.: class A has higher priority than class B; and class A wants all the available bandwidth. packets in class B will not get sent. applications falling into class B are broken. –we’ve observed latencies of >1 sec for default priority packets, when demand has been particularly high.

53 SAC Campus Bandwidth Management Solutions53 rate policies manage bandwidth of individual flows. –minimum (guaranteed) bandwidth per flow (can be 0) –fixed or burstable with configurable priority –optional maximum bandwidth per flow. –for TCP, PacketShaper uses “Rate Control” (TCP manipulation). –for UDP, PacketShaper uses bounded-delay scheduling. –in order to guarantee bandwidth to individual flows, need to have some idea about the maximum number of concurrent flows; if total guaranteed bandwidth exceeds available bandwidth, flows will be squeezed to < 1Kbps.

54 SAC Campus Bandwidth Management Solutions54 Berkeley PacketShaper configuration

55 SAC Campus Bandwidth Management Solutions55 general advice keep it simple: start with just a few partitions and non-default policies. make changes slowly and methodically. –conserves packetshaper resources for other matters, such as passive analysis of “interesting” patterns; –very simple to explain to campus users; –avoids concerns about deciding which packets are “better” than others (does your campus have a policy that addresses this issue?); –minimizes unexpected/undesired side effects from packetshaper activities. avoid giving students reasons for port hopping, tunneling, and other tricks to get around traffic- shaping.

56 SAC Campus Bandwidth Management Solutions56 other PacketShaper features traffic discovery – enabling traffic discovery on a class causes the PacketShaper to create sub-classes for applications as they appear. a very easy way to see what’s going on within a class. top talkers/top listeners – enabling one or both of these on a class will cause the PacketShaper to collect a list of the top senders/receivers withing a class. a very easy way to see what hosts are contributing to bandwidth utilization. statistics collection and reporting. pre-defined reports give easy views of the “health” of the managed link; for more complicated reports, statistics can be exported into an Excel spreadsheet. –limited statistics collection in the ISP models that Berkeley purchased.

57 SAC Campus Bandwidth Management Solutions57 why not traffic-shape on router(s)? routers are designed to forward packets as fast they can, usually via an optimized forwarding path in the router. adding complexity to the forwarding decision can take packets out of that optimized path, causing: –increased packet handling time (latency); –increased processor activity (which can rob cycles with other processor intensive activities, e.g. routing protocols, forwarding table updates, monitoring and management, …) routers cannot offer nearly as much control and specification (configuration complexity) as shapers, and cannot monitor as well either.

58 SAC Campus Bandwidth Management Solutions58 why not traffic-shape on router(s)? (2) routers cannot support the sophisticated shaping mechanisms for which shapers are designed. –in particular, Cisco’s CAR controls the bandwidth in/out of an interface by dropping packets, but that can be painful for users. after all, we generally interpret packet loss as a sign of network problems. routers generally will classify packets according to packet header data (src IP, dst IP, TCP /port, UDP /port, ICMP ), whereas shapers can look farther into the packet for classification. –routers can be fooled by “port-hopping” applications.

59 SAC Campus Bandwidth Management Solutions59 what about using QoS? traffic-shaping is “using QoS”; shapers are QoS appliances. router-based QoS is accomplished by classifying packets, putting them into different queues, and servicing the queues according to different priorities, different rules. –if the output link is not fully utilized, then there’s enough capacity to service all the queues; i.e. this cannot provide rate-limiting, only different latencies. –in other words, router-based QoS will not effectively limit bandwidth at a rate lower than the capacity of the link. –still, there may be a role for router-based QoS as part of an overall traffic-shaping plan.

60 SAC Campus Bandwidth Management Solutions60 “scavenger” service QBSS ( QB one S cavenger S ervice) is a “less than best effort” service defined within the diffserv framework. –packets tagged with QBSS codepoint are queued separately, and the QBSS queue is serviced only after best effort queues are serviced; except that one QBSS packet is allowed through every once in awhile to keep TCP sessions alive. –intent is to enable applications to tag packets for delivery at times when link is not fully utilized. implementing QBSS on routers and on shapers is probably a good thing.

61 SAC Campus Bandwidth Management Solutions61 why not charge users? instead of traffic-shaping/rate-limiting, why not charge users for their share of traffic? pros: +avoids all the problems with traffic-shaping. +scales with demand, capitalizes bandwidth growth. –until a larger circuit is needed. +incentive for users to manage their bandwidth usage. cons: –costs of measurement, billing, and collection. –there are no tools for users to monitor and control their bandwidth usage. –disincentive to using the network.

62 SAC Campus Bandwidth Management Solutions62 netflow NetFlow was originally implemented by Cisco as a mechanism for fast forwarding packets in routers with minimal processor activity. subsequently added the ability to collect flow information and send it a collector on an external device (usually a unix host). there are multiple formats for the exported records; version 5 (most detailed) includes –src IP, dst IP, protocol, src port, dst port, flow start and end times, # bytes, #packets, ingress interface, egress interface. Juniper, Foundry, others have similar facilities.

63 SAC Campus Bandwidth Management Solutions63 netflow (2) utilities, both commercial and non-commercial, are available for post-processing the flow export records. –most campuses that i’ve talked to use the free flow- tools package developed by Mark Fullmer and others. – non-commercial packages are geared toward technical folk: unix utilities, designed to be used in scripts or from the command line; good for producing regularly run reports as well as on-the- fly investigations.

64 SAC Campus Bandwidth Management Solutions64 netflow (3) e.g.: daily report showing top TCP/UDP ports at UCB, inbound and outbound. Date: Fri, 2 Aug :08: (PDT) From: To: Subject: Campus port usage stats port Gigabytes MPackets KFlows http (26.10%) (27.02%) (43.40%) kazaa (16.89%) (13.19%) (7.13%) nntp (8.47%) (8.53%) (0.25%) gnutella (5.91%) (8.74%) (1.32%) eDonkey (4.80%) (4.30%) (1.57%) ftp-data (2.29%) (1.52%) (0.10%) napster (2.27%) (2.83%) (0.12%) GateCrasher (1.91%) (1.97%) (0.04%) (1.62%) (1.01%) 2.36 (0.00%) rtsp 8.32 (1.11%) (1.35%) (0.21%) https 6.70 (0.90%) (1.44%) (3.00%) smtp 5.19 (0.70%) (1.09%) (1.89%) (0.64%) 7.85 (0.61%) (0.10%) (0.60%) 5.84 (0.45%) 1.57 (0.00%) blackjack 4.08 (0.55%) 7.50 (0.58%) (0.23%) domain 3.94 (0.53%) (2.49%) (23.08%) (0.42%) 3.82 (0.30%) 0.83 (0.00%)...

65 SAC Campus Bandwidth Management Solutions65 netflow (4) e.g.: daily report showing top kazaa users at UCB, outbound.

66 SAC Campus Bandwidth Management Solutions66 netflow (5) it’s useful to keep export records around for a few months, but this starts to use up significant disk space. –Berkeley’s export records consume about 1 GB/day, compressed. –we currently have a 30 GB filesystem for storing the export records, holds just under 1 month’s data. –we are putting together a 30TB RAID array using over-the-counter components. we have some concerns about the possibility that these could be subpoenaed. there are also privacy concerns.

67 SAC Campus Bandwidth Management Solutions67 cricket, mrtg use SNMP to collect interface counters from routers, maintain longitudinal usage data, graph over time. can be configured to work with measurements other than bytes/sec: packets/sec, errors/sec, memory usage, CPU usage –packets/sec particularly useful for detecting and tracking down DoS attacks.

68 SAC Campus Bandwidth Management Solutions68 the view from -6 feet: true examples of traffic management activities case study #1: experiences. case study #2: excessive bandwidth use by residents of a particular housing unit that is not part of the UCB Residence Halls network. case study #3: interactive game using ftp semantics, being served from a compromised host.

69 SAC Campus Bandwidth Management Solutions69 case study #1 traffic is very asymmetric: –client sends tiny (<100B) request for a workunit (WU); server returns 350KB WU. client analyzes WU, later sends a small report (< 1KB). –100,000’s of clients all over the world, so server sends Mbps, 24x7. solid graph is data from line graph is data to one day averages Oct Jan 2002

70 SAC Campus Bandwidth Management Solutions70 case study #1 (2) in January 2002, latency across the RoC PacketShaper, normally 1-2 ms, was ms during busy times. –folks around campus complained. folks did not complain; they were not experiencing the large latencies. –had previously applied a 0-30 Mbps burstable partition on to “cap” spikes from the server. –this was giving preferential access to bandwidth, inside the 70 Mbps RoC partition, even though they had the default priority (3). no reason to expect this behavior. 7:00PM 23 Jan 2002: removed the 0-30 Mbps partition. that same night, the server went offline at midnight; came back online at 7:30AM 24 Jan –uncapped, the server spiked to Mbps, causing ms latencies through the PacketShaper.

71 SAC Campus Bandwidth Management Solutions71 case study #1 (3) SETI folks suggested giving their traffic a lower priority; so we gave them priority(2). –seemed to work: they were getting 7-10 Mbps during the busy times of the day, and 35+ in the early morning hours. –but, during the busy times, WU downloads were taking a lot longer, and the server was keeping connections open much longer. this resulted in periods when the server had it’s maximum number of sessions open, and would reject any new attempts to connect. users started complaining, threatening to stop running the client. SETI folks agreed to pay for 20 Mbps guaranteed bandwidth, and the ability to burst when there was available bandwidth. –configured a rate policy with 12.5Kbps guaranteed to each flow, burstable at priority(2), and with a Mbps burstable partition for the class aggregate. (server allows maximum of 1600 sessions; 20M bps / 1600 = 12.5Kbps) – traffic got priority(3) for each flow, with a Mbps burstable partition.

72 SAC Campus Bandwidth Management Solutions72 case study #1 (4) June 1, 2002: switched to their own ISP: –100 Mbps Internet service from Cogent; –half of a 100 Mbps circuit from campus to PAIX, purchased from campus; –using Cogent-provided IP address space; –not a campus address; no access to Internet2 or campus Internet service. –yet another border router. they’re getting more bandwidth for their $$$, but it’s lower quality: higher latency, more packet loss.

73 SAC Campus Bandwidth Management Solutions73 case study #2 remember this report showing top kazaa users? –highlighted lines show users in a particular housing unit.

74 SAC Campus Bandwidth Management Solutions74 this unit is on multiple subnets so the total bandwidth being used wasn’t apparent in standard reports and graphs. created a PacketShaper class to match all subnets, found they were using Mbps outbound! turned on traffic discovery, found that eDonkey, Gnutella, and Kazaa accounted for nearly all the bandwidth. Berkeley policy does not permit rate-limiting P2P; however, units are not allowed to gobble excessive amounts of bandwidth. senior management requested an 8Mbps limit. case study #2 (2)

75 SAC Campus Bandwidth Management Solutions75 case study #2 (3) applied a 0-8Mbps burstable partition:

76 SAC Campus Bandwidth Management Solutions76 case study #2 (4) they’re still doing lots of P2P, but at least it’s not adversely affecting the rest of campus (as much)!

77 SAC Campus Bandwidth Management Solutions77 case study #3 ftp traffic is generally not high in the campus port usage report, but i noticed that the PacketShaper was reporting abnormally high levels. …

78 SAC Campus Bandwidth Management Solutions78 case study #3 (2) “top-talkers” was already enabled, but had been collecting statistics for about 3 weeks. reset “top-talkers” and waited about 15 minutes.

79 SAC Campus Bandwidth Management Solutions79 case study #3 (3) examined recent netflow exports for the top talker: what application uses 27015/TCP and 64998/UDP?

80 SAC Campus Bandwidth Management Solutions80 case study #3 (4)

81 SAC Campus Bandwidth Management Solutions81 case study #3 (5) Berkeley has no policy against game-playing over the net, but it seemed strange to see this in this particular department. alerted the departmental security contact (informally, since there was not really any incident observed); he found that the host had been compromised and the game installed by the attacker. the game has been removed, and system security tightened up (hopefully).

82 SAC Campus Bandwidth Management Solutions82 can we block or limit peer-to-peer traffic? technically, yes. shapers have the ability to do this. politically? what does your campus’s policy say? –Berkeley follows the UC system Electronics Communication Policy (ECP) for determining appropriate use. –ECP does not distinguish between personal and professional use for faculty and students, so the fact that P2P applications are used primarily for entertainment is inconsequential. –ECP does distinguish for staff however, entertainment is considered incidental use, and incidental use is cannot interfere with non-incidental use. staff have been instructed to not use P2P for entertainment.

83 SAC Campus Bandwidth Management Solutions83 can we block or limit peer-to-peer traffic? (2) what about copyright infringement? –if infringement is reported, campus moves to stop the activity –for any reasonable report, irrespective of whether it conforms to DCMA guidelines or not. –but copyright infringement is not a problem with the applications themselves, it’s a problem with how they are primarily used. –there are non-infringing uses for P2P applications, we don’t want to block or limit those. it seems to me that we have a technical solution for a sociological problem; we need a sociological solution. ethics as a second language?

84 SAC Campus Bandwidth Management Solutions84 can we block or limit peer-to-peer traffic? (3) for Berkeley, for now, the answer is no. we manage bandwidth solely for the purpose of cost containment, we do not engage in content policing on the shapers.

85 SAC Campus Bandwidth Management Solutions85 resources –O’Reilly Network web site devoted to P2P activities, news, discussions. –majordomo mailing list, devoted to using Packeteer PacketShapers at educational institutions. –to subscribe, send mail to with the following command in the body of your message: unsubscribe packeteer-edu –archived at “QoS Appliances: Disease or Cure” –panel discussion at NLANR/Internet2 Joint Techs Workshop, Jan 28, –includes extensive list of traffic-shapers and QoS appliances. – ntations/ QoSpanel1_files/slide0001.htm

Download ppt "SAC 2002-08-07 Campus Bandwidth Management Solutions1 Campus Bandwidth Management Solutions ken lindahl UC Berkeley SAC 7 Aug."

Similar presentations

Ads by Google