1. Campus Bandwidth Management Solutions
ken lindahl UC Berkeley
SAC 7 Aug 2002
Copyright ken lindahl This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.
SAC Campus Bandwidth Management Solutions

2. where i’m coming from
technical lead for UC Berkeley’s external connectivity for the past 7+ years, including: CalREN-2 and Internet2, commodity Internet service and private peerings.
this talk is very much shaped by my experiences at Berkeley, probably 95%, with the other 5% coming from conversations and collaborations with network engineers from other academic institutions.
in particular, Berkeley uses Packeteer PacketShapers; all of my experience is with PacketShapers.
i think it’s important to keep that in mind as you listen to this talk.
SAC Campus Bandwidth Management Solutions

3. “solutions”
there’s no “one size fits all” solution to managing campus bandwidth.
in fact, it’s probably cleaner to say there’s no “solution,” just tools, techniques, strategies…
if you’re not sure whether something makes sense for your campus, please ask.
i’ll take questions or comments at any time.
SAC Campus Bandwidth Management Solutions

4. why manage campus bandwidth?
cost
generally, we’re talking about ISP costs, which typically are usage-sensitive.
Berkeley’s ISP traffic for Oct June (with rate-limiting)
cost: about $300 per Mbps (95th %-ile) per month
about $300,000 Jul 2001-Jun 2002
SAC Campus Bandwidth Management Solutions

5. why manage campus bandwidth ? (2)
cost
performance
i.e. congestion control/avoidance.
minimize negative effects of “less desirable” traffic (music, video, software sharing) on “more desirable” data (research, learning, business operations); i.e. “protect mission-critical applications.”
generally, we’re talking about ISP links, since it’s generally easier and more effective to add bandwidth on interior links.
and we’re also talking about cost, since if cost were not an issue, it would be easier and more effective to add bandwidth to ISP links.
SAC Campus Bandwidth Management Solutions

6. why manage campus bandwidth ? (3)
cost
performance
policy
some campuses have policies requiring some form of traffic shaping, e.g. blocking peer-to-peer applications because they are used primarily for exchange of copyrighted materials.
Berkeley does not have such a policy.
SAC Campus Bandwidth Management Solutions

7. why manage campus bandwidth ? (4)
cost
performance
policy
other reasons? (audience)
SAC Campus Bandwidth Management Solutions

8. “you can’t go fast only on Internet2”
“While the focus of I2 is properly on Abilene and related high performance networks initiatives, what one does to enable high network throughput for I2 purposes has an impact on commodity Internet transit traffic levels, too -- you can’t go fast only on Internet2.
“… Pretty much everybody has engineered and deployed fast campus networks that enable high throughput to Internet2 -- and to the commodity Internet. Those networks usually encompass not only faculty offices and classrooms, but also student residence halls (aka “the dorms”). And students are using what’s been built…”
- Joe St Sauver, University of Oregon
at NLANR/Internet2 Joint Techs Workshop, Jan 28, 2002
SAC Campus Bandwidth Management Solutions

9. “you can’t go fast only on Internet2” (2)
“Many sites may have noticed surprising traffic levels associated with dorm users of peer to peer applications such as Kazaa/ Morpheus/FastTrack, (running on 1214/ TCP) although Gnutella clients (such as BearShare/LimeWire) are also often seen.
“Typically, when such applications begin to be used in an unshaped residence hall environment, they routinely use all or most of the available commodity bandwidth.”
- Joe St Sauver, University of Oregon
at NLANR/Internet2 Joint Techs Workshop, Jan 28, 2002
SAC Campus Bandwidth Management Solutions

10. terminology
i kept finding myself typing “dedicated traffic-shaping device,” or “specialized traffic-shaping device,” or “traffic-shaping appliance,”...
instead, let me just say “shaper” to mean a network appliance dedicated to traffic-shaping, specialized for that purpose.
also, traffic-shaping means any intentional effort to forward packets in a manner other than FIFO, as fast as they arrive.
including rate-limiting, filtering, QoS/CoS, …
SAC Campus Bandwidth Management Solutions

11. where to manage campus bandwidth
traffic-shape where the problem is: usually at or near campus border router.
external networks
(ISPs, GigaPoP, Abilene, etc.)
traffic-shape here
or here
border router
or here R R R
Campus network
SAC Campus Bandwidth Management Solutions

12. where to manage – outside border router
pros:
uses shaper.
carries only external traffic; cannot interfere with internal traffic.
cons:
may require WAN interface (ATM, POS, T-3, T-1); less flexible and more expensive than ethernet interface.
external networks
(ISPs, GigaPoP, Abilene, etc.)
border router R R R
Campus network
SAC Campus Bandwidth Management Solutions

13. where to manage – on border router
pros:
does not require a shaper.
carries only external traffic; cannot interfere with internal traffic.
cons:
router-based traffic-shaping generally more limited, less effective than dedicated traffic-shaping devices.
puts additional CPU load on router, possibly introducing performance loss.
external networks
(ISPs, GigaPoP, Abilene, etc.)
border router R R R
Campus network
in a few minutes, i’m going to try to convince you that traffic-shaping on routers is not a good idea.
SAC Campus Bandwidth Management Solutions

14. where to manage – inside border router
pros:
uses a shaper.
may use more flexible and less expensive ethernet interfaces.
allows “partioning” of campus network, and the use of multiple shapers.
cons:
carries some internal traffic: can interfere with internal traffic.
greater bandwidth on managed link (internal traffic), may requires more capable (more expensive) shaper.
external networks
(ISPs, GigaPoP, Abilene, etc.)
border router R R R
Campus network
SAC Campus Bandwidth Management Solutions

15. the Internet2 problem
Internet2 costs are usage-insensitive, and Internet2 was designed to “go fast”…
we don’t want to rate-limit Internet2 traffic.
many campuses get Internet service from their gigaPoP, often over a single circuit. i.e.: usage-sensitive and usage-insensitive traffic share a single link.
difficult to manage just one and not the other when they’re commingled on a single link.
“you can’t go fast only on Internet2”
SAC Campus Bandwidth Management Solutions

16. Berkeley traffic-shaping, version 1
our initial attempts at rate-limiting were against the residence halls:
approx Mbps ethernet drops (two per room);
100 Mbps routed backbone;
in Fall 1999, ResHalls’ Internet bandwidth soared from Mbps to Mbps, 24x7;
not centrally funded, ResHalls must pay for network service, including Internet service.
good environment for testing new technologies
single 100 Mbps point of connection to rest-of-campus (RoC) network;
very simple policy: rate-limit all traffic to 15 Mbps (to start), each direction.
SAC Campus Bandwidth Management Solutions

17. Berkeley traffic-shaping, version 1 (2)
analyzed one week’s worth of netflow records to identify the 2400 most heavily used (by bandwidth) Internet2 routes.
at that time, packetshaper supported max 5000 classes.
these accounted for 99.2% of Internet2 traffic during the week (Jan 2000).
created {in,out}bound classes based on those prefixes, plus campus routes, with no rate-limit;
applied 15 Mbps rate-limit to remaining traffic.
campus traffic and approx 99.2% of Internet2 traffic was excluded from the rate limit.
SAC Campus Bandwidth Management Solutions

18. Berkeley traffic-shaping, version 1 (3)
it worked, but…
never found time to keep the exempt classes up to date;
the number of Internet2 routes increased over time;
so did the numbers of other rate-insensitive prefixes.
so, class definitions based on usage-insensitive destinations didn’t scale well;
and, we still had to worry about data to/from the rest of campus (RoC).
btw, most ResHall traffic was to/from the commodity Internet; performance was bad and the residents let us know about it.
SAC Campus Bandwidth Management Solutions

19. Berkeley traffic-shaping, version 2
most of the UC campuses needed to manage ISP bandwidth to contain costs, so we came up with a design to deliver traffic to/from ISP over a separate circuit from our Internet2 traffic.
connected the ISP circuit to a second border router and engineered the routing so that only traffic to/from ISP passed through the shapers.
added a second shaper for RoC-ISP traffic.
at that time, the combined needs of RoC (60 Mbps) and reshalls (30 Mbps) practically exceeded capacity of current PacketShapers.
SAC Campus Bandwidth Management Solutions

20. UCB border topology RoC shaper reshall shaper
CalREN2, Internet2,
& commodity peerings
(rate-independent costs)
Commodity transit
service (ISP)
(rate-dependent costs)
EBGP
EBGP
border routers
inr-000
inr-new-666
IBGP peers
RoC shaper
inr-201
inr-202
reshall
shaper R R R
inr-205
Rest-of-campus (RoC) network R R R
ResHall network
SAC Campus Bandwidth Management Solutions

21. UCB: inward route announcements
CalREN2, Internet2,
& commodity peerings
(rate-independent costs)
Commodity transit
service (ISP)
(rate-dependent costs)
inr-000
inr-new-666
I2 routes
default route
RoC shaper
inr-201
inr-202
default route
reshall
shaper
I2 routes R R R
inr-205
Rest-of-campus (RoC) network R R R
ResHall network
SAC Campus Bandwidth Management Solutions

22. UCB: outbound data paths
CalREN2, Internet2,
& commodity peerings
(rate-independent costs)
Commodity transit
service (ISP)
(rate-dependent costs)
inr-000
inr-new-666
RoC shaper
inr-201
inr-202
reshall
shaper R R R
inr-205
Rest-of-campus (RoC) network R R R
ResHall network
SAC Campus Bandwidth Management Solutions

23. UCB: outward route announcements
CalREN2, Internet2,
& commodity peerings
(rate-independent costs)
Commodity transit
service (ISP)
(rate-dependent costs)
inr-000
inr-new-666
RoC+reshall routes
RoC routes
RoC shaper
inr-201
inr-202
reshall routes
reshall
shaper
reshall routes R R R
inr-205
Rest-of-campus (RoC) network R R R
ResHall network
SAC Campus Bandwidth Management Solutions

24. UCB: inbound data paths
CalREN2, Internet2,
& commodity peerings
(rate-independent costs)
Commodity transit
service (ISP)
(rate-dependent costs)
inr-000
inr-new-666
RoC shaper
inr-201
inr-202
reshall
shaper R R R
inr-205
Rest-of-campus (RoC) network R R R
ResHall network
SAC Campus Bandwidth Management Solutions

25. RoC traffic oct 2001-jul 2002
solid graph is data from campus to ISP line graph is data to campus from ISP one day averages
before March 1: rate-limited at 70 Mbps, including
March 1-June 1: rate-limited at 90 Mbps, with 20Mbps guaranteed to server
after June 1: rate limited at 70 Mbps, without
SAC Campus Bandwidth Management Solutions

26. RoC traffic 5-15 jul 2002
solid graph is data from campus to ISP line graph is data to campus from ISP 30 minute averages
rate-limited at 70 Mbps (raised to 90 15jul2002)
SAC Campus Bandwidth Management Solutions

27. ResHall traffic oct 2001-jul 2002
solid graph is data from campus to ISP line graph is data to campus from ISP one day averages
rate-limited at 40 Mbps (likely to be increased for Fall semester)
SAC Campus Bandwidth Management Solutions

28. ResHall traffic jul 2002
solid graph is data from campus to ISP line graph is data to campus from ISP 30 minute averages
rate-limited at 40 Mbps
SAC Campus Bandwidth Management Solutions

29. total ISP traffic oct 2001-jul 2002
line graph is data from campus to ISP solid graph is data to campus from ISP one day averages
SAC Campus Bandwidth Management Solutions

30. total ISP traffic oct 2001-jul 2002
CalREN2 routing broken
CalREN2 routing fixed
Christmas break
Spring break (prim. Reshalls)
Reshalls decant for summer
starts using different ISP
SAC Campus Bandwidth Management Solutions

31. traffic-shaping at Berkeley today
all ISP traffic is shaped (mostly, just rate-limited).
only ISP traffic is shaped; other traffic doesn’t even go through the PacketShapers.
costs:
PacketShapers, and staff-time to manage them.
additional routers, and staff-time to manage them.
routing complexity.
even greater complexity inside CalREN-2 (gigaPoP).
separation of ISP traffic and usage-insensitive traffic must be carried throughout the gigaPoP.
routing is very complex and stuff happens (routing anomalies).
the CalREN-2 NOC is very good, still, i think we tested their tolerance for network over-engineering.
SAC Campus Bandwidth Management Solutions

32. under the hood: underlying mechanisms
token bucket
queuing
TCP manipulation
flows
SAC Campus Bandwidth Management Solutions

33. token bucket one of the oldest rate-limiting techniques.
tokens are put into the “bucket” at a fixed rate, which equals the desired average bandwidth.
a packet is forwarded if the bucket contains a sufficient number of tokens: one token for every byte in the packet. those tokens are then removed from the bucket.
if the bucket contains an insufficient number of tokens, the packet is discarded.
the “size” of the bucket determines the maximum burst size.
discarding packets is pretty harmful for some applications: audio, video, telnet, ssh
SAC Campus Bandwidth Management Solutions

34. token bucket (2)
discarding packets is pretty harmful for some applications:
audio, video, telnet, ssh
token bucket drops packets indiscriminantly.
modern shapers strive to not drop packets,
and when they must drop packets, to do it with some intelligence.
Cisco’s CAR is a token bucket mechanism.
some folks did some research to see how to set the CAR parameters to minimize the pain; scales only to about 12 Mbps; above that, the parameters can’t be set correctly.
SAC Campus Bandwidth Management Solutions

35. queuing on ingress, packets are placed into one or more queues.
queue(s) are serviced according to the desired traffic-shaping characteristics for each queue.
“servicing queues” means removing packets from the queues, sending them out the egress interface.
SAC Campus Bandwidth Management Solutions

36. queuing (2)
introduce delay in order to limit bandwidth: if delivering the next packet would cause the bandwidth to go too high, simply leave the packet on the queue for a short period.
while a packet is being transmitted, instantaneous bandwidth is 100% of link speed.
while no packet is being transmitted, the instantaneous bandwidth is 0.
by increasing the amount of time when no packet is being transmitted, the average bandwidth is reduced.
SAC Campus Bandwidth Management Solutions

37. queuing (3)
problem: if there’s too much demand, queues will fill and subsequent packets will be dropped rather than queued.
dropped packets generally will be retransmitted, which ironically means increased bandwidth on the ingress link.
users can detect packet loss, and will perceive it as a network problem.
this is why most shapers have lots of RAM: they need large buffers to avoid queue overflow.
some shaper’s will temporarily exceed the configured maximum bandwidth to avoid packet loss.
SAC Campus Bandwidth Management Solutions

38. queuing (4)
problem: holding packets on the queue increases the latency on the link.
users can detect latency (ping, traceroute, telnet, ssh) and will perceive it as a network problem.
increased latency leads to longer-lived sessions (e.g. , ftp, http…); servers will need to hold sessions open for longer periods, and will refuse new connections if the number of session reaches the maximum number allowed. at the same time, the efficiency of the server is reduced, and other problems can occur, e.g. disk space exhaustion.
SAC Campus Bandwidth Management Solutions

39. queuing (5)
problem: holding packets on the queue can introduce jitter: variation in the latency for packets in a single stream.
very painful for keystroke-driven applications such as telnet, ssh.
can dramatically reduce quality of audio and video sessions.
SAC Campus Bandwidth Management Solutions

40. queuing (6)
packets can placed onto different queues, with different servicing rules.
different priorities: service queue A first, service queue B only if A is empty, service queue C only if A and B are empty, …
low priority queues can be “starved.”
must be careful not to break routing, DNS, etc. by accidentally giving these packets a low priority!
different bandwidth limits for the different queues, e.g. assign peer-to-peer applications to a queue with a low bandwidth limit.
preserves other bandwidth for “more worthy” applications.
SAC Campus Bandwidth Management Solutions

41. TCP manipulation
TCP uses a connection paradigm, with a “sliding window” mechanism that enables the sender to increase bandwidth if there is capacity available or decrease bandwidth if there is congestion. this mechanism can be manipulated to regulate the bandwidth used by the sender.
TCP connections are always bi-directional – both end-hosts can send and receive data – however, most connections have one end acting as sender and the other end as receiver.
sender sends data segments and the receiver ACKs data segments as they are received.
several things determine how many segments can be “in-flight” (sent but not ACKed) at any moment.
SAC Campus Bandwidth Management Solutions

42. TCP manipulation (2)
RWIN – receiver window size, tells sender how much data the receiver is able to receive.
shaper can rewrite RWIN field in ACKs from receiver: reducing the RWIN value will cause the sender to slow down, since there can be fewer segments in-flight.
delayed ACKs – cause the sender to over-estimate the number of in-flight segments; sender will slow down to let receiver to “catch up.”
also, sender uses ACKs to estimate round-trip time (RTT), and retransmits lost packets based on RTT. shaper can manipulate RTT to prevent the sender from undesirable retransmissions.
SAC Campus Bandwidth Management Solutions

43. TCP manipulation can break some applications.
works only with TCP.
applying non-TCP packets is likely to have unintended effects.
works best with relatively long-lived, bursty TCP connections.
TCP manipulation can break some applications.
e.g. IPSEC VPNs
don’t understand exactly how/why this is breaking, so finding other sensitive applications is trial-and-error.
SAC Campus Bandwidth Management Solutions

44. flows
a flow is a sequence of packets with the same (src IP, dst IP, protocol, src port, dst port) info.
a flow starts when the first packet is seen, and ends after a period during which no packet is seen.
a flow is uni-directional.
flow-based mechanisms provide finer-grained control, can prevent some of the problems described previously.
in effect, flow-based mechanisms act as though each flow has it’s own queue.
but flow setup and teardown can burden an under-powered shaper.
SAC Campus Bandwidth Management Solutions

45. how are these mechanisms used for traffic-shaping?
classification
partitions
policies
SAC Campus Bandwidth Management Solutions

46. classification
specify classes (of packets) to which traffic-shaping mechanisms will be applied. packet attributes on which classes can be specified include:
source and/or destination IP address or address range (i.e. network prefixes);
can be used to providing different service levels to different campus groups (departments, workgroups, research units…)
protocol: TCP or UDP or ICMP or other
TCP or UDP ports
need to take care to avoid mismanaging traffic. E.g. KaZaA uses port 1214/TCP; we might severely constrain traffic using 1214/TCP. however, when a client initiates a connection to a server, the client chooses a random port > 1024, the server uses a well-known port < so, 1214/TCP might be used by a client contacting a web server (80/TCP).
SAC Campus Bandwidth Management Solutions

47. classification (2) class criteria can be used in combination.
application content in packet data
shapers can look deeper than the IP/TCP/UDP/ICMP headers, looking for specific content that identifies the application.
avoids the misidentification problem of using the TCP or UDP port; defeats “port hopping” applications.
ToS bits in IP header (CoS/QoS)
802.1Q VLAN label
MPLS VPN label
class criteria can be used in combination.
classes can be grouped to form a larger class.
tag info, that would be set elsewhere, on another device.
SAC Campus Bandwidth Management Solutions

48. partitions fractional allocations of available bandwidth
30 Mbps
15 Mbps
100 Mbps
75 Mbps
drawing shows a 100 Mbps link, rate-limited at 75 Mbps total bandwidth, using 4 partitions:
75 Mbps partition for all traffic, enclosing
30 Mbps partition for class A traffic
30 Mbps partition for class B traffic
15 Mbps partition for class C traffic
SAC Campus Bandwidth Management Solutions

49. partitions (2)
partitions manage the aggregate bandwidth of all flows in a class (or group of classes); all the flows in the class are managed as a group.
partitions can be nested (hierarchical).
are either fixed or burstable:
fixed partitions don’t change size; may lead to unused available bandwidth.
burstable partitions can “grow” to use available bandwidth; specify minimum partition size (guaranteed bandwidth) and maximum partition size (burst size).
SAC Campus Bandwidth Management Solutions

50. partitions (3)
caution: complicated schemes can have unintended effects.
example:
intent: rate-limit total campus traffic at 70 Mbps, including with the additional constraint that not go above 30 Mbps.
configuration: 70 Mbps partition for all traffic, containing 0-30 (min-max) partition for
result: got preferred access to bandwidth, up to 30 Mbps. other traffic was effectively constrained to 40 Mbps.
SAC Campus Bandwidth Management Solutions

51. policies
manage bandwidth utilization of individual flows, rather than aggregate bandwidth.
policies can be applied to classes with partitions, providing management of individual flows within the aggregate bandwidth management of the enclosing partition.
two kinds of policies perform bandwidth management: priority policies and rate policies.
three other kinds of policies affect bandwidth only indirectly: discard policies, no admission policies, and ignore policies.
SAC Campus Bandwidth Management Solutions

52. priority policies
establish priority for flows without specifying any particular bandwidth.
manage competion for limited bandwidth: packets with higher priority get sent before packets with lower priority.
when bandwidth is scarce, lower priority packets will be held on queues longer, will get less bandwidth.
packets could get held for very long times, like forever; need to be careful to avoid “starving” applications.
e.g.: class A has higher priority than class B; and class A wants all the available bandwidth. packets in class B will not get sent. applications falling into class B are broken.
we’ve observed latencies of >1 sec for default priority packets, when demand has been particularly high.
SAC Campus Bandwidth Management Solutions

53. rate policies manage bandwidth of individual flows.
minimum (guaranteed) bandwidth per flow (can be 0)
fixed or burstable with configurable priority
optional maximum bandwidth per flow.
for TCP, PacketShaper uses “Rate Control” (TCP manipulation).
for UDP, PacketShaper uses bounded-delay scheduling.
in order to guarantee bandwidth to individual flows, need to have some idea about the maximum number of concurrent flows; if total guaranteed bandwidth exceeds available bandwidth, flows will be squeezed to < 1Kbps.
SAC Campus Bandwidth Management Solutions

54. Berkeley PacketShaper configuration
SAC Campus Bandwidth Management Solutions

55. general advice
keep it simple: start with just a few partitions and non-default policies. make changes slowly and methodically.
conserves packetshaper resources for other matters, such as passive analysis of “interesting” patterns;
very simple to explain to campus users;
avoids concerns about deciding which packets are “better” than others (does your campus have a policy that addresses this issue?);
minimizes unexpected/undesired side effects from packetshaper activities.
avoid giving students reasons for port hopping, tunneling, and other tricks to get around traffic-shaping.
SAC Campus Bandwidth Management Solutions

56. other PacketShaper features
traffic discovery – enabling traffic discovery on a class causes the PacketShaper to create sub-classes for applications as they appear. a very easy way to see what’s going on within a class.
top talkers/top listeners – enabling one or both of these on a class will cause the PacketShaper to collect a list of the top senders/receivers withing a class. a very easy way to see what hosts are contributing to bandwidth utilization.
statistics collection and reporting. pre-defined reports give easy views of the “health” of the managed link; for more complicated reports, statistics can be exported into an Excel spreadsheet.
limited statistics collection in the ISP models that Berkeley purchased.
SAC Campus Bandwidth Management Solutions

57. why not traffic-shape on router(s)?
routers are designed to forward packets as fast they can, usually via an optimized forwarding path in the router.
adding complexity to the forwarding decision can take packets out of that optimized path, causing:
increased packet handling time (latency);
increased processor activity (which can rob cycles with other processor intensive activities, e.g. routing protocols, forwarding table updates, monitoring and management, …)
routers cannot offer nearly as much control and specification (configuration complexity) as shapers, and cannot monitor as well either.
SAC Campus Bandwidth Management Solutions

58. why not traffic-shape on router(s)? (2)
routers cannot support the sophisticated shaping mechanisms for which shapers are designed.
in particular, Cisco’s CAR controls the bandwidth in/out of an interface by dropping packets, but that can be painful for users. after all, we generally interpret packet loss as a sign of network problems.
routers generally will classify packets according to packet header data (src IP, dst IP, TCP/port, UDP/port, ICMP), whereas shapers can look farther into the packet for classification.
routers can be fooled by “port-hopping” applications.
SAC Campus Bandwidth Management Solutions

59. what about using QoS?
traffic-shaping is “using QoS”; shapers are QoS appliances.
router-based QoS is accomplished by classifying packets, putting them into different queues, and servicing the queues according to different priorities, different rules.
if the output link is not fully utilized, then there’s enough capacity to service all the queues; i.e. this cannot provide rate-limiting, only different latencies.
in other words, router-based QoS will not effectively limit bandwidth at a rate lower than the capacity of the link.
still, there may be a role for router-based QoS as part of an overall traffic-shaping plan.
SAC Campus Bandwidth Management Solutions

60. “scavenger” service
QBSS (QBone Scavenger Service) is a “less than best effort” service defined within the diffserv framework.
packets tagged with QBSS codepoint are queued separately, and the QBSS queue is serviced only after best effort queues are serviced; except that one QBSS packet is allowed through every once in awhile to keep TCP sessions alive.
intent is to enable applications to tag packets for delivery at times when link is not fully utilized.
implementing QBSS on routers and on shapers is probably a good thing.
SAC Campus Bandwidth Management Solutions

61. why not charge users?
instead of traffic-shaping/rate-limiting, why not charge users for their share of traffic?
pros:
avoids all the problems with traffic-shaping.
scales with demand, capitalizes bandwidth growth.
until a larger circuit is needed.
incentive for users to manage their bandwidth usage.
cons:
costs of measurement, billing, and collection.
there are no tools for users to monitor and control their bandwidth usage.
disincentive to using the network.
SAC Campus Bandwidth Management Solutions

62. netflow
NetFlow was originally implemented by Cisco as a mechanism for fast forwarding packets in routers with minimal processor activity.
subsequently added the ability to collect flow information and send it a collector on an external device (usually a unix host).
there are multiple formats for the exported records; version 5 (most detailed) includes
src IP, dst IP, protocol, src port, dst port, flow start and end times, # bytes, #packets, ingress interface, egress interface.
Juniper, Foundry, others have similar facilities.
SAC Campus Bandwidth Management Solutions

63. netflow (2)
utilities, both commercial and non-commercial, are available for post-processing the flow export records.
most campuses that i’ve talked to use the free flow-tools package developed by Mark Fullmer and others.
non-commercial packages are geared toward technical folk: unix utilities, designed to be used in scripts or from the command line; good for producing regularly run reports as well as on-the-fly investigations.
SAC Campus Bandwidth Management Solutions

64. netflow (3)
e.g.: daily report showing top TCP/UDP ports at UCB, inbound and outbound.
Date: Fri, 2 Aug :08: (PDT)
From:
To:
Subject: Campus port usage stats
port            Gigabytes       MPackets        KFlows
http             (26.10%)  (27.02%)  (43.40%)
kazaa            (16.89%)  (13.19%)  (7.13%)
nntp            63.19 (8.47%)    (8.53%)   (0.25%)
gnutella        44.12 (5.91%)    (8.74%)   (1.32%)
eDonkey         35.81 (4.80%)   55.64 (4.30%)    (1.57%)
ftp-data        17.07 (2.29%)   19.71 (1.52%)   50.69 (0.10%)
napster         16.95 (2.27%)   36.66 (2.83%)   62.10 (0.12%)
GateCrasher     14.22 (1.91%)   25.49 (1.97%)   20.51 (0.04%)
49999           12.06 (1.62%)   13.04 (1.01%)   2.36 (0.00%)
rtsp            8.32 (1.11%)    17.41 (1.35%)    (0.21%)
https           6.70 (0.90%)    18.68 (1.44%)    (3.00%)
smtp            5.19 (0.70%)    14.10 (1.09%)    (1.89%)
1026            4.75 (0.64%)    7.85 (0.61%)    52.27 (0.10%)
64997           4.48 (0.60%)    5.84 (0.45%)    1.57 (0.00%)
blackjack       4.08 (0.55%)    7.50 (0.58%)     (0.23%)
domain          3.94 (0.53%)    32.26 (2.49%)    (23.08%)
55851           3.17 (0.42%)    3.82 (0.30%)    0.83 (0.00%)
...
SAC Campus Bandwidth Management Solutions

65. netflow (4)
e.g.: daily report showing top kazaa users at UCB, outbound.
SAC Campus Bandwidth Management Solutions

66. netflow (5)
it’s useful to keep export records around for a few months, but this starts to use up significant disk space.
Berkeley’s export records consume about 1 GB/day, compressed.
we currently have a 30 GB filesystem for storing the export records, holds just under 1 month’s data.
we are putting together a 30TB RAID array using over-the-counter components.
we have some concerns about the possibility that these could be subpoenaed.
there are also privacy concerns.
SAC Campus Bandwidth Management Solutions

67. cricket, mrtg
use SNMP to collect interface counters from routers, maintain longitudinal usage data, graph over time.
can be configured to work with measurements other than bytes/sec: packets/sec, errors/sec, memory usage, CPU usage
packets/sec particularly useful for detecting and tracking down DoS attacks.
SAC Campus Bandwidth Management Solutions

68. the view from -6 feet: true examples of traffic management activities
case study #1: experiences.
case study #2: excessive bandwidth use by residents of a particular housing unit that is not part of the UCB Residence Halls network.
case study #3: interactive game using ftp semantics, being served from a compromised host.
SAC Campus Bandwidth Management Solutions

69. case study #1 SETI@home traffic is very asymmetric:
client sends tiny (<100B) request for a workunit (WU); server returns 350KB WU. client analyzes WU, later sends a small report (< 1KB).
100,000’s of clients all over the world, so server sends Mbps, 24x7.
Oct Jan 2002
solid graph is data from line graph is data to one day averages
SAC Campus Bandwidth Management Solutions

70. case study #1 (2)
in January 2002, latency across the RoC PacketShaper, normally 1-2 ms, was ms during busy times.
folks around campus complained.
folks did not complain; they were not experiencing the large latencies.
had previously applied a 0-30 Mbps burstable partition on to “cap” spikes from the server.
this was giving preferential access to bandwidth, inside the 70 Mbps RoC partition, even though they had the default priority (3). no reason to expect this behavior.
7:00PM 23 Jan 2002: removed the 0-30 Mbps partition.
that same night, the server went offline at midnight; came back online at 7:30AM 24 Jan 2002.
uncapped, the server spiked to Mbps, causing ms latencies through the PacketShaper.
SAC Campus Bandwidth Management Solutions

71. case study #1 (3)
SETI folks suggested giving their traffic a lower priority; so we gave them priority(2).
seemed to work: they were getting 7-10 Mbps during the busy times of the day, and 35+ in the early morning hours.
but, during the busy times, WU downloads were taking a lot longer, and the server was keeping connections open much longer. this resulted in periods when the server had it’s maximum number of sessions open, and would reject any new attempts to connect. users started complaining, threatening to stop running the client.
SETI folks agreed to pay for 20 Mbps guaranteed bandwidth, and the ability to burst when there was available bandwidth.
configured a rate policy with 12.5Kbps guaranteed to each flow, burstable at priority(2), and with a Mbps burstable partition for the class aggregate. (server allows maximum of 1600 sessions; 20M bps / 1600 = 12.5Kbps)
traffic got priority(3) for each flow, with a 0-50 Mbps burstable partition.
SAC Campus Bandwidth Management Solutions

72. case study #1 (4) June 1, 2002: SETI@home switched to their own ISP:
100 Mbps Internet service from Cogent;
half of a 100 Mbps circuit from campus to PAIX, purchased from campus;
using Cogent-provided IP address space;
not a campus address; no access to Internet2 or campus Internet service.
yet another border router.
they’re getting more bandwidth for their $$$, but it’s lower quality: higher latency, more packet loss.
SAC Campus Bandwidth Management Solutions

73. case study #2 remember this report showing top kazaa users?
highlighted lines show users in a particular housing unit.
SAC Campus Bandwidth Management Solutions

74. case study #2 (2)
this unit is on multiple subnets so the total bandwidth being used wasn’t apparent in standard reports and graphs.
created a PacketShaper class to match all subnets, found they were using Mbps outbound!
turned on traffic discovery, found that eDonkey, Gnutella, and Kazaa accounted for nearly all the bandwidth.
Berkeley policy does not permit rate-limiting P2P; however, units are not allowed to gobble excessive amounts of bandwidth.
senior management requested an 8Mbps limit.
SAC Campus Bandwidth Management Solutions

75. case study #2 (3) applied a 0-8Mbps burstable partition:
SAC Campus Bandwidth Management Solutions

76. case study #2 (4)
they’re still doing lots of P2P, but at least it’s not adversely affecting the rest of campus (as much)!
SAC Campus Bandwidth Management Solutions

77. case study #3
ftp traffic is generally not high in the campus port usage report, but i noticed that the PacketShaper was reporting abnormally high levels. …
SAC Campus Bandwidth Management Solutions

78. case study #3 (2)
“top-talkers” was already enabled, but had been collecting statistics for about 3 weeks.
reset “top-talkers” and waited about 15 minutes.
SAC Campus Bandwidth Management Solutions

79. case study #3 (3) examined recent netflow exports for the top talker:
what application uses 27015/TCP and 64998/UDP?
SAC Campus Bandwidth Management Solutions

80. case study #3 (4) http://www.portsdb.org/
SAC Campus Bandwidth Management Solutions

81. case study #3 (5)
Berkeley has no policy against game-playing over the net, but it seemed strange to see this in this particular department.
alerted the departmental security contact (informally, since there was not really any incident observed); he found that the host had been compromised and the game installed by the attacker.
the game has been removed, and system security tightened up (hopefully).
SAC Campus Bandwidth Management Solutions

82. can we block or limit peer-to-peer traffic?
technically, yes. shapers have the ability to do this.
politically? what does your campus’s policy say?
Berkeley follows the UC system Electronics Communication Policy (ECP) for determining appropriate use.
ECP does not distinguish between personal and professional use for faculty and students, so the fact that P2P applications are used primarily for entertainment is inconsequential.
ECP does distinguish for staff however, entertainment is considered incidental use, and incidental use is cannot interfere with non-incidental use. staff have been instructed to not use P2P for entertainment.
SAC Campus Bandwidth Management Solutions

83. can we block or limit peer-to-peer traffic? (2)
what about copyright infringement?
if infringement is reported, campus moves to stop the activity
for any reasonable report, irrespective of whether it conforms to DCMA guidelines or not.
but copyright infringement is not a problem with the applications themselves, it’s a problem with how they are primarily used.
there are non-infringing uses for P2P applications, we don’t want to block or limit those.
it seems to me that we have a technical solution for a sociological problem; we need a sociological solution.
ethics as a second language?
SAC Campus Bandwidth Management Solutions

84. can we block or limit peer-to-peer traffic? (3)
for Berkeley, for now, the answer is no.
we manage bandwidth solely for the purpose of cost containment, we do not engage in content policing on the shapers.
SAC Campus Bandwidth Management Solutions

85. resources http://www.openp2p.com/
O’Reilly Network web site devoted to P2P activities, news, discussions.
majordomo mailing list, devoted to using Packeteer PacketShapers at educational institutions.
to subscribe, send mail to with the following command in the body of your message: unsubscribe packeteer-edu
archived at
“QoS Appliances: Disease or Cure”
panel discussion at NLANR/Internet2 Joint Techs Workshop, Jan 28, 2002.
includes extensive list of traffic-shapers and QoS appliances.
SAC Campus Bandwidth Management Solutions