Presentation is loading. Please wait.

Presentation is loading. Please wait.

Successful Bandwidth Management at Carnegie Mellon Peter Hill & Kevin Miller Internet2 Joint Techs – August 2003.

Similar presentations


Presentation on theme: "Successful Bandwidth Management at Carnegie Mellon Peter Hill & Kevin Miller Internet2 Joint Techs – August 2003."— Presentation transcript:

1 Successful Bandwidth Management at Carnegie Mellon Peter Hill & Kevin Miller Internet2 Joint Techs – August 2003

2 2 About Us Responsible for campus backbone, Internet connection, 80% of data ports Internet connectivity (commodity & research) through Pittsburgh GigaPOP –Historically, net exporter of data to Internet 8000 students, 1300 faculty, 2000 staff 3:1 computer:person ratio –10,000+ registered Wireless cards

3 3 Nov 2001 100 Mbps No Bandwidth Management Timeline – Nov 2001 Demand rising faster than willingness to supply P2P file-swapping allowed users to easily create high volume servers Outbound Commodity Internet Traffic

4 4 No Bandwidth Management Outbound bandwidth hits limits of OC-3 ATM interface –High rate of packet loss from single cell drops –TCP retransmits cause egress meltdown Upgraded physical connection to GbE Demand continues rising

5 5 Nov 2001Mar 2002 100 Mbps110 Mbps No Bandwidth Management Bandwidth Limiting Needed Timeline – March 2002 GbE link to GigaPOP uses 802.1q trunk, separate Vlans for commodity vs. research traffic Router unable to rate-limit outbound traffic on single Vlan

6 6 Emergency Solution Complex, ‘delicate’ engineering to split commodity vs. research traffic from internal core to border router –Multiple OSPF processes –Research route redistribution to OSPF Applied rate limits to commodity traffic upon ingress to border router

7 7 Emergency Solution 75Mbps rate limit – single queue, tail drop only –Results weren’t pretty: high latency Out In

8 8 Nov 2001Mar 2002 Summer 2002 100 Mbps110 Mbps65 Mbps No Bandwidth Management Bandwidth Management Needed Tail Drop; Summer Recess Timeline – Summer 2002 Messy solution to a complicated problem No discrimination between legitimate and discretionary traffic Research traffic unaffected, though

9 9 Tail Drop Issues School resumes: hard limits hit immediately P2P consuming a significant amount of bandwidth Users reporting problems accessing email from home Interactive sessions suffer (SSH, Telnet)

10 10 High priority traffic? High priority traffic? Nine machines over.5% each; 21% total Nine machines over.5% each; 21% total (Un)fair Bandwidth Access

11 11 (Un)fair Bandwidth Access On the same day, 47% of all traffic was easily classifiable as P2P 18% of traffic was HTTP Other traffic: 35% –Believe ~28% is port- hopping P2P

12 12 Researching Solutions Middlebox Traffic Shapers –Compared Allot NetEnforcer vs. Packeteer Packetshaper Determined NetEnforcer better matched our requirements –Adds slight delays to constrain TCP –Fair bandwidth distribution among flows –Easier to configure policy –Better performance when demand equals limit

13 13 Raised Bandwidth Limit Nov 2002: Campus limit raised to 120Mbps

14 14 Nov 2001Mar 2002 Summer 2002 Jan 2003 100 Mbps110 Mbps65 Mbps120 Mbps No Bandwidth Management Bandwidth Management Needed Tail Drop; Summer Recess Traffic Shaping Timeline – January 2003 NetEnforcer deployed in October Policy developed in late fall Implemented policy in early January

15 15 NetEnforcer Policy Technical bandwidth policy using NetEnforcer –Used per-flow class-based fair bandwidth queuing P2P traffic, classified by port number High Priority Low Priority Network critical traffic Interactive (SSH, telnet); limited per-flow Traffic on well-known service ports (IMAP, HTTP) Non-classified traffic

16 16 NetEnforcer Policy Improved interactive performance Fewer complaints about accessing campus services remotely Traffic consistently at 120Mbps Winter Recess

17 17 Limits of Shaping Solution Per-host fair queuing not possible –User with 100 flows allowed 100 times bandwidth of user with one flow Poor SSH X forwarding performance High latency for UDP gaming services Demand still high – socially, nothing had changed

18 18 Tighter P2P Limits Software to classify P2P traffic at Layer 7 available in February 2003 –Added L-7 classification –Put absolute bandwidth cap on P2P

19 19 Nov 2001April 2002 Summer 2002 Jan 2003Feb 2003 100 Mbps110 Mbps65 Mbps120 Mbps110 Mbps No Bandwidth Management Bandwidth Management Needed Tail Drop; Summer Recess Traffic Shaping Traffic Shaping w/ L7 P2P Timeline – February 2003 Demand still close to limit Consider alternate approach: social engineering –Identifying machines using high bandwidth

20 20 Solution Add a targeted user education/dialogue component to strictly technical solutions –Blanket notices were/are ineffective –Why not before? Usage accounting software was in development Only anecdotal information on ‘top talkers’ Unsure of community reaction Didn’t know how easy it would be

21 21 Solution Created official Campus Bandwidth Usage Guidelines –Established a daily per-host bandwidth usage limit 1GB per day for wired machines (outbound to commodity Internet) 250MB per day for wireless (to any destination) –Published guidelines and requested comments from campus community –Began direct notification of hosts exceeding guidelines on February 24, 2003

22 22 Guideline Enforcement Accounting data generated –‘Argus’ utility captures raw packet data from egress span port –Post-processing aggregates flows by host, sums bandwidth usage –Nightly top-talkers report Hostname iMB oMB iKp oKp flows iMbs oMbs ips ops web4.andrew.cmu. 3456 107621 51917 83038 1948667 0.1 2.0 120 192

23 23 Guideline Enforcement Over-usage notification –Initial mail worded carefully, informed owner of quota and overage, and requested information on usage reasons Approximately a 0.25 FTE process Automated in July Disconnections handled like other abuse incidents

24 24 Positive Results Immediately noticed a decline in bandwidth utilization (within a few hours of first notifications) Applied Strict Layer-7 P2P Limits First Usage Notices Sent

25 25 Positive Results Very few negative responses –Some hosts granted exclusions (official web servers) Many notified were completely unaware of bandwidth usage –P2P programs left running in background –Compromised machines Trojan FTP servers serving French films Favorite responses –“i don't understand--what's bandwidth?? how do i reduce it?? what am i doing wrong?? what's going on???” –“i have no idea what bandwidth is or how one goes about using it too much.”

26 26 Positive Results 489Users have only received one message 102Additional users have two messages 29Hosts have had network access revoked 27Hosts marked as “exempt” from limits 39+Machines were found to be compomised Now: (Summer) Summary:

27 27 Nov 2001April 2002 Summer 2002 Jan 2003Feb 2003May 2003 100 Mbps110 Mbps65 Mbps120 Mbps110 Mbps60 Mbps No Bandwidth Management Bandwidth Management Needed Tail Drop; Summer Recess Traffic Shaping w/ L7 P2P Bandwidth Guidelines & Traffic Shaping Timeline – May 2003 With guideline enforcement, traffic drops by half NetEnforcer still has an impact on P2P – need to assess legitimate and discretionary P2P uses

28 28 Considerations Per-machine limits might create artificial commodity in addresses Role of Enterprise or Service Provider? –Packet shaping tends to apply Enterprise mindset – determine organization priorities –Bandwidth quotas use Service Provider mindset – how are quotas applied (port, machine, user?)

29 29 Questions? Argus: http://www.qosient.com/argushttp://www.qosient.com/argus Related links/resources: http://www.net.cmu.edu/pres/jt0803 http://www.net.cmu.edu/pres/jt0803 –Packet shaper evaluation –Argus accounting –Usage guidelines Peter Hill: peterjhill@cmu.edupeterjhill@cmu.edu Kevin Miller: kcm@cmu.edukcm@cmu.edu

30 30 NetEnforcer History DateAction Oct 3 2002Deployed NetEnforcer 701F Moved limits to NetEnforcer Campus & dormitory separate limits Jan 2003Implemented specific policies Port-based classification Services given higher priority P2P (Layer 4) given lower priority Feb 2003Added P2P application-layer classification Hard bandwidth limits for P2P August 2003Deploying NetEnforcer 802F Upgrade

31 31 Argus Probes


Download ppt "Successful Bandwidth Management at Carnegie Mellon Peter Hill & Kevin Miller Internet2 Joint Techs – August 2003."

Similar presentations


Ads by Google