1. Data Center Traffic and Measurements: Available Bandwidth Estimation Hakim Weatherspoon Assistant Professor, Dept of Computer Science CS 5413: High Performance Systems and Networking November 14, 2014 Slides from ACM SIGCOMM conference on Internet measurement (IMC), 2014, presentation of “MinProbe: Accurate, Minimum Overhead, Available Bandwidth Estimation in High Speed Wired Networks”

2. Overview and Basics Data Center Networks – Basic switching technologies – Data Center Network Topologies (today and Monday) – Software Routers (eg. Click, Routebricks, NetMap, Netslice) – Alternative Switching Technologies – Data Center Transport Data Center Software Networking – Software Defined networking (overview, control plane, data plane, NetFGPA) – Data Center Traffic and Measurements – Virtualizing Networks – Middleboxes Advanced Topics Where are we in the semester?

3. Goals for Today MinProbe: Accurate, Minimum Overhead, Available Bandwidth Estimation in High Speed Wired Networks – Ki Suh Lee, Erluo Li, ChiunLin Lim, Kevin Tang and Hakim Weatherspoon. In Proceedings of the 14th ACM SIGCOMM conference on Internet measurement (IMC), November 2014.

4. Available Bandwidth Estimation Basic building block – Network Protocol – Networked Systems – Distributed Systems 4 Which of the two paths has more available bandwidth? End-to-end: How to measure without access to anything in the network? How do I measure with minimum overhead?

5. Available Bandwidth Estimation 5 Passive Measurement – Polling counters: Port Stats or Flow Stats Active Measurement – Probe Packets: Packet Pair or Packet Train Probe Packets:

6. Narrow link: least capacity Tight link: least available bandwidth Active Measurement 6 Narrow link Network Capacity Cross Traffic Available Bandwidth Tight link

7. Estimate available bandwidth by saturating the tight link Active Measurement 7 Estimate Available Bandwidth? Saturate/Congest Tight Link Packet Queuing Delays Increase 7 Narrow link Network Capacity Cross Traffic Available Bandwidth Measuring (Increased) Queuing Delay Tight link

8. Estimate available bandwidth by saturating the tight link Active Measurement 8 Rate Probe Train # Narrow link Network Capacity Cross Traffic L A 1 == L B 1, no congestion LA1LA1 LB1LB1 1Gbps L A 8 < L B 8, congestion! LA8LA8 LB8LB8 8Gbps 6Gbps L A 4 == L B 4, no congestion LA4LA4 LB4LB4 4Gbps Available Bandwidth Tight link

9. By measuring the increase in packet train length*, we can compute the queuing delay experienced, hence estimate the available bandwidth. * Increase in packet train length == increase in sum of inter- packet gap 9

10. Limitations of Available Bandwidth Estimation Intrusive/Expensive – 100s of probe packets per packet train Inaccurate – Especially in high speed network Does not work well for bursty traffic 10

11. Goals of Available Bandwidth Estimation Cheap – Use as little probe packets as possible Accurate – How close is the estimation to the actual value Works for all traffic patterns 11

12. MinProbe: Bandwidth Estimation in PHY 12 Cheap – Use as little probe packets as possible – 100s of probe packets per train  20 packets per train – Using application traffic as probe Accurate – How close is the estimation to the actual value – Error < 0.4Gbps, with a resolution of 0.1Gbps Works for all traffic patterns – Works with bursty cross traffic

13. Outline 13 Introduction Challenges Design: MinProbe Evaluation Conclusion

14. Challenges 14 LA8LA8 LB8LB8 Cannot Control at 100ps Cannot Measure at 100ps

15. Challenges State-of-art (software) tools do not work at high speed because they cannot control and capture inter-packet spacing with required precision. 15 LA8LA8 LB8LB8 Cannot Control at 100ps Cannot Measure at 100ps

16. Outline 16 Introduction Challenges Design: MinProbe Evaluation Conclusion

17. 17 MinProbe can accurately control and measure inter-packet gaps, thus can accurately control and measure probe train lengths, enabling accurate available bandwidth estimation.

18. MinProbe: Better Accuracy 18 Application Transport Network Data Link Physical Packet iPacket i+1Packet i+2

19. MinProbe: Better Accuracy Idle Characters (/I/) – Each bit ~100 picoseconds – 7~8 bit special character in the physical layer – 700~800 picoseconds to transmit – Only in PHY 19 Packet iPacket i+1Packet i+2 Application Transport Network Data Link Physical

20. MinProbe: Better Accuracy Probe Generation 20 1.0Gbps Rate Probe # 4.0Gbps 8.0Gbps By modulating Inter-packet gap at PHY layer, we can generate accurate probe rate. 1500 B 382 /I/s 8Gbps 1500 B 2290 /I/s 4Gbps 1500 B 11374 /I/s 1Gbps

21. MinProbe: Better Accuracy Probe Generation 21 1.0Gbps Rate Probe # 4.0Gbps 8.0Gbps

22. MinProbe: Better Accuracy With shared N and (Rmin, Rstep, Rmax) Emulate existing bandwidth estimation algorithm – Pathload, Pathchirp, SLoPS … 22 1.0Gbps Rate Probe # 4.0Gbps 8.0Gbps ……

23. MinProbe: Low Overhead 23 Reduce the # of probe packets required

24. SoNIC (NSDI 2013) MinProbe: Very Low Overhead 24 Incoming Traffic Modulated Probe Traffic MinProbe Middlebox IPG Tx/Rx Forwarding Path Mbox Daemon Estimator Flow Table Mbox Ctrl Userspace Kernel space IPG Tx/Rx Estimator

25. MinProbe: Bandwidth Estimation in PHY 25 Incoming traffic Packet Size 792 Bytes 200 /I/s @ 8Gbps Modulated traffic Un-modulated traffic 200 /I/s @ 8Gbps

26. Outline 26 Introduction Challenges Design: MinProbe Evaluation Conclusion

27. Questions: Can MinProbe accurately estimate avail-bw at 10Gbps? Can existing estimation algorithms work with MinProbe? How do the following parameters affect accuracy? – Packet train length – probe packet size distribution – cross packet size distribution – cross packet burstiness Does MinProbe work in the wild, Internet? Does MinProbe work in rate limiting environments? 27 Can MinProbe accurately estimate avail-bw at 10Gbps? Can existing estimation algorithms work with MinProbe? How do the following parameters affect accuracy? – Packet train length – probe packet size distribution – cross packet size distribution – cross packet burstiness Does MinProbe work in the wild, Internet? Does MinProbe work in rate limiting environments?

28. Experiment Setup 28 Controlled EnvironmentNational Lambda Rail NYC Cornell(NYC) BostonChicago Cleveland

29. MinProbe: Can we measure at 10Gbps? Cross Traffic 29 tight link Probe Traffic MinProbe 10Gbps Probe Traffic Cross Traffic, paced at 1.0, 3.0, 6.0, 8.0Gbps

30. MinProbe: Can we measure at 10Gbps? Cross Traffic 30 10Gbps tight link MinProbe 0.1Gbps Rate … 9.6 Gbps 0.2Gbps … Cross Traffic, paced at 1.0, 3.0, 6.0, 8.0Gbps Probe Traffic

31. MinProbe: Can we measure at 10Gbps? 31 10Gbps 0.1Gbps Rate … 9.6 Gbps 0.2Gbps … Available Bandwidth = 2Gbps Cross Traffic 8Gbps 0.1Gbps 8.1Gbps

32. MinProbe: Can we measure at 10Gbps? 32 10Gbps 0.1Gbps Rate … 9.6 Gbps 0.2Gbps … Available Bandwidth = 2Gbps Cross Traffic 8Gbps 0.2Gbps 8.2Gbps

33. MinProbe: Can we measure at 10Gbps? 33 10Gbps 0.1Gbps Rate … 9.6 Gbps 0.2Gbps … Available Bandwidth = 2Gbps Cross Traffic 8Gbps 0.3Gbps 8.3Gbps

34. MinProbe: Can we measure at 10Gbps? 34 10Gbps 0.1Gbps Rate … 9.6 Gbps 0.2Gbps … Available Bandwidth = 2Gbps Cross Traffic 8Gbps 1.0Gbps 9.0Gbps 1.9Gbps 9.9Gbps 2.0Gbps Queued 10.0Gbps 3.0Gbps Queued 10.0Gbps 4.0Gbps Queued 10.0Gbps 5.0Gbps Queued 10.0Gbps 6.0Gbps Queued 10.0Gbps

35. MinProbe: Can we measure at 10Gbps? 35 10Gbps Available Bandwidth = 4Gbps Cross Traffic 6Gbps 0.1Gbps Rate … 9.6 Gbps 0.2Gbps …

36. MinProbe: Can we measure at 10Gbps? 36 10Gbps Available Bandwidth = 7Gbps Cross Traffic 3Gbps 0.1Gbps Rate … 9.6 Gbps 0.2Gbps …

37. MinProbe: Can we measure at 10Gbps? 37 10Gbps 0.1Gbps Rate … 9.6 Gbps 0.2Gbps … Available Bandwidth = 9Gbps Cross Traffic 1Gbps

38. MinProbe: Can we measure at 10Gbps? 38 10Gbps

39. MinProbe: National Lambda Rail 39 10Gbps NYC Cornell(NYC) Cornell(Ithaca) BostonChicago Cleveland

40. Questions: Can MinProbe accurately estimate avail-bw at 10Gbps? Can existing estimation algorithms work with MinProbe? How do the following parameters affect accuracy? – Packet train length – probe packet size distribution – cross packet size distribution – cross packet burstiness Does MinProbe work in the wild, Internet? Does MinProbe work in rate limiting environments? 40

41. MinProbe: In rate limiting environment? Rate Limiters may be an issue? – Provisioned 1Gbps virtual network – If the probe train is less than 1200 packets, no packet loss 41 No

42. Outline 42 Introduction Challenges Design: MinProbe Evaluation Conclusion

43. Modulation of probe packets in PHY Accurate control & measure of packet timing Enabled available bandwidth estimation in 10Gbps Accurate, Minimum Overhead, Available Bandwidth Estimation in High Speed Wired Networks 43

44. Before Next time Project Interim report – Due Monday, November 24. – And meet with groups, TA, and professor Fractus Upgrade: Should be back online Required review and reading for Monday, November 17 – The Xen-Blanket: Virtualize Once, Run Everywhere, D. Williams, H. Jamjoom, and H. Weatherspoon. ACM European Conference on Computer Systems (EuroSys), April 2012, pages 113-126. – http://dl.acm.org/citation.cfm?id=2168849 – http://fireless.cs.cornell.edu/publications/xen-blanket.pdf Check piazza: http://piazza.com/cornell/fall2014/cs5413 Check website for updated schedule