1. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap Algorithms for Counting Active Flows on High Speed Links Cristian Estan, George Varghese, Mike Fisk Computer Science and Engineering Department, University of California, San Diego

2. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Why count flows? Detect port/IP scans Identify DoS attacks Estimate spreading rate of a worm Packet scheduling Dave Plonkas FlowScan

3. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Existing flow counting solutions Server NetFlow data Analysis Traffic reports Network Operations Center Router Fast link Memory Network Memory size & bandwidth Networkbandwidth

4. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Motivating question Can we count flows at line speeds at the router? –Wrong solution – counters –Naïve solution – use hash tables (like NetFlow) –Our approach – use bitmaps

5. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting algorithms A family of algorithms that can be used as building blocks in various systems Algorithms can be adapted to application Low memory and per packet processing Generalize flows to distinct header patterns –Count flows or source addresses to detect attack –Count destination address+port pairs to detect scan

6. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Talk structure Per packet processing for bitmap algorithms Computing flow count estimates from bitmaps Variance analysis of estimates Derived algorithms Related work Measurements Conclusions

7. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap HASH(green)=10001001 Set bits in the bitmap using hash of the flow ID of incoming packets

8. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap HASH(blue)=00100100 Different flows have different hash values

9. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap HASH(green)=10001001 Packets from the same flow always hash to the same bit

10. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap HASH(violet)=10010101 Collisions OK, estimates compensate for them

11. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap HASH(orange)=11110011

12. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap HASH(pink)=11100000

13. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap HASH(yellow)=01100011 As the bitmap fills up, estimates get inaccurate

14. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap Solution: use more bits HASH(green)=10001001

15. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap Solution: use more bits Problem: memory scales with the number of flows HASH(blue)=00100100

16. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – virtual bitmap Solution: a) store only a portion of the bitmap b) multiply estimate by scaling factor

17. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – virtual bitmap HASH(pink)=11100000

18. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – virtual bitmap HASH(yellow)=01100011 Problem: estimate inaccurate when few flows active

19. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – multiple bmps Solution: use many bitmaps, each accurate for a different range

20. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – multiple bmps HASH(pink)=11100000

21. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – multiple bmps HASH(yellow)=01100011

22. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – multiple bmps Use this bitmap to estimate number of flows

23. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – multiple bmps Use this bitmap to estimate number of flows

24. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – multires. bmp Problem: must update up to three bitmaps per packet Solution: combine bitmaps into one OR

25. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 HASH(pink)=11100000 Bitmap counting – multires. bmp

26. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – multires. bmp HASH(yellow)=01100011

27. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Talk structure Per packet processing for bitmap algorithms Computing flow count estimates from bitmaps Variance analysis of estimates Derived algorithms Related work Measurements Conclusions

28. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Basic estimates Direct bitmap Virtual bitmap

29. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Multiresolution bitmap estimate Find most accurate component Estimate number of flows hashing to it Apply scaling factor

30. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Talk structure Per packet processing for bitmap algorithms Computing flow count estimates from bitmaps Variance analysis of estimates Derived algorithms Related work Measurements Conclusions

31. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Relative error in estimates Direct bitmap Virtual bitmap Multiresolution bitmap

32. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Error of virtual bitmap Flow density (flows/bit) Average (relative) error

33. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Memory requirements Direct bitmap< N / ln (Nε 2 +1) Virtual bitmap1.5441/ ε 2 Multiresolution bitmap0.9186 ln (Nε 2 ) / ε 2 +ct.

34. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 100 million flows, error 1% Hash table*1.21 Gbytes Direct bitmap1.29 Mbytes Virtual bitmap*1.88 Kbytes Multiresolution bitmap10.33 Kbytes

35. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Talk structure Per packet processing for bitmap algorithms Computing flow count estimates from bitmaps Variance analysis of estimates Derived algorithms Related work Measurements Conclusions

36. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Triggered bitmap Need multiple instances of counting algorithm (e.g. port scan detection) Many instances count few flows Triggered bitmap –Allocate small direct bitmap to new sources –If number of bits set exceeds trigger value, allocate large multiresolution bitmap

37. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Talk structure Per packet processing for bitmap algorithms Computing flow count estimates from bitmaps Variance analysis of estimates Derived algorithms Related work Measurements Conclusions

38. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Related work Flajolet, Martin (1985) probabilistic counting –Memory use similar to multiresolution bitmap Whang et al (1990) introduce direct bitmap You, Chang (1996) use virtual bitmap Chauduri, Motwani, Narasayya (1998) –Counting flows without bias impossible from sampled data Duffield, Lund, Thorup (2002) –Accurate solutions based on counting TCP SYN flags

39. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Talk structure Per packet processing for bitmap algorithms Computing flow count estimates from bitmaps Variance analysis of estimates Derived algorithms Related work Measurements Conclusions

40. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Multires. bmp. vs. prob. counting Number of flows (log scale) Average (relative) error

41. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Scan detection memory usage Interval length Snort (naïve) Probabilistic counting Triggered bitmap 12 seconds1.94 M2.42 M0.37 M 600 seconds49.60 M22,34 M5.59 M

42. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Talk structure Per packet processing for bitmap algorithms Computing flow count estimates from bitmaps Variance analysis of estimates Derived algorithms Related work Measurements Conclusions

43. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 A family of counting algorithms SettingAlgorithmApplications General countingMultiresolution bmp.Track infections Narrow rangeVirtual bitmapTriggers (e.g. DoS) Small counts commonTriggered bitmapPort scans StationarityAdaptive bitmapMeasurement Add and deleteIncrement-decrementScheduling

44. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting algorithms A family of algorithms that can be used as building blocks in various systems Algorithms can be adapted to application Low memory and per packet processing –With 2Kbytes error around 1%

45. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 The end Bitmap algorithms will be available at: http://ial.ucsd.edu/bitmaps/ Any questions? Acknowledgements: Vern Paxson, David Moore, Philippe Flajolet, Marianne Durand, Alex Snoeren, K Claffy, Stefan Savage, Florin Baboescu, NIST,NSF

46. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Adaptive bitmap Virtual bitmap measures accurately number of flows if range known in advance Often number of flows does not change rapidly Measurement repeated Can use previous measurement to tune virtual bitmap Combine a large virtual bitmap with a small multiresolution bitmap used for tuning

47. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Adaptive bitmap accuracy Number of flows (log scale) Average (relative) error

48. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 With 2 kilobytes of memory With 2 kilobytes of memory Adaptive bitmap (min avg max) Probabilistic counting (min avg max) Trace1-4.4% 1.1% 4.7%-9.5% 2.8% 13.3% Trace2-1.9% 0.7% 2.0%-6.9% 2.8% 7.6% Trace3-1.8% 0.6% 1.8%2.4% 10.2% 17.7%

49. Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Increment-decrement algorithms Active flow defined as flow with packets in queue Must support additions and deletions Replace bits of bitmap with counters –Increment when packet arrives –Decrement when packet leaves –Estimate number of flows based on zero counters