Download presentation

Presentation is loading. Please wait.

Published byAdam Power Modified over 2 years ago

1
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap Algorithms for Counting Active Flows on High Speed Links Cristian Estan, George Varghese, Mike Fisk Computer Science and Engineering Department, University of California, San Diego

2
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Why count flows? Detect port/IP scans Identify DoS attacks Estimate spreading rate of a worm Packet scheduling Dave Plonkas FlowScan

3
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Existing flow counting solutions Server NetFlow data Analysis Traffic reports Network Operations Center Router Fast link Memory Network Memory size & bandwidth Networkbandwidth

4
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Motivating question Can we count flows at line speeds at the router? –Wrong solution – counters –Naïve solution – use hash tables (like NetFlow) –Our approach – use bitmaps

5
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting algorithms A family of algorithms that can be used as building blocks in various systems Algorithms can be adapted to application Low memory and per packet processing Generalize flows to distinct header patterns –Count flows or source addresses to detect attack –Count destination address+port pairs to detect scan

6
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Talk structure Per packet processing for bitmap algorithms Computing flow count estimates from bitmaps Variance analysis of estimates Derived algorithms Related work Measurements Conclusions

7
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap HASH(green)= Set bits in the bitmap using hash of the flow ID of incoming packets

8
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap HASH(blue)= Different flows have different hash values

9
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap HASH(green)= Packets from the same flow always hash to the same bit

10
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap HASH(violet)= Collisions OK, estimates compensate for them

11
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap HASH(orange)=

12
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap HASH(pink)=

13
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap HASH(yellow)= As the bitmap fills up, estimates get inaccurate

14
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap Solution: use more bits HASH(green)=

15
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap Solution: use more bits Problem: memory scales with the number of flows HASH(blue)=

16
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – virtual bitmap Solution: a) store only a portion of the bitmap b) multiply estimate by scaling factor

17
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – virtual bitmap HASH(pink)=

18
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – virtual bitmap HASH(yellow)= Problem: estimate inaccurate when few flows active

19
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – multiple bmps Solution: use many bitmaps, each accurate for a different range

20
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – multiple bmps HASH(pink)=

21
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – multiple bmps HASH(yellow)=

22
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – multiple bmps Use this bitmap to estimate number of flows

23
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – multiple bmps Use this bitmap to estimate number of flows

24
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – multires. bmp Problem: must update up to three bitmaps per packet Solution: combine bitmaps into one OR

25
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 HASH(pink)= Bitmap counting – multires. bmp

26
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – multires. bmp HASH(yellow)=

27
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Talk structure Per packet processing for bitmap algorithms Computing flow count estimates from bitmaps Variance analysis of estimates Derived algorithms Related work Measurements Conclusions

28
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Basic estimates Direct bitmap Virtual bitmap

29
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Multiresolution bitmap estimate Find most accurate component Estimate number of flows hashing to it Apply scaling factor

30
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Talk structure Per packet processing for bitmap algorithms Computing flow count estimates from bitmaps Variance analysis of estimates Derived algorithms Related work Measurements Conclusions

31
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Relative error in estimates Direct bitmap Virtual bitmap Multiresolution bitmap

32
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Error of virtual bitmap Flow density (flows/bit) Average (relative) error

33
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Memory requirements Direct bitmap< N / ln (Nε 2 +1) Virtual bitmap1.5441/ ε 2 Multiresolution bitmap ln (Nε 2 ) / ε 2 +ct.

34
Bitmap algorithms for flow counting – Internet Measurement Conference, October million flows, error 1% Hash table*1.21 Gbytes Direct bitmap1.29 Mbytes Virtual bitmap*1.88 Kbytes Multiresolution bitmap10.33 Kbytes

35
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Talk structure Per packet processing for bitmap algorithms Computing flow count estimates from bitmaps Variance analysis of estimates Derived algorithms Related work Measurements Conclusions

36
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Triggered bitmap Need multiple instances of counting algorithm (e.g. port scan detection) Many instances count few flows Triggered bitmap –Allocate small direct bitmap to new sources –If number of bits set exceeds trigger value, allocate large multiresolution bitmap

37
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Talk structure Per packet processing for bitmap algorithms Computing flow count estimates from bitmaps Variance analysis of estimates Derived algorithms Related work Measurements Conclusions

38
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Related work Flajolet, Martin (1985) probabilistic counting –Memory use similar to multiresolution bitmap Whang et al (1990) introduce direct bitmap You, Chang (1996) use virtual bitmap Chauduri, Motwani, Narasayya (1998) –Counting flows without bias impossible from sampled data Duffield, Lund, Thorup (2002) –Accurate solutions based on counting TCP SYN flags

39
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Talk structure Per packet processing for bitmap algorithms Computing flow count estimates from bitmaps Variance analysis of estimates Derived algorithms Related work Measurements Conclusions

40
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Multires. bmp. vs. prob. counting Number of flows (log scale) Average (relative) error

41
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Scan detection memory usage Interval length Snort (naïve) Probabilistic counting Triggered bitmap 12 seconds1.94 M2.42 M0.37 M 600 seconds49.60 M22,34 M5.59 M

42
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Talk structure Per packet processing for bitmap algorithms Computing flow count estimates from bitmaps Variance analysis of estimates Derived algorithms Related work Measurements Conclusions

43
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 A family of counting algorithms SettingAlgorithmApplications General countingMultiresolution bmp.Track infections Narrow rangeVirtual bitmapTriggers (e.g. DoS) Small counts commonTriggered bitmapPort scans StationarityAdaptive bitmapMeasurement Add and deleteIncrement-decrementScheduling

44
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting algorithms A family of algorithms that can be used as building blocks in various systems Algorithms can be adapted to application Low memory and per packet processing –With 2Kbytes error around 1%

45
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 The end Bitmap algorithms will be available at: Any questions? Acknowledgements: Vern Paxson, David Moore, Philippe Flajolet, Marianne Durand, Alex Snoeren, K Claffy, Stefan Savage, Florin Baboescu, NIST,NSF

46
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Adaptive bitmap Virtual bitmap measures accurately number of flows if range known in advance Often number of flows does not change rapidly Measurement repeated Can use previous measurement to tune virtual bitmap Combine a large virtual bitmap with a small multiresolution bitmap used for tuning

47
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Adaptive bitmap accuracy Number of flows (log scale) Average (relative) error

48
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 With 2 kilobytes of memory With 2 kilobytes of memory Adaptive bitmap (min avg max) Probabilistic counting (min avg max) Trace1-4.4% 1.1% 4.7%-9.5% 2.8% 13.3% Trace2-1.9% 0.7% 2.0%-6.9% 2.8% 7.6% Trace3-1.8% 0.6% 1.8%2.4% 10.2% 17.7%

49
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Increment-decrement algorithms Active flow defined as flow with packets in queue Must support additions and deletions Replace bits of bitmap with counters –Increment when packet arrives –Decrement when packet leaves –Estimate number of flows based on zero counters

Similar presentations

© 2016 SlidePlayer.com Inc.

All rights reserved.

Ads by Google