Hash Functions for Network Applications (II) Yaxuan Qi NSLab, RIIT Tsinghua University

Outline Concept and Theory (1~2) Applications (3~4) Hash functions Bloom Filters Applications (3~4)

Basic Idea Packet: header & payload. L3-L4 header Forwarding Engine: Router, Firewall, IPS. Rule Set: exact match; prefix match; range match ACTION: drop/accept Ruleset may contain thousands of rules, then How to efficiently lookup the table?

Technique Packet: header & payload. L3-L4 header Forwarding Engine: Router, Firewall, IPS. Rule Set: exact match; prefix match; range match ACTION: drop/accept Ruleset may contain thousands of rules, then How to efficiently lookup the table?

False Positive n: number of messages m: number of bloom bits k: number of hash functions False Positive p(y是fp ) = p(y不属于X)*p(y对应的k个bits都是1) = p(y对应的k个bits都是1) 考虑对y对应的特定的k个bits, 都被set(由X引起)的概率 首先考虑1个指定bit被set(由X引起)的概率…

Math (I) Two potential assumptions: m: big enough… kn/m: constant… n: number of messages m: number of bloom bits k: number of hash functions Two potential assumptions: m: big enough… kn/m: constant…

n: number of messages m: number of bloom bits k: number of hash functions In practice If the number of 0 bits in the array is substantially less than expected, then the probability of a false positive will be higher than the quantity f that we computed.

Optimal Number of Hash Functions Given m and n minimizes f as a function of k Two competing forces k ?? (from view of search) more chances to find a 0 bit for an element that is not a match (from view of construction) increases the fraction of 0 bits in the array

Math (II) In practice, k must be an integer, and a smaller, suboptimal k might be preferred since this reduces the number of hash functions that have to be computed.

Optimization: Summary Assumption We have good hash functions, look random. Given m bits for filter and n elements, choose number k of hash functions to minimize false positives: Let As k increases more chances to find a 0 but more 1’s in the array. Conclusion

Partial Bloom Filters The total number of bits is still m, but the bits are divided equally among the k hash functions. Each hash function has a range of m/k consecutive bit, make parallelization of array accesses. Packet: header & payload. L3-L4 header Forwarding Engine: Router, Firewall, IPS. Rule Set: exact match; prefix match; range match ACTION: drop/accept Ruleset may contain thousands of rules, then How to efficiently lookup the table? Though the probability of a false positive is actually always at least as large with this division, the difference is small...

Counting Bloom Filters: Idea

Counting Bloom filters: Implementation 4 bits is enough...

Compressed Bloom Filters: Problem

Compressed Bloom Filters: Motivation Insight: Bloom filter is not just a data structure, it is also a message. If the Bloom filter is a message, worthwhile to compress it Further reduce traffic of URL exchanging Compressing bit vectors is easy. Arithmetic coding gets close to entropy. Can Bloom filters be compressed? Bloom filter looks like a random string

Compression: Technique

Compression: Results z/n = 8 Original Compressed At k = m (ln 2) /n, false positives are maximized with a compressed Bloom filter. Best case without compression is worst case with compression; compression always helps. Side benefit: Use fewer hash functions with compression; possible speedup (depend on the bottleneck: memory or link).

Bloom Filter vs. Perfect Hash If the set X of n elements is fixed, one can find a perfect hash function for X plus a fully uniform random hash function Then build a table with n entries of j bits each Mapping each X to n j-bit index, thus the false positive is exactly (1/2)j . matches the lower bound of bloom filter: HOWEVER any change in the set X would require an expensive recomputation of a perfect hash function.

Bloom Filter: Tricks Union (combining two BFs) The same m and the same hash functions Just OR the two bit vectors of the original Bloom filters Shrinking (halve a big BF) just OR the first and second halves together the highest order bit can be masked Intersection (estimation)

Applications

Questions?