Presentation is loading. Please wait.

Presentation is loading. Please wait.

An Optimal Algorithm for the Distinct Elements Problem

Similar presentations

Presentation on theme: "An Optimal Algorithm for the Distinct Elements Problem"— Presentation transcript:

1 An Optimal Algorithm for the Distinct Elements Problem
Daniel Kane, Jelani Nelson, David Woodruff PODS, 2010

2 Problem Description Given a long stream of values from a universe of size n each value can occur any number of times count the number F0 of distinct values See values one at a time One pass over the stream Too expensive to store set of distinct values Algorithms should: Use a small amount of memory Have fast update time (per value processing time) Have fast recovery time (time to report the answer)

3 Randomized Approximation Algorithms
3, 141, 59, 265, 3, 58, 9, 7, 9, 32, 3, , 338, 32, 4, … Consider algorithms that store a subset S of distinct values E.g., S = {3, 9, 32, 265} Main drawback is that S needs to be large to know if next value is a new distinct value Any algorithm (whether it stores a subset of values or not) that is either deterministic or computes F0 exactly must use ¼ F0 memory Hence, algorithms must be randomized and settle for an approximate solution: output F 2 [(1-ε)F0, (1+ε)F0] with good probability

4 Problem History Long sequence of work on the problem
Flajolet and Martin introduced problem, FOCS 1983 Alon, Bar-Yossef, Beyer, Brody, Chakrabarti, Durand, Estan, Flajolet, Fisk, Fusy, Gandouet, Gemulla, Gibbons, P. Haas, Indyk, Jayram, Kumar, Martin, Matias, Meunier, Reinwald, Sismanis, Sivakumar, Szegedy, Tirthapura, Trevisan, Varghese, W Previous best algorithm: O(ε-2 log log n + log n) bits of memory and O(ε-2) update and reporting time Known lower bound on the memory: (ε-2 + log n) Our result: Optimal O(ε-2 + log n) bits of memory and O(1) update and reporting time

5 Previous Approaches Suppose we randomly hash F0 values into a hash table of 1/ε2 buckets and keep track of the number C of non-empty buckets If F0 < 1/ε2, there is a way to estimate F0 up to (1 ± ε) from C Problem: if F0 À 1/ε2, with high probability, every bucket contains a value, so there is no information Solution: randomly choose Slog n µ Slog n - 1 µ Slog n - 2  µ S1 µ {1, 2, …, n}, where |Si| ¼ n/2i Problem: It takes 1/ε2 log n bits of memory to keep track of this information stream: 3, 141, 59, 265, 3, 58, 9, 7, 9, 32, 3, , 338, 32, 4, … Si = {1, 3, 7, 9, 265} i-th substream: 3, 265, 3, 9, 7, 9, 3, … Run hashing procedure on each substream There is an i for which the # of distinct values in i-th substream ¼ 1/ε2 Hashing procedure on i-th substream works

6 Our Techniques Observation: - Have 1/ε2 global buckets
- In each bucket we keep track of the index i of the set Si for the largest i for which Si contains a value hashed to the bucket - This gives O(1/ε2 log log n) bits of memory New Ideas: - Can show with high probability, at every point in the stream, most buckets contain roughly the same index - We can just keep track of the offsets from this common index - We pack the offsets into machine words and use known fast read/write algorithms to variable length arrays to efficiently update offsets - Occasionally we need to decrement all offsets. Can spread the work across multiple updates

Download ppt "An Optimal Algorithm for the Distinct Elements Problem"

Similar presentations

Ads by Google