Presentation is loading. Please wait.

Presentation is loading. Please wait.

Approximate Aggregation Techniques for Sensor Databases John Byers Computer Science Department Boston University Joint work with Jeffrey Considine, George.

Similar presentations


Presentation on theme: "Approximate Aggregation Techniques for Sensor Databases John Byers Computer Science Department Boston University Joint work with Jeffrey Considine, George."— Presentation transcript:

1 Approximate Aggregation Techniques for Sensor Databases John Byers Computer Science Department Boston University Joint work with Jeffrey Considine, George Kollios and Feifei Li

2 Sensor Network Model Large set of sensors distributed in a sensor field. Large set of sensors distributed in a sensor field. Communication via a wireless ad-hoc network. Communication via a wireless ad-hoc network. Node and links are failure-prone. Node and links are failure-prone. Sensors are resource-constrained Sensors are resource-constrained –Limited memory, battery-powered, messaging is costly.

3 Sensor Databases Useful abstraction: Useful abstraction: –Treat sensor field as a distributed database But: data is gathered, not stored nor saved. But: data is gathered, not stored nor saved. –Express query in SQL-like language COUNT, SUM, AVG, MIN, GROUP-BY COUNT, SUM, AVG, MIN, GROUP-BY –Query processor distributes query and aggregates responses –Exemplified by systems like TAG (Berkeley/MIT) and Cougar (Cornell)

4 A B C D E G H I J H A Motivating Example Each sensor has a single sensed value. Each sensor has a single sensed value. Sink initiates one-shot queries such as: What is the… Sink initiates one-shot queries such as: What is the… –maximum value? –mean value? Continuous queries are a natural extension. Continuous queries are a natural extension. F 124

5 MAX Aggregation (no losses) Build spanning tree Build spanning tree Aggregate in-network Aggregate in-network –Each node sends one summary packet –Summary has MAX of entire sub-tree One loss could lose MAX of many nodes One loss could lose MAX of many nodes –Neighbors of sink are particularly vulnerable A B C D E G H I J H F MAX=12

6 MAX Aggregation (with loss) Nodes send summaries over multiple paths Nodes send summaries over multiple paths –Free local broadcast –Always send MAX value observed MAX is “infectious” MAX is “infectious” –Harder to lose –Just need one viable path to the sink Relies on duplicate- insensitivity of MAX Relies on duplicate- insensitivity of MAX A B C D E G H I J H F MAX=12

7 AVG Aggregation (no losses) Build spanning tree Build spanning tree Aggregate in-network Aggregate in-network –Each node sends one summary packet –Summary has SUM and COUNT of sub-tree Same reliability problem as before Same reliability problem as before A B C D E G H I J H F 12 2,1 6,1 9,2 15,3 6,110,1 26,4 12,1 10,2 9,1 AVG=70/10=7

8 AVG Aggregation (naive) What if redundant copies of data are sent? What if redundant copies of data are sent? AVG is duplicate- sensitive AVG is duplicate- sensitive –Duplicating data changes aggregate –Increases weight of duplicated data A B C D E G H I J H F 12 2,1 6,1 9,2 15,3 6,122,2 26,4 12,1 10,2 9,1 AVG=82/11≠7 12,1

9 AVG Aggregation (TAG++) Can compensate for increased weight [MFHH’02] Can compensate for increased weight [MFHH’02] –Send halved SUM and COUNT instead Does not change expectation! Does not change expectation! Only reduces variance Only reduces variance A B C D E G H I J H F 12 2,1 6,1 9,2 15,3 6,116,0.5 20,3.5 6,0.5 10,2 9,1 AVG=70/10=7 6,0.5

10 AVG Aggregation (LIST) Can handle duplicates exactly with a list of pairs Can handle duplicates exactly with a list of pairs Transmitting this list is expensive! Transmitting this list is expensive! Lower bound: linear space is necessary if we demand exact results. Lower bound: linear space is necessary if we demand exact results. A B C D E G H I J H F 12 A,2 B,6 B,6; D,3 A,2; G,7; H,6 H,6 F,1;I,10 C,9; E,1; F,1; H,4 F,1 C,9;E,1 C,9 AVG=70/10=7 F,1

11 Classification of Aggregates TAG classifies aggregates according to TAG classifies aggregates according to –Size of partial state –Monotonicity –Exemplary vs. summary –Duplicate-sensitivity MIN/MAX (cheap and easy) MIN/MAX (cheap and easy) –Small state, monotone, exemplary, duplicate-insensitive COUNT/SUM/AVG (considerably harder) COUNT/SUM/AVG (considerably harder) –Small state and monotone, BUT duplicate-sensitive –Cheap if aggregating over tree without losses –Expensive with multiple paths and losses

12 Design Objectives for Robust Aggregation Admit in-network aggregation of partial values. Admit in-network aggregation of partial values. Let representation of aggregates be both order- insensitive and duplicate-insensitive. Let representation of aggregates be both order- insensitive and duplicate-insensitive. Be agnostic to routing protocol Be agnostic to routing protocol –Trust routing protocol to be best-effort. –Routing and aggregation can be logically decoupled [NG ’03]. –Some routing algorithms better than others (multipath). Exact answers incur extremely high cost. Exact answers incur extremely high cost. –We argue that it is reasonable to use aggregation methods that are themselves approximate.

13 Outline Introduction Introduction Sketch Theory and Practice Sketch Theory and Practice –COUNT sketches (old) –SUM sketches (new) –Practical realizations for sensor nets Experiments Experiments Conclusions Conclusions

14 COUNT Sketches Problem: Estimate the number of distinct item IDs in a data set with only one pass. Problem: Estimate the number of distinct item IDs in a data set with only one pass. Constraints: Constraints: –Small space relative to stream size. –Small per item processing overhead. –Union operator on sketch results. Exact COUNT is impossible without linear space. Exact COUNT is impossible without linear space. First approximate COUNT sketch in [FM’85]. First approximate COUNT sketch in [FM’85]. –O(log N) space, O(1) processing time per item.

15 Counting Paintballs Imagine the following scenario: Imagine the following scenario: –A bag of n paintballs is emptied at the top of a long stair-case. –At each step, each paintball either bursts and marks the step, or bounces to the next step. 50/50 chance either way. Looking only at the pattern of marked steps, what was n?

16 Counting Paintballs (cont) What does the distribution of paintball bursts look like? What does the distribution of paintball bursts look like? –The number of bursts at each step follows a binomial distribution. –The expected number of bursts drops geometrically. –Few bursts after log 2 n steps 1 st 2 nd S th B(n,1/2) B(n,1/2 S ) B(n,1/4) B(n,1/2 S )

17 Counting Paintballs (cont) Many different estimator ideas [FM'85,AMS'96,GGR'03,DF'03,...] Many different estimator ideas [FM'85,AMS'96,GGR'03,DF'03,...] Example: Let pos denote the position of the highest unmarked stair, Example: Let pos denote the position of the highest unmarked stair, E(pos) ≈ log 2 ( n)  2 (pos) ≈ Standard variance reduction methods apply Standard variance reduction methods apply Either O(log n) or O(log log n) space Either O(log n) or O(log log n) space

18 Back to COUNT Sketches The COUNT sketches of [FM'85] are equivalent to the paintball process. The COUNT sketches of [FM'85] are equivalent to the paintball process. –Start with a bit-vector of all zeros. –For each item, Use its ID and a hash function for coin flips. Use its ID and a hash function for coin flips. Pick a bit-vector entry. Pick a bit-vector entry. Set that bit to one. Set that bit to one. These sketches are duplicate-insensitive: These sketches are duplicate-insensitive: {x} {y} {x,y}  A,B (Sketch(A)  Sketch(B)) = Sketch(A  B)

19 Application to Sensornets Each sensor computes k independent sketches of itself using its unique sensor ID. Each sensor computes k independent sketches of itself using its unique sensor ID. –Coming next: sensor computes sketches of its value. Use a robust routing algorithm to route sketches up to the sink. Use a robust routing algorithm to route sketches up to the sink. Aggregate the k sketches via in-network XOR. Aggregate the k sketches via in-network XOR. –Union via XOR is duplicate-insensitive. The sink then estimates the count. The sink then estimates the count. Similar to gossip and epidemic protocols. Similar to gossip and epidemic protocols.

20 SUM Sketches Problem: Estimate the sum of values of distinct <> pairs in a data stream with repetitions. ( ≥ 0, integral). Problem: Estimate the sum of values of distinct pairs in a data stream with repetitions. (value ≥ 0, integral). Obvious start: Emulate insertions into a COUNT sketch and use the same estimators. Obvious start: Emulate value insertions into a COUNT sketch and use the same estimators. –For, imagine inserting,, …,

21 SUM Sketches (cont) But what if the value is 1,000,000? But what if the value is 1,000,000? Main Idea (details on next slide): Main Idea (details on next slide): –Recall that all of the low-order bits will be set to 1 w.h.p. inserting such a value. –Just set these bits to one immediately. –Then set the high-order bits carefully.

22 Simulating a set of insertions Set all the low-order bits in the “safe” region. Set all the low-order bits in the “safe” region. –First S = log v – 2 log log v bits are set to 1 w.h.p. Statistically estimate number of trials going beyond “safe” region Statistically estimate number of trials going beyond “safe” region –Probability of a trial doing so is simply 2 -S –Number of trials ~ B (v, 2 -S ). [Mean = O(log 2 v)] For trials and bits outside “safe” region, set those bits manually. For trials and bits outside “safe” region, set those bits manually. –Running time is O(1) for each outlying trial. Expected running time: O(log v) + time to draw from B (v, 2 -S ) + O(log 2 v)

23 Sampling for Sensor Networks We need to generate samples from B (n, p). We need to generate samples from B (n, p). –With a slow CPU, very little RAM, no floating point hardware General problem: sampling from a discrete pdf. General problem: sampling from a discrete pdf. Assume can draw uniformly at random from [0,1]. Assume can draw uniformly at random from [0,1]. With an event space of size N: With an event space of size N: –O(log N) lookups are immediate. Represent the cdf in an array of size N. Represent the cdf in an array of size N. Draw from [0, 1] and do binary search. Draw from [0, 1] and do binary search. –Cleverer methods for O(log log N), O(log* N) time Amazingly, this can be done in constant time!

24 Walker’s Alias Method A – 0.10 B – 0.25 C – 0.05 D – 0.25 E – 0.35 A – 0.10 B – 0.15 C – 0.05 D – 0.25 E – 0.35 B – 0.10 A – 0.10 B – 0.15 C – 0.05 D – 0.20 E – 0.35 B – 0.10 D – 0.05 A – 0.10 B – 0.15 C – 0.05 D – 0.20E – 0.20 B – 0.10 D – 0.05 E – 0.15 Theorem [Walker ’77]: For any discrete pdf D over a sample space of size n, a table of size O(n) can be constructed in O(n) time that enables random variables to be drawn from D using at most two table lookups. Theorem [Walker ’77]: For any discrete pdf D over a sample space of size n, a table of size O(n) can be constructed in O(n) time that enables random variables to be drawn from D using at most two table lookups. n 1/n

25 Binomial Sampling for Sensors Recall we want to sample from B(v,2 -S ) for various values of v and S. Recall we want to sample from B(v,2 -S ) for various values of v and S. –First, reduce to sampling from G(1 – 2 -S ). –Truncate distribution to make range finite (recursion to handle large values). –Construct tables of size 2 S for each S of interest. –Can sample B(v,2 -S ) in O(v · 2 -S ) expected time.

26 The Bottom Line –SUM inserts in O(log 2 (v)) time with O(v / log 2 (v)) space O(log 2 (v)) time with O(v / log 2 (v)) space O(log(v)) time with O(v / log(v)) space O(log(v)) time with O(v / log(v)) space O(v) time with naïve method O(v) time with naïve method –Using O(log 2 (v)) method, 16 bit values (S ≤ 8) and 64 bit probabilities Resulting lookup tables are ~ 4.5KB Resulting lookup tables are ~ 4.5KB Recursive nature of G(1 – 2 -S ) lets us tune size further Recursive nature of G(1 – 2 -S ) lets us tune size further –Can achieve O(log v) time at the cost of bigger tables

27 Outline Introduction Introduction Sketch Theory and Practice Sketch Theory and Practice Experiments Experiments Conclusions Conclusions

28 Experimental Results Used TAG simulator Used TAG simulator Grid topology with sink in the middle Grid topology with sink in the middle –Grid size [default: 30 by 30] –Transmission radius [default: 8 neighbors on the grid] –Node, packet, or link loss [default: 5% link loss rate] –Number of bit vectors [default: 20 bit-vectors of 16 bits (compressed)].

29 Experimental Results We consider four main methods. We consider four main methods. –TAG1: transmit aggregates up a single spanning tree –TAG2: Send a 1/k fraction of the aggregated values to each of k parents. –SKETCH: broadcast an aggregated sketch to all neighbors at level i –1 –LIST: explicitly enumerate all pairs and broadcast to all neighbors at level i – 1. LIST vs. SKETCH measures the penalty associated with approximate values. LIST vs. SKETCH measures the penalty associated with approximate values.

30 COUNT vs Link Loss (grid)

31

32 SUM vs Link Loss (grid)

33 Message Cost Comparison Strategy Total Data Bytes Messages Sent Messages Received TAG TAG SKETCH LIST

34 Outline Introduction Introduction Sketch Theory and Practice Sketch Theory and Practice Experiments Experiments Conclusions Conclusions

35 Our Work in Context In parallel with our efforts, In parallel with our efforts, –Nath and Gibbons (Intel/CMU) What are the essential properties of duplicate insensitivity? What are the essential properties of duplicate insensitivity? What other aggregates can be sketched? What other aggregates can be sketched? –Bawa et al (Stanford) What routing methods are necessary to guarantee the validity and semantics of aggregates? What routing methods are necessary to guarantee the validity and semantics of aggregates?

36 Conclusions Duplicate-insensitive sketches fundamentally change how aggregation works Duplicate-insensitive sketches fundamentally change how aggregation works –Routing becomes logically decoupled –Arbitrarily complex aggregation scenarios are allowable – cyclic topologies, multiple sinks, etc. Extended list of non-trivial aggregates Extended list of non-trivial aggregates –We added SUM, MEAN, VARIANCE, STDEV, … Resulting system performs better Resulting system performs better –Moderate cost (tunable) for large reliability boost

37 Ongoing Work What else can we sketch? What else can we sketch? –Clear need to extend expressiveness of sketches –Also: what are the limits of duplicate-insensitive ones? Distributed streaming model Distributed streaming model –Monitor and sketch streams of data –Collect sketches and estimate global properties Traffic monitoring Traffic monitoring –Identifying large flows, flows with large changes –Both already done with counting Bloom filters [KSGC’03,CM’04] We can make those duplicate-insensitive! We can make those duplicate-insensitive! Aggregation via random sampling Aggregation via random sampling

38 Future Directions (cont)

39 Message Cost Comparison Strategy Total Data Bytes Messages Sent Messages Received TAG TAG SKETCH LIST

40 Thank you! More questions?

41 Multipath Routing Braided Paths: Two paths from the source to the sink that differ in at least two nodes

42 Design Objectives (cont) Final aggregate is exact if at least one representative from each leaf survives to reach the sink. Final aggregate is exact if at least one representative from each leaf survives to reach the sink. This won’t happen in practice in sensornets without extremely high cost. This won’t happen in practice in sensornets without extremely high cost. It is reasonable to hope for approximate results. It is reasonable to hope for approximate results. We argue that it is reasonable to use aggregation methods that are themselves approximate. We argue that it is reasonable to use aggregation methods that are themselves approximate.

43 Goal of This Work So far, we’ve seen ideas of So far, we’ve seen ideas of –In-network aggregation (low traffic per link) –Multi-path routing (reliability of individual items) These usually don’t combine well These usually don’t combine well –Only works for duplicate-insensitive aggregates such as MIN/MAX, AND/OR What about all the other aggregates? What about all the other aggregates? –We want them cheap, reliable, and correct

44 Contributions of This Work Propose duplicate-insensitive sketches to approximately aggregate data Propose duplicate-insensitive sketches to approximately aggregate data –Difficulty was noted [MFHH’02] –Approximation is necessary With duplicate-insensitive sketches, any best-effort routing method can be employed With duplicate-insensitive sketches, any best-effort routing method can be employed Design new duplicate-insensitive sketches Design new duplicate-insensitive sketches –SUM => MEAN, VARIANCE, STDEV, …

45 Routing Methodologies Considerable work on reliable delivery via multipath routing Considerable work on reliable delivery via multipath routing –Directed diffusion [IGE ’00] –“Braided” diffusion [GGSE ’01] –GRAdient Broadcast [YZLZ ’02] Broadcast intermediate results along gradient back to source Broadcast intermediate results along gradient back to source Can dynamically control width of broadcast Can dynamically control width of broadcast Trade off fault tolerance and transmission costs Trade off fault tolerance and transmission costs Our approach similar to GRAB: Our approach similar to GRAB: –Broadcast. Grab if upstream, ignore if downstream Common goal: try to get at least one copy to sink

46 SUM Sketches (cont) Remaining questions: Remaining questions: –What should S be when inserting ? When using analysis of [FM’85] When using analysis of [FM’85] –S ≈ log 2 (v) – 2 log 2 log(v) –Expected time = O(log 2 (v)) + sample time Can go farther keeping high probability… Can go farther keeping high probability… –S ≈ log 2 (v) – log 2 log(v) –Expected time = O(log(v)) + sample time –How do we sample the binomial distribution? Space requirements may affect choice of S Space requirements may affect choice of S

47 SUM Sketches (cont) Reduction to COUNT sketches: Reduction to COUNT sketches: –Pick a prefix length S The first S bits should be set with high probability. The first S bits should be set with high probability. –Set the first S bits to one. –Sample from B(v, 2 -S ) to figure out how many items would pick bits after the first S bits. –Simulate the insertion of those items. Expected time = Expected time = O(S) + sample time + O(v·2 -S )

48 Sampling Constraints Sensor motes have very limited resources Sensor motes have very limited resources –Slow CPU –Very little RAM –No floating point hardware Sampling from B(n, p) isn’t easy normally Sampling from B(n, p) isn’t easy normally –Obviously O(log n) time and O(n) space –O(np) expected time (and good FP hardware) with standard reduction to geometric distribution How hard is this sampling problem anyway? How hard is this sampling problem anyway?

49 COUNT vs Diameter (grid)

50 COUNT vs Link Loss (random)


Download ppt "Approximate Aggregation Techniques for Sensor Databases John Byers Computer Science Department Boston University Joint work with Jeffrey Considine, George."

Similar presentations


Ads by Google