1. A Thin Monitoring Layer for Top-k Aggregation Queries over a Database Foteini AlvanakiSebastian Michel Saarland University DBRank 2013, Riva Del Garda, Italy

2. Data Cube sum(price*quantity)

3. Data Cube BrandProduct TypeCountrysum(Price*Quantity) Brand1Type1Country11234 Brand1Type2Country13522 Brand1Type11234 Brand1Type23522 Brand1Country14756 Type1Country11234 Type2Country13522 Brand14756 Type11234 Type23522 Country14756 1.What are the top-2 product types with the highest revenue of each brand in each country? 2.What are the top-2 brands with the highest revenue in each country?

4. Top-k Queries Primary Attribute: The attribute/dimension over which the selection is performed (e.g. product type) Secondary Attributes: Used to filter specific results (e.g. brand, country) Aggregated Attributes: Used to compute an aggregated score (e.g. price, quantity) Aggregate Function: e.g. sum One top-k query for each combination of secondary attribute instances (filtering condition)

5. Filtering Conditions: Example (1) brand={X}-country = {Y, W} brand=X AND country=Y brand=X AND country=W SELECT type, SUM(price*quantity) FROM relation WHERE brand=X AND country=Y GROUP BY type ORDER BY SUM(price*quantity) LIMIT K SELECT type, SUM(price*quantity) FROM relation WHERE brand=X AND country=W GROUP BY type ORDER BY SUM(price*quantity) LIMIT K

6. Filtering Conditions: Example (2) country = {Y, W}-brand={X} country=Y country=W brand=X SELECT type, SUM(price*quantity) FROM relation WHERE country=Y GROUP BY type ORDER BY SUM(price*quantity) LIMIT K SELECT type, SUM(price*quantity) FROM relation WHERE country=W GROUP BY type ORDER BY SUM(price*quantity) LIMIT K SELECT type, SUM(price*quantity) FROM relation WHERE brand=X GROUP BY type ORDER BY SUM(price*quantity) LIMIT K

7. Filtering Conditions: Example (3) country = {Y, W}-brand={X} SELECT type, SUM(price*quantity) FROM relation GROUP BY type ORDER BY SUM(price*quantity) LIMIT K

8. Updates Insertions to the underlying database that contain all information related to the top-k queries INSERT INTO relation (type, brand, country, price, quantity) VALUES (T, X, Y, 100, 3)

9. Problem How to maintain all these queries in the presence of fast updates?

10. Outline Setting/Problem Algorithms – Naïve Approach – Estimates Approach – Groups Approach Experimental Results Conclusions

11. Example SELECT type, SUM(price*quantity) FROM relation WHERE brand=X AND country=Y GROUP BY type ORDER BY SUM(price*quantity) LIMIT 2 Update: (type, X, Y, 300)

12. Naïve Approach Case 1: type in the top-2, e.g. (B,X,Y,300) TypeScore A3452 B2406 +300 TypeScore A3452 B2706 Case 2: type NOT in the top-2, e.g. (K,X,Y,300) Verification Query: SELECT type, SUM(price*quantity) FROM relation WHERE brand=X AND country=Y AND type=K GROUP BY type

13. Estimates Approach In-memory Structures top-(k+N) instances with exact aggregated scores B instances with estimated aggregated scores best possible score (basic score) + inserted values TypeScore A3452 B2406 C2356 D2167 E1987 top-2 top-5 TypeScore O1990 P2112 Q2076 R1997 Buffer

14. Estimates Approach Case 1.1: type in the top-2, e.g. (B,X,Y,300) TypeScore A3452 B2406 C2356 D2167 E1987 top-2 top-5 TypeScore A3452 B2706 C2356 D2167 E1987 top-2 top-5 +300

15. Estimates Approach Case 1.2: type in the top-5, e.g. (D,X,Y,300) TypeScore A3452 B2406 C2356 D2167 E1987 top-2 top-5 TypeScore A3452 B2406 C2356 D2467 E1987 top-2 top-5 +300 TypeScore A3452 D2467 B2406 C2356 E1987

16. Estimates Approach Case 2: type in the Buffer, e.g. (P,X,Y,300) TypeScore A3452 B2406 C2356 D2167 E1987 top-2 top-5 TypeScore O1990 P2112 Q2076 R1997 Buffer +300 TypeScore O1990 P2412 Q2076 R1997 Buffer Verification Query: SELECT type, SUM(price*quantity) FROM relation WHERE brand=X AND country=Y AND type=P GROUP BY type

17. Estimates Approach Sub-case 2.1: score(P) < score(E), e.g. score(P) = 756 TypeScore A3452 B2406 C2356 D2167 E1987 top-2 top-5 TypeScore O1990 P756 Q2076 R1997 Buffer TypeScore O1990 Q2076 R1997

18. Estimates Approach Sub-case 2.2: score(P) > score(E), e.g. score(P) = 2178 TypeScore A3452 B2406 C2356 D2167 E1987 top-2 top-5 TypeScore O1990 P2178 Q2076 R1997 Buffer TypeScore A3452 B2406 C2356 D2167 P2178 TypeScore O1990 Q2076 R1997

19. Estimates Approach Sub-case 2.3: score(P) > score(B), e.g. score(P) = 2407 TypeScore A3452 B2406 C2356 D2167 E1987 top-2 top-5 TypeScore O1990 P2407 Q2076 R1997 Buffer TypeScore A3452 P2407 B2406 C2356 D2167 TypeScore O1990 Q2076 R1997

20. Estimates Approach Buffer Full Reset Query Estimated Score(T) = basic score + 300 = 2287 Case 3: type NOT in in-memory structures, e.g. (T,X,Y,300) SELECT type, SUM(price*quantity) FROM relation WHERE brand=X AND country=Y AND type IN (O,P,Q,R) GROUP BY type

21. Estimates Approach score(O)=1254, score(P)=432, score(Q)=2050, score(R)=1990 TypeScore A3452 B2406 C2356 D2167 E1987 top-2 top-5 TypeScore O1990 P2112 Q2076 R1997 Buffer TypeScore T2287 TypeScore A3452 B2406 C2356 D2167 Q2050 Case 3: type NOT in in-memory structures, e.g. (T,X,Y,300)

22. Queries Characteristics SAME primary attribute SAME aggregate attributes SAME aggregate function SAME top-k condition DIFFERENT filtering condition

23. Lattice organisation

24. Groups Approach The updates are forwarded from top to bottom in the lattice Each ranking forwards the queried results to the rankings lying in lower levels in the lattice

25. Groups Approach: Example SELECT type, SUM(price*quantity) FROM relation WHERE brand=X GROUP BY type ORDER BY SUM(price*quantity) LIMIT 2 Update: (type, X, Y, 300) Ranking: brand=X, country=ANY

26. Groups Approach Case 2: type in the Buffer, e.g. (P,X,Y,300) TypeScore A3452 B2406 C2356 D2167 E1987 top-2 top-5 TypeScore O1990 P2112 Q2076 R1997 Buffer +300 TypeScore O1990 P2412 Q2076 R1997 Buffer Verification Query

27. Groups Approach Case 2: type in the Buffer, e.g. (P,X,Y,300) Verification Query: SELECT type, brand, country, price*quantity FROM relation WHERE brand=X AND type=P Buffer Reset Query: SELECT type, brand, country, price*quantity FROM relation WHERE brand=X AND type IN (O,P,Q,R) Case 4: type NOT in in-memory structures, e.g. (T,X,Y,300)

28. Groups Approach Tuples (type, brand, country, price*quantity) limited to those satisfying its filtering condition Uses them to compute the scores. Forwards them to the rankings lower in lattice Rankings receiving tuples use those qualifying to their filtering condition to compute the scores

29. Groups Approach: Verification Query SELECT brand, country, price*quantity FROM relation WHERE brand=X AND type=P A set with (brand, country, price*quantity) tuples limited to those that have brand=X Uses them to compute score(P). Forwards them to the rankings lower in lattice ({brand=X, country=Y}, {brand=X, country=W})

30. Groups Approach Buffer Full Reset Query Estimated Score(T) = score(E) + 300 = 2287 SELECT type, brand, country, price*quantity FROM relation WHERE brand=X AND type IN (O,P,Q,R) Case 4: type NOT in in-memory structures, e.g. (T,X,Y,300)

31. Outline Problem Algorithms Naïve Approach Estimates Approach Groups Approach Experimental Results Conclusions

32. Experiments (1) TPC-H data Select on part.p_partKey (200,000 unique values) Filter on customer.c_mktsegment, orders.o_orderpriority and region.r_name Aggregation sum on lineitem.l_quantity 216 total rankings 30,000 updates/insertions

33. Experiments (2) Updates Random: inserts quantity between 1 and 50 for a random part.p_partKey. 80-20: inserts quantity between 1 and 50 for a part.p_partKey selected according to the 80-20 rule N-extra Gap Difference between top-k and top-(k+N) scores 100% (1*50) and 200% (2*50)

34. 80-20 Updates: Queries

35. 80-20 Updates: Time

36. Random Updates: Queries

37. Random Updates: Time

38. Naïve Approach 80-20 updates: 239,985 Verification Queries, 4 secs/update Random updates: 239,977 Verification Queries, 4 secs/update

39. Outline Problem Algorithms Naïve Approach Estimates Approach Groups Approach Experimental Results Conclusions

40. Conclusion Two algorithms to maintain top-k rankings in the presence of fast updates arriving in an underlying database Exact top-k results Faster than a Naïve approach while Groups Approach limits further the communication with the database Preliminary results which provide insights on the impact of the various parameters in the effectiveness of our methods

41. Thank you!

42. Additional Instances