Optimal Aggregation Algorithms for Middleware By Ronald Fagin, Amnon Lotem, and Moni Naor

Overview  Databases and data types  Fagin’s Algorithm  Threshold Algorithm  Advantages

Multimedia vs String  Early databases  Modern and Middleware databases  “fuzzy” attributes  Querying a database (x, g)

Naïve Algorithm  Find the top 2 objects R1R1 X1X1 1 X2X2 0.8 X3X3 0.5 X4X4 0.3 X5X5 0.1 R2R2 X2X2 0.8 X3X3 0.7 X1X1 0.3 X4X4 0.2 X5X5 0.1 R3R3 X4X4 0.8 X3X3 0.6 X1X1 0.2 X5X5 0.1 X2X2 0

R1R1 X1X1 1 X2X2 0.8 X3X3 0.5 X4X4 0.3 X5X5 0.1 R2R2 X2X2 0.8 X3X3 0.7 X1X1 0.3 X4X4 0.2 X5X5 0.1 R3R3 X4X4 0.8 X3X3 0.6 X1X1 0.2 X5X5 0.1 X2X2 0 Naïve Algorithm

R1R1 X1X1 1 X2X2 0.8 X3X3 0.5 X4X4 0.3 X5X5 0.1 R2R2 X2X2 0.8 X3X3 0.7 X1X1 0.3 X4X4 0.2 X5X5 0.1 R3R3 X4X4 0.8 X3X3 0.6 X1X1 0.2 X5X5 0.1 X2X2 0 Naïve Algorithm X1X1 1.5

R1R1 X1X1 1 X2X2 0.8 X3X3 0.5 X4X4 0.3 X5X5 0.1 R2R2 X2X2 0.8 X3X3 0.7 X1X1 0.3 X4X4 0.2 X5X5 0.1 R3R3 X4X4 0.8 X3X3 0.6 X1X1 0.2 X5X5 0.1 X2X2 0 Naïve Algorithm X1X1 1.5 X2X2 1.6

R1R1 X1X1 1 X2X2 0.8 X3X3 0.5 X4X4 0.3 X5X5 0.1 R2R2 X2X2 0.8 X3X3 0.7 X1X1 0.3 X4X4 0.2 X5X5 0.1 R3R3 X4X4 0.8 X3X3 0.6 X1X1 0.2 X5X5 0.1 X2X2 0 Naïve Algorithm X1X1 1.5 X2X2 1.6 X3X3 1.8

R1R1 X1X1 1 X2X2 0.8 X3X3 0.5 X4X4 0.3 X5X5 0.1 R2R2 X2X2 0.8 X3X3 0.7 X1X1 0.3 X4X4 0.2 X5X5 0.1 R3R3 X4X4 0.8 X3X3 0.6 X1X1 0.2 X5X5 0.1 X2X2 0 Naïve Algorithm X1X1 1.5 X2X2 1.6 X3X3 1.8 X4X4 1.3

R1R1 X1X1 1 X2X2 0.8 X3X3 0.5 X4X4 0.3 X5X5 0.1 R2R2 X2X2 0.8 X3X3 0.7 X1X1 0.3 X4X4 0.2 X5X5 0.1 R3R3 X4X4 0.8 X3X3 0.6 X1X1 0.2 X5X5 0.1 X2X2 0 Naïve Algorithm X1X1 1.5 X2X2 1.6 X3X3 1.8 X4X4 1.3 X5X5 0.3

Naïve Algorithm X3X3 1.8 X2X2 1.6 X1X1 1.5 X4X4 1.3 X5X5 0.3 Top-2 objects

Fagin’s Algorithm  Sequential access in parallel until k matches  Perform random access  Compute the grade for each R object R1R1 X1X1 1 X2X2 0.8 X3X3 0.5 X4X4 0.3 X5X5 0.1 R2R2 X2X2 0.8 X3X3 0.7 X1X1 0.3 X4X4 0.2 X5X5 0.1 R3R3 X4X4 0.8 X3X3 0.6 X1X1 0.2 X5X5 0.1 X2X2 0

Fagin’s Algorithm  Sequential Access R1R1 X1X1 1 X2X2 0.8 X3X3 0.5 X4X4 0.3 X5X5 0.1 R2R2 X2X2 0.8 X3X3 0.7 X1X1 0.3 X4X4 0.2 X5X5 0.1 R3R3 X4X4 0.8 X3X3 0.6 X1X1 0.2 X5X5 0.1 X2X2 0

Fagin’s Algorithm  Sequential Access R1R1 X1X1 1 X2X2 0.8 X3X3 0.5 X4X4 0.3 X5X5 0.1 R2R2 X2X2 0.8 X3X3 0.7 X1X1 0.3 X4X4 0.2 X5X5 0.1 R3R3 X4X4 0.8 X3X3 0.6 X1X1 0.2 X5X5 0.1 X2X2 0

Fagin’s Algorithm Since k=2, and X 1 and X 3 have been seen in all lists  Sequential Access R1R1 X1X1 1 X2X2 0.8 X3X3 0.5 X4X4 0.3 X5X5 0.1 R2R2 X2X2 0.8 X3X3 0.7 X1X1 0.3 X4X4 0.2 X5X5 0.1 R3R3 X4X4 0.8 X3X3 0.6 X1X1 0.2 X5X5 0.1 X2X2 0

Fagin’s Algorithm  Perform random accesses to obtain the scores of all seen objects R1R1 X1X1 1 X2X2 0.8 X3X3 0.5 X4X4 0.3 X5X5 0.1 R2R2 X2X2 0.8 X3X3 0.7 X1X1 0.3 X4X4 0.2 X5X5 0.1 R3R3 X4X4 0.8 X3X3 0.6 X1X1 0.2 X5X5 0.1 X2X2 0

Fagin’s Algorithm  Compute score for all objects and return top k R1R1 X1X1 1 X2X2 0.8 X3X3 0.5 X4X4 0.3 X5X5 0.1 R2R2 X2X2 0.8 X3X3 0.7 X1X1 0.3 X4X4 0.2 X5X5 0.1 R3R3 X4X4 0.8 X3X3 0.6 X1X1 0.2 X5X5 0.1 X2X2 0 X3X3 1.8 X2X2 1.6 X1X1 1.5 X4X4 1.3

Threshold Algorithm  Sequential access for top k matches  Define threshold value τ  Find all seen object and compute scores  Maintain list of top k objects  Continue until top-k >= τ  Output graded set

Threshold Algorithm  Sequential access R1R1 X1X1 1 X2X2 0.8 X3X3 0.5 X4X4 0.3 X5X5 0.1 R2R2 X2X2 0.8 X3X3 0.7 X1X1 0.3 X4X4 0.2 X5X5 0.1 R3R3 X4X4 0.8 X3X3 0.6 X1X1 0.2 X5X5 0.1 X2X2 0

Threshold Algorithm  Set τ to be the aggregate of the scores seen in this access R1R1 X1X1 1 X2X2 0.8 X3X3 0.5 X4X4 0.3 X5X5 0.1 R2R2 X2X2 0.8 X3X3 0.7 X1X1 0.3 X4X4 0.2 X5X5 0.1 R3R3 X4X4 0.8 X3X3 0.6 X1X1 0.2 X5X5 0.1 X2X2 0 Τ = 2.6

Threshold Algorithm  Random access and compute scores R1R1 X1X1 1 X2X2 0.8 X3X3 0.5 X4X4 0.3 X5X5 0.1 R2R2 X2X2 0.8 X3X3 0.7 X1X1 0.3 X4X4 0.2 X5X5 0.1 R3R3 X4X4 0.8 X3X3 0.6 X1X1 0.2 X5X5 0.1 X2X2 0 Τ = 2.6 X1X1 1.5 X2X2 1.6 X4X4 1.3 Top-k

Threshold Algorithm  Sequential access R1R1 X1X1 1 X2X2 0.8 X3X3 0.5 X4X4 0.3 X5X5 0.1 R2R2 X2X2 0.8 X3X3 0.7 X1X1 0.3 X4X4 0.2 X5X5 0.1 R3R3 X4X4 0.8 X3X3 0.6 X1X1 0.2 X5X5 0.1 X2X2 0 X1X1 1.5 X2X2 1.6 X4X4 1.3 Top-k

Threshold Algorithm  Set τ to be the aggregate of the scores seen in this access R1R1 X1X1 1 X2X2 0.8 X3X3 0.5 X4X4 0.3 X5X5 0.1 R2R2 X2X2 0.8 X3X3 0.7 X1X1 0.3 X4X4 0.2 X5X5 0.1 R3R3 X4X4 0.8 X3X3 0.6 X1X1 0.2 X5X5 0.1 X2X2 0 Τ = 2.1 X1X1 1.5 X2X2 1.6 X4X4 1.3 Top-k

Threshold Algorithm  Random access and compute scores R1R1 X1X1 1 X2X2 0.8 X3X3 0.5 X4X4 0.3 X5X5 0.1 R2R2 X2X2 0.8 X3X3 0.7 X1X1 0.3 X4X4 0.2 X5X5 0.1 R3R3 X4X4 0.8 X3X3 0.6 X1X1 0.2 X5X5 0.1 X2X2 0 Τ = 2.1 X1X1 1.5 X2X2 1.6 X4X4 1.3 X3X3 1.8 Top-k

X1X1 1.5 X2X2 1.6 X4X4 1.3 X3X3 1.8 Top-k Threshold Algorithm  Sequential Access R1R1 X1X1 1 X2X2 0.8 X3X3 0.5 X4X4 0.3 X5X5 0.1 R2R2 X2X2 0.8 X3X3 0.7 X1X1 0.3 X4X4 0.2 X5X5 0.1 R3R3 X4X4 0.8 X3X3 0.6 X1X1 0.2 X5X5 0.1 X2X2 0

X1X1 1.5 X2X2 1.6 X4X4 1.3 X3X3 1.8 Top-k Threshold Algorithm  Set τ to be the aggregate of the scores seen in this access R1R1 X1X1 1 X2X2 0.8 X3X3 0.5 X4X4 0.3 X5X5 0.1 R2R2 X2X2 0.8 X3X3 0.7 X1X1 0.3 X4X4 0.2 X5X5 0.1 R3R3 X4X4 0.8 X3X3 0.6 X1X1 0.2 X5X5 0.1 X2X2 0 Τ = 1

Threshold Algorithm  Stop when top-k >= τ R1R1 X1X1 1 X2X2 0.8 X3X3 0.5 X4X4 0.3 X5X5 0.1 R2R2 X2X2 0.8 X3X3 0.7 X1X1 0.3 X4X4 0.2 X5X5 0.1 R3R3 X4X4 0.8 X3X3 0.6 X1X1 0.2 X5X5 0.1 X2X2 0 Τ = 1 X1X1 1.5 X2X2 1.6 X4X4 1.3 X3X3 1.8 Top-k

Comparison  Naïve Algorithm Buffer space required = number of objects The cost is linear Not efficient for large databases

Comparison  Fagin’s Algorithm Large buffer space required Random access is done at the end Optimal under certain aggregate functions

Comparison  The Threshold Algorithm Buffer space bounded by k Objects not seen < τ Less object access required Always optimal

Sources  Query.pdf Query.pdf  earcher/files/us-fagin/jcss03.pdf earcher/files/us-fagin/jcss03.pdf