1. A Black-Box Approach to Query Cardinality Estimation
Tanu Malik, Randal Burns
The Johns Hopkins University
Nitesh V. Chawla
Notre Dame University

2. The Black Box Approach
Estimate query result sizes without knowledge of
Underlying data distributions
Query execution plan
Machine learning techniques
Group queries into syntactic families (templates)
Learn in a high-dimension, complex input space
Attributes, operators, function arguments, aggregates
Partition input space
Learn regression functions in each partition
Self-tuning, self-correcting models
When compared with bottom-up estimation
Produces accurate, highly compact, and fast models
Lose ability to evaluate sub-plans
Independent in estimation process as well
Note in contrast to a bottom up approach

3. Are new techniques needed?
Working with federated and remote data sources
No access to data (privacy and performance concerns)
Many data sources (can’t keep estimates for all)
Our motivation: caching in federations
Ask the DB optimizer?
Other applications
Replica maintenance
Grid workflow
Distributed query schedulers
Economic caching framework, caching decisions rely on query result size

4. Astronomy Example Typical query Sample bottom-up plan
User-defined functions
Mathematical expressions
Sample bottom-up plan
Many sub-estimates
Toug time w/ UDFs and with transformed variables

5. The Spatial Function Executed at the backend database
Data distribution and queries in attribute domains
Function computes a range query
Showing the access pattern at execution

6. Workload Observed at Cache
Point queries in 3-dimensional space
2-d projection on attributes shown
Query result-size (log cardinality)
Cache does not know domains, only witnesses query paramters and their yields

7. Learning Query yields are k-means clustered into classes
Two-shown, typically 4-8
Two classes shown, typically 4-8

8. Learning Query yields are k-means clustered into classes
Class boundaries and regression functions
Learning techniques: model trees, classification and regression, and locally-weighted regression
Two classes shown, typically 4-8

9. Virtues of the Black Box
No errors from modeling assumptions, because it makes no assumptions
Conditional independence
Join distributions
Accurate estimates for complex queries
User-defined functions
High-dimensional queries
Multi-way joins
Point queries
Performance (later)

10. Drawbacks of the Black Box
Semantic losses
Does not use indexes, uniqueness, constraints
When available, treat as exceptions
Not integrated with query execution plans
No sub-plan estimates
No what-if scenarios can be explored
Parallel execution
Operator re-ordering
Not naturally suited to the database optimization
It’s a middleware technique

11. Overview of Results
How many trees?
How accurate?

12. Space and Time
How big?
How fast?

13. A Black-Box Approach to Query Cardinality Estimation
Tanu Malik, Randal Burns
The Johns Hopkins University
Nitesh V. Chawla
Notre Dame University

14. Quick Comparison Self-tuning histograms, e.g. STHoles, STGrid, others
Machine learning, self-tuning, based on observed workload
Produce an estimated data distribution
Histograms limited to range queries
Costing User-Defined Functions [He et al. 2005]
Estimate based on weighted nearest k-neighbors
Restricted to function arguments
Does not build a model
The Black Box approach
Data independent in both inputs and estimation
Rich input space: enumerated domains, operators, and aggregates
Compact models, summary data structures