Download presentation
Presentation is loading. Please wait.
1
GSLPI: a Cost-based Query Progress Indicator
Jiexing Li#, Rimma Nehme*, Jeff Naughton# #University of Wisconsin-Madison *Microsoft Jim Gray Systems Lab
2
Progress indicator (PI)
A PI provides feedback to users on how much of the task has been completed or when the task will finish. It is useful for workload management, admission control, skew handling, etc.
3
Main results Existing PIs generate inaccurate results, if a query has phases with different performance characteristics. A new PI that provides more accurate predictions by using optimizerβs cost estimates, and deeper analysis of query pipelines.
4
Execution plan Blocking operator: no outputs until the operator has consumed at least one of its inputs (e.g., sort and hash match) Pipeline: a subtree of concurrently executing operators delimited by blocking operators Table Scan [A] Filter Hash Match Scan [B] Hash Match Pipeline: P2 Filter Table Scan [B] Execution plan: a tree where the nodes are physical operators Blocking operator: an operator which does not produce any outputs until it has consumed at least one of its inputs (sort, hash match) Pipeline: a maximum subtree of concurrently executing operators Driving node: a set of all leaf nodes of the pipeline (except those that are in the inner subtree of a nested loop join) Pipeline: P1 Table Scan [A] An execution plan
5
Basic ideas of previous PIs
Pipelines are processed one after another. Each of them processes a certain number of tuples. Speed: how fast tuples can be processed. Estimated cardinalities Current speed (future speed is unknown) Time P1 P2 Pn β¦ Cardinality N1 Cardinality N2 Cardinality Nn Estimated remaining time = πππ‘ππ ππ’ππππ ππ π’ππππππ βππ π‘π’ππππ πππππ
6
Limitations of previous PIs
select count(distinct orderkey) from lineitem Compute Scalar P2 Hash Match (orderkey) Table Scan [lineitem] P1 P1 P2 This query contains 2 pipelines: one scans the lineitem table (slow), one does the aggregation (fast). Execution plan Test results for TPC-H 10GB data
7
Limitations of Previous PIs (cont.)
select count(distinct orderkey) from lineitem Before P2 starts, only the processing speed for P1 is known. Over estimates the execution time for P2 by using the current speed of P1 for future pipeline P2. P1 P2 This query contains 2 pipelines: one scans the lineitem table (slow), one does the aggregation (fast).
8
GSLPI improvement ? Different phases have different processing speeds.
How does GSLPI estimate the processing speeds for future pipelines? P2 P1 Cardinality N1 Cardinality N2 Speed S1 Speed S2 ?
9
GSLPI: a cost-based progress indicator
Pn β¦ Cardinality N1 Cardinality N2 Cardinality Nn Further division (uniform speed) P1 P2 Pβ2 Pn β¦ N1 N2 Nβ2 Nn ? Cost: work to process a tuple Cost C1 Cost C2 Cost Cβ2 Cost Cn ? The wall-clock pipeline cost is the maximum amount of non-overlapping CPU and I/O done by a pipeline from its beginning until the end. Details for calculating the speed: in the next hidden slide. Speed: adjusts based on the cost Speed S1 Speed S2 Speed Sβ2 Speed Sn Estimates the remaining time Estimated T1 Estimated T2 Estimated Tβ2 Estimated Tn
10
GSLPI: Cost Redistributes the cost for operators on the boundaries.
select count(distinct orderkey) from lineitem Redistributes the cost for operators on the boundaries. P1 scans a tuple and inserts it into the hash table. P2 fetches a tuple from the hash table and counts. Splits the cost for Hash Match operator. Identifies bottleneck cost. Due to concurrent execution. Estimated [CPU, IO] Compute Scalar [0, 0] P2 Hash Match (orderkey) [275, 0] Table Scan [lineitem] P1 [66, 767] Execution plan
11
GSLPI: Speed select count(distinct orderkey) from lineitem
P2 should be able to process its tuples much faster than P1. Adjusts the speeds for future pipelines based on how difficult it is to process a tuple. Compute Scalar P2 Hash Match (orderkey) Table Scan [lineitem] P1 Execution plan
12
Experimental evaluation
Database: 10GB TPC-H. Setup: Intel Core 2 Duo CPU and 4GB RAM. Tested on 22 TPC-H queries. They usually contain 6 to 16 operators, which may be divided into 3 to 9 different pipelines. Estimated the remaining time in every 10 seconds.
13
Example of progress estimation
P6, P7 P1, P2, P3 Experiment results for TPC-H Q5
14
Error metric Similar to Q5, 20 out of 22 TPC-H queries process tuples at different speeds during their executions. Estimated percentage of completion: Error: the distance between fi and the actual percentage of completion. For each query, we calculated the average error and the maximum error. fi= ππππ π ππππ‘ πΈπ π‘ππππ‘ππ ππ₯πππ’π‘πππ π‘πππ
15
GSLPI accuracy Query Avg. Err. Max. Err. Q1 0.50% 0.60% Q12 10.50%
24.70% Q2 0.90% 1.90% Q13 1.20% 3.50% Q3 4.00% 10.40% Q14 1.00% 2.20% Q4 1.50% 8.50% Q15 0.10% Q5 2.40% 10.30% Q16 1.60% Q6 0.30% 0.70% Q17 Q7 3.60% 9.40% Q18 Q8 3.20% 8.20% Q19 0.20% Q9 1.30% 6.30% Q20 16.80% 41.10% Q10 1.70% 7.70% Q21 5.70% 16.00% Q11 1.10% Q22 Due to cardinality estimation error!
16
Conclusions Investigated the limitations with previous PIs.
Developed a new progress indicator: GSLPI. Evaluated their performance. More accurate than previous PIs. Remaining progress estimation error for TPC-H queries is mostly due to cardinality estimation error!
17
Thank you!
Similar presentations
