Presentation on theme: "Modern Performance - SQL Server Joe Chang yahoo."— Presentation transcript:
Modern Performance - SQL Server Joe Chang yahoo
About Joe SQL Server consultant since 1999 Query Optimizer execution plan cost formulas (2002) True cost structure of SQL plan operations (2003?) Database with distribution statistics only, no data 2004 Decoding statblob/stats_stream – writing your own statistics Disk IO cost structure Tools for system monitoring, execution plan analysis See ExecStats Download: Blog:
Overview Why performance is still important today? – Brute force? Yes, but … Special Topics Automating data collections SQL Server Engine – What developers/DBA need to know?
CPU & Memory 2001 versus – 4 sockets, 4 cores Pentium III Xeon, 900MHz 4-8GB memory? Xeon MP – 4 sockets, 8 cores each 4 x 8 = 32 cores total Westmere-EX 1TB (64x16GB) Sandy Bridge E5: 768GB (48 x 16GB), 15 cores in Xeon E7 v2 3TB (96 x 32GB) FSB P L2 PP P MCH QPI PCI-E MI PCI-E C1C6 C2C5 C3C4 LLC QPI MI C7C0 QPI DMI 2 PCI-E MI PCI-E C1C6 C2C5 C3C4 LLC QPI MI C7C0 QPI PCI-E MI PCI-E C1C6 C2C5 C3C4 LLC QPI MI C7C0 PCI-E MI PCI-E C1C6 C2C5 C3C4 LLC QPI MI C7C0 Each core today is more than 10x over PIII _____ 2013 __ GB $191 __ $180 32GB $794 __ $650
CPU & Memory 2001 versus 2014 QPI DMI 2 PCI-E QPI PCI-E QPI MI PCI-E MI C1 C2 C3 C0 C4 C8 C7 C6 C9 C5 LLC C8 C7 C6 C9 C5 QPI MI PCI-E MI C1 C2 C3 C0 C4 C8 C7 C6 C9 C5 LLC C8 C7 C6 C9 C5 QPI MI PCI-E MI C1 C2 C3 C0 C4 C8 C7 C6 C9 C5 LLC C8 C7 C6 C9 C5 QPI MI PCI-E MI C1 C2 C3 C0 C4 C8 C7 C6 C9 C5 LLC C8 C7 C6 C9 C5 Xeon E7 v2 (Ivy Bridge) 4 x 15 = 60 cores 3TB (96 x 32GB) 24 DIMMs per socket (12 shown) 2001 – 4 sockets, 4 cores Pentium III Xeon, 900MHz 4-8GB memory? Xeon MP FSB P L2 PP P MCH Each core today is more than 10x over Pentium III (700MHz?) Mem___2013 __ GB __ $191 __ $180 32GB __ $794 __ $650
SAN Node GB Node GB Switch SP ASP B 8 Gb FC 24 GB x4 SAS 2GB/s SSD x8 SSD x8 Data 5Data 6Data 7 Data 1Data 2Data 3Data 4 Data 8 Data 9 Data 13 Data 10 Data 14 Data 11 Data 15 Data 12 Data 16 SSD 1SSD 2SSD 3SSD 4 Log 1Log 2Log 3Log 4 Log volume SSD10K7.2K Hot Spares Node 1Node 2 Switch SP ASP B 8 Gbps FC or 10Gbps FCOE 768 GB x4 SAS 2GB/s 24 GB Main Volume
Performance Past, Present, Future When will servers be so powerful that … – Been saying this for a long time Today – 10 to 100X overkill – 32-cores in 2012, 60-cores in 2014 – Enough memory that IO is only sporadic – Unlimited IOPS with SSD What can go wrong? Today’s topic
Special Topics Data type mismatch Multiple Optional Search Arguments (SARG) – Function on SARG Parameter Sniffing versus Variables Statistics related (big topic) first OR, then AND/OR combinations Complex Query with sub-expressions Parallel Execution Not in order of priority
1a. Data type mismatch nvarchar(25) = N'Customer# ' SELECT * FROM CUSTOMER WHERE C_NAME SELECT * FROM CUSTOMER WHERE C_NAME = auto-parameter discovery? Unable to use index seek Table column is varchar Parameter/variable is nvarchar
1b. Type Mismatch – Row Estimate SELECT * FROM CUSTOMER WHERE C_NAME LIKE 'Customer# %' SELECT * FROM CUSTOMER WHERE C_NAME LIKE N’Customer# %' Row estimate error could have severe consequences in a complex query
SELECT TOP + Row Estimate Error SELECT TOP 1000 [Document].[ArtifactID] FROM [Document] (NOLOCK) WHERE [Document].[AccessControlListID_D] IN (1, , ) AND EXISTS ( SELECT [DocumentBatch].[BatchArtifactID] FROM [DocumentBatch] (NOLOCK) INNER JOIN [Batch] (NOLOCK) ON [Batch].ArtifactID = [DocumentBatch].[BatchArtifactID] WHERE [DocumentBatch].[DocumentArtifactID] = [Document].[ArtifactID] AND [Batch].[Name] LIKE N'%Value%' ) ORDER BY [Document].[ArtifactID] Data type mismatch – results in estimate rows high Top clause – easy to find first 1000 rows In fact, there are few rows that match SARG Wrong plan for evaluating large number of rows
2. Multiple Optional SARG int = 1 SELECT * FROM LINEITEM WHERE IS NULL OR L_ORDERKEY AND IS NULL OR L_PARTKEY AND IS NOT NULL IS NOT NULL)
IF block int = 1 IF IS NOT NULL) SELECT * FROM LINEITEM WHERE (L_ORDERKEY AND IS NULL OR L_PARTKEY ELSE IF IS NOT NULL) SELECT * FROM LINEITEM WHERE (L_PARTKEY Need to consider impact of Parameter Sniffing, Consider the OPTIMIZER FOR hint These are actually the stored procedure parameters
Dynamically Built Parameterized SQL int = nvarchar(100) = N‘/* Comment */ SELECT * FROM LINEITEM WHERE = int' IF IS NOT NULL) + N' AND L_ORDERKEY IF IS NOT NULL) + N' AND L_PARTKEY IF block is easier for few options Dynamically built parameterized SQL better for many options Consider /*comment*/ to help identify source of SQL
2b. Function on column SARG SELECT COUNT(*), SUM(L_EXTENDEDPRICE) FROM LINEITEM WHERE YEAR(L_SHIPDATE) = 1995 AND MONTH(L_SHIPDATE) = 1 SELECT COUNT(*), SUM(L_EXTENDEDPRICE) FROM LINEITEM WHERE L_SHIPDATE BETWEEN ' ' AND ' ' int = 1 SELECT COUNT(*), SUM(L_EXTENDEDPRICE) FROM LINEITEM WHERE L_SHIPDATE AND
Estimated versus Actual Plan - rows Estimated Plan – 1 row??? Actual Plan – actual rows 77,356
3 Parameter Sniffing -- first call, procedure compiles with these parameters exec = = ' ' -- subsequent calls, procedure executes with original plan exec = = ' ' Need different execution plans for narrow and wide range Options: 1) WITH RECOMPILE 2) main procedure calls 1 of 2 identical sub-procedures One sub-procedure is only called for narrow range Other called for wide range Skewed data distributions also important Example: Large & small customers Assuming date data type
4 Statistics Auto-recompute points Sampling strategy – How much to sample - theory? – Random pages versus random rows – Histogram Equal and Range Rows – Out of bounds, value does not exist – etc. Statistics Used by the Query Optimizer in SQL Server 2008 Writer: Eric N. Hanson and Yavor Angelov Contributor: Lubor Kollar
Statistics Structure Stored (mostly) in binary field Scalar values Density Vector – limit 30, half in NC, half Cluster key Histogram Up to 200 steps Consider not blindly using IDENTITY on critical tables Example: Large customers get low ID values Small customers get high ID values
Statistics Auto/Re-Compute Automatically generated on query compile Recompute at 6 rows, 500, every 20%? Has this changed?
Statistics Sampling Sampling theory – True random sample – Sample error - square root N Relative error 1/ N SQL Server sampling – Random pages But always first and last page??? – All rows in selected pages
Row Estimate Problems Skewed data distribution Out of bounds Value does not exist
Loop Join - Table Scan on Inner Source Estimated out from first 2 tabes (at right) is zero or 1 rows. Most efficient join to third table (without index on join column) is a loop join with scan. If row count is 2 or more, then a fullscan is performed for each row from outer source Default statistics rules may lead to serious ETL issues Consider custom strategy
Compile Parameter Not Exists Main procedure has cursor around view_Servers First server in view_Servers is ’CAESIUM’ Cursor executes sub-procedure for each Server sql: SELECT MAX(ID) FROM TReplWS WHERE Hostname But CAESIUM does not exist in TReplWS!
Good and Bad Plan?
SqlPlan Compile Parameters
31 Compile parameter values at bottom of sqlplan file
More Plan Details Query with joining 6 tables Each table has too many indexes Row estimate is high – plan cost is high Query optimizer tries really really hard to find better plan Actual rows is moderate, any plan works
5a Single Table OR -- Single table SELECT * FROM LINEITEM WHERE L_ORDERKEY = 1 OR L_PARTKEY =
5a Join 2 Tables, OR in SARG -- subsequent calls, procedure executes with original plan SELECT O_ORDERDATE, O_ORDERKEY, L_SHIPDATE, L_QUANTITY FROM LINEITEM INNER JOIN ORDERS ON O_ORDERKEY = L_ORDERKEY WHERE L_PARTKEY = OR O_CUSTKEY =
5a UNION (ALL) instead of OR SELECT O_ORDERDATE, O_ORDERKEY, L_SHIPDATE, L_QUANTITY, O_CUSTKEY, L_PARTKEY FROM LINEITEM INNER JOIN ORDERS ON O_ORDERKEY = L_ORDERKEY WHERE L_PARTKEY = UNION (ALL) SELECT O_ORDERDATE, O_ORDERKEY, L_SHIPDATE, L_QUANTITY, O_CUSTKEY, L_PARTKEY FROM LINEITEM INNER JOIN ORDERS ON O_ORDERKEY = L_ORDERKEY WHERE O_CUSTKEY = AND (L_PARTKEY <> OR L_PARTKEY IS NULL) -- Caution: select list should have keys to ensure correct rows UNION removes duplicates (with Sort operation) UNION ALL does not -- Hugo Kornelis trick --
5b AND/OR Combinations Hash Join is good method to process many rows – Requirement is equality join condition In complex SQL with AND/OR or IN NOT IN combinations – Query optimizer may not be to determine that equality join condition exists – Execution plan will use loop join, – and attempt to force hash join will be rejected Re-write using UNION in place of OR And LEFT JOIN in place of NOT IN SELECT xx FROM A WHERE col1 IN (expr1) AND col2 NOT IN (expr2) SELECT xx FROM A WHERE (expr1) AND (expr2 OR expr3) More on AND/OR combinations:
Complex Query with Sub-expression Query complexity – really high compile cost Repeating sub-expressions (including CTE) – Must be evaluated multiple times Main Problem - Row estimate error propagation Solution/Strategy – Get a good execution plan – Temp table when estimate is high, actual is low. More on AND/OR combinations: When Estimate is low, and actual rows is high, need to balance temp table insert overhead versus plan benefit. Would a join hint work?
Temp Table and Table Variable Forget what other people have said – Most is Temp Tables – subject to statistics auto/re-compile Table variable – no statistics, assumes 1 row Question: In each specific case: does the statistics and recompile help or not? – Yes: temp table – No: table variable
Parallelism Designed for 1998 era – Cost Threshold for Parallelism: default 5 – Max Degree of Parallelism – instance level – OPTION (MAXDOP n) – query level Today – complex system – 32 cores – Plan cost 5 query might run in 10ms? – Some queries at DOP 4 – Others at DOP 16? More on Parallelism: Really need to rethink parallelism / NUMA strategies Number of concurrently running queries x DOP less than number of logical/physical processors?
Full-Text Search Loop Join with FT as inner Source Full Text search Potentially executed many times
varchar(max) stored in lob pages Disk IO to lob pages is synchronous? – Must access row to get 16 byte link? – Feature request: index pointer to lob SQL PASS 2013 Understanding Data Files at the Byte Level Mark Rasmussen
Summary Hardware today is really powerful – Storage may not be – SAN vendor disconnect Standard performance practice – Top resource consumers, index usage But also Look for serious blunders
Special Topics Data type mismatch Multiple Optional Search Arguments (SARG) – Function on SARG Parameter Sniffing versus Variables Statistics related (big topic) AND/OR Complex Query with sub-expressions Parallel Execution
SQL Server Edition Strategies Enterprise Edition – per core licensing costs – Old system strategy 4 (or 2)-socket server, top processor, max memory – Today: How many cores are necessary 2 socket system, max memory (16GB DIMMs) Is standard edition adequate – Low cost, but many important features disabled BI edition – 16 cores – Limited to 64GB for SQL Server process
New Features in SQL Server 2005 – Index included columns – Filtered index – CLR 2008 – Partitioning – Compression 2012 – Column store (non-clustered) 2014 – Column store clustered – Hekaton