1. SQL on Fire! Part 1 Tips and Tricks around SQL

2. Agenda  Part I  SQL vs. SQL PL example  Error handling  Tables out of nowhere  Pivoting  Aggregation  Deleting duplicate rows  Alternatives to Min and Max  Part II  Mass deletes  Order processing  Moving rows between tables  Recursion  Merge  Calculating nesting levels Easy Difficult

3. Motivation – The OLTP Mantra  Reduce Codepath  Reduce Logical and Physical I/O  Reduce Network Traffic  Minimize Lock Contention  Avoid Deadlocks High performance starts with the application.

4. SQL vs. SQL PL - Example  DDL CREATE TABLE emp(empID INTEGER NOT NULL salary INTEGER, name VARCHAR(20), deptID INTEGER); CREATE UNIQUE INDEX empIdxPK ON emp(empID) INCLUDE (deptID); ALTER TABLE emp ADD PRIMARY KEY (empID);

5. SQL vs. SQL PL EmpIDSalaryNameDeptID 110000Jones100 212000Wang200 311000Zhang100 420000Smith300 515000Kim200 613000Chen400 714000Jou300 811000Yung100 916000Hall400 1014000Zhu100 1113000Xia200

6. SQL vs. SQL PL  Business request: Give everyone in deptID 100 a 5% raise.

7. SQL vs. SQL PL – First Attempt DECLARE empcur CURSOR FOR SELECT * FROM emp FOR UPDATE; OPEN empcur; emploop: LOOP FETCH empcur INTO vempid, vsalary, vname, vdeptid; IF SQLSTATE = ‘02000’ THEN LEAVE emploop; END IF; IF deptid = 100 THEN SET vsalary = vsalary * 1.05; UPDATE emp SET salary = vsalary WHERE CURRENT OF empcur; END IF; END LOOP emploop; CLOSE empcur; Read whole table Only need few rows

8. SQL vs. SQL PL - Second Attempt DECLARE empcur CURSOR FOR SELECT * FROM emp WHERE deptID = 100 FOR UPDATE; OPEN empcur; emploop: LOOP FETCH empcur INTO vempid, vsalary, vname, vdeptid; SET vsalary = vsalary * 1.05; IF SQLSTATE = ‘02000’ THEN LEAVE emploop; END IF; UPDATE emp SET salary = vsalary WHERE CURRENT OF empcur; END LOOP emploop; CLOSE empcur; Retrieve all columns Use only one column

9. SQL vs. SQL PL - Third Attempt DECLARE empcur CURSOR FOR SELECT salary FROM emp WHERE deptID = 100 FOR UPDATE; OPEN empcur; emploop: LOOP FETCH empcur INTO vsalary; SET vsalary = vsalary * 1.05; IF SQLSTATE = ‘02000’ THEN LEAVE emploop; END IF; UPDATE emp SET salary = vsalary WHERE CURRENT OF empcur; END LOOP emploop; CLOSE empcur; Use cursor Trivial logic

10. SQL vs. SQL PL - Final Attempt UPDATE emp SET salary = salary * 1.05 WHERE deptID = 100; Tips: WHERE deptID IN (100, 200) WHERE deptID IN (SELECT deptID FROM dept WHERE deptlocation = ‘Shenzhen’) SET salary = salary * CASE WHEN name = ‘Zhang’ THEN 1.05 ELSE 1.03 END

11. SQL vs. SQL PL – Aggregation  Return all departments with more than 5 employees: SELECT deptID FROM emp GROUP BY deptID HAVING COUNT(*) > 5  Return the salary of the most highly paid employee by department for departments with more than 5 employees: SELECT deptID, MAX(salary) FROM emp GROUP BY deptID HAVING COUNT(*) > 5

12. Create table if it does not exist BEGIN DECLARE CONTINUE HANDLER FOR SQLSTATE '42710' BEGIN /* Already exists? Ignore */ END; CREATE TABLE T(c1 INT); END Tip 1: There is no need to avoid SQL errors as long as they are handled… Tip 2:.. except when statement rollback implies undo operation from the log

13. SQL PL – DDL cleanup BEGIN DECLARE CONTINUE HANDLER FOR SQLSTATE '42704' BEGIN /* Does not exist? Ignore */ END; DROP TABLE T1; DROP TABLE T2; DROP SEQUENCE S1; END Tip: Lost errors are very hard to find. So be specific and never ignore generic SQLEXCEPTION.

14. SQL PL – Is this table empty? IF EXISTS(SELECT 1 FROM emp WHERE dept = 100) THEN.. END IF; Instead of: IF EXISTS(SELECT 1 FROM emp WHERE empid = 5) THEN UPDATE emp SET salary = salary * 1.05 WHERE empID = 5; END IF; Use: UPDATE emp SET salary = salary * 1.05 WHERE empID = 5; Tip: Don’t be afraid of NOTFOUND (SQLSTATE ‘02000’). NOT FOUND not an error

15. Tables from nowhere  Create tables without INSERT: VALUES (1, 2), (3, 4), …. (10, 11) or VALUES (CAST(? AS INTEGER), CAST(? AS DOUBLE)), (?, ?), … (?, ?)  Use anywhere like INSERT INTO T VALUES (1, 2), (3, 4), (5, 6)

16. VALUES for multi row INSERT CREATE TABLE T(c1 INT, c2 VARCHAR(10)); INSERT INTO T VALUES (?, ?), (?, ?), (?, ?), … (?, ?); Tip: For mass inserts prepare INSERT statements with: 1, 10, 100, 1000, 10000 rows of parameter markers each. Execute the biggest that fits the remaining load in a loop.

17. PIVOT and UNPIVOT YearQuarterResults 2004120 2004230 2004315 2004410 2005118 2005240 2005312 2005427 CREATE TABLE Sales(Year INTEGER, Quarter INTEGER, Results INTEGER);

18. PIVOT and UNPIVOT SalesQ1Q2Q3Q4 200420301510 200518401227 CREATE TABLE SalesAgg(year INTEGER, q1 INTEGER, q2 INTEGER, q3 INTEGER, q4 INTEGER);

19. PIVOT SELECT Year, MAX(CASE WHEN Quarter = 1 THEN Results END) AS Q1, MAX(CASE WHEN Quarter = 2 THEN Results END) AS Q2, MAX(CASE WHEN Quarter = 3 THEN Results END) AS Q3, MAX(CASE WHEN Quarter = 4 THEN Results END) AS Q4 FROM Sales GROUP BY Year Tip: Use MAX() because it is supported for all comparable types including strings

20. PIVOT Access Plan: ----------- RETURN ( 1) | GRPBY ( 2) | FETCH ( 3) /----+---\ IXSCAN TABLE: SALES ( 4) | INDEX: SALESIDX

21. UNPIVOT SELECT Year, Quarter, Results FROM SalesAgg AS S, LATERAL(VALUES(1, S.q1), (2, S.q2), (3, S.q3), (4, S.q4)) AS Q(Quarter, Results); Tip: Use LATERAL (or TABLE) to allow correlation of S.q* to the left.

22. UNPIVOT Access Plan: ----------- RETURN ( 1) | NLJOIN ( 2) /------+-----\ TBSCAN TBSCAN ( 3) ( 4) | | TABLE: SALESAGG TABFNC: GENROW

23. Aggregation  Problem: DB2 does not support user defined aggregates  Thoughts: –XMLAGG() provides aggregation without loss of data –Use mathematical rules  Examples: –Aggregate concatenation –Aggregate multiplication

24. XML functions – a primer  XMLAGG() –Aggregates XML values into an XML sequence  XMLELEMENT() –Tags a scalar value and returns XML e.g. XMLELEMENT(NAME ‘x’ 5) => 5  XMLSERIALIZE() –Casts an XML value into a string

25. Aggregate concatenation CREATE TABLE Employee(name VARCHAR(15), dept VARCHAR(15)); NameDept MisoSolutions JohnDevelopment SergeSolutions LeeL3 MarkID JackL3 LilyQuality BerniSolutions

26. Aggregate concatenation DeptNames SolutionsBerni, Miso, Serge DevelopmentJohn L3Jack, Lee IDMark QualityLily

27. Aggregate concatenation SELECT Dept, SUBSTR(Names, 1, LENGTH(names) -1) FROM (SELECT Dept, REPLACE (REPLACE (XMLSERIALIZE (CONTENT XMLAGG(XMLELEMENT(NAME a, name) ORDER BY name) AS VARCHAR(60)), ' ', ''), ' ', ',') AS Names FROM Employee GROUP BY Dept) AS X; Strip last comma Replace end tags with commas Strip start tags XML to VARCHAR Aggregate in order of names Tag a name to become XML

28. Aggregate multiplication CREATE TABLE probabilities(modelVARCHAR(10), event VARCHAR(10), percent FLOAT); ModelEventPercent Boing 737Engine 10.001 Airbus 320Engine 10.0009 Boing 737Engine 20.001 Airbus 320Engine 20.0009 Airbus 320Fin0.002 Boing 737Fin0.0018

29. Aggregate multiplication  a = EXP(LOG(a)) X* Y = EXP(LOG(X * Y)) X * Y = EXP(LOG(X) + LOG(Y))  PROD(X i ) i=1..n = EXP(SUM(LOG(X i ) i=1..n )

30. Aggregate multiplication SELECT model, DEC(EXP(SUM(LOG(percent))), 8, 7) AS percent FROM probabilities WHERE event IN (‘Engine 1’, ‘Fin’) GROUP BY model ModelPercent Boing 7370.000018 Airbus 3200.000018

31. Retrieving MAXimum row CREATE TABLE emp(name VARCHAR(10), dept VARCHAR(10), salary INTEGER); CREATE INDEX emp_ind ON emp(dept, salary DESC);  Standard using selfjoin: SELECT name, salary FROM emp WHERE salary = (SELECT MAX(salary) FROM emp WHERE dept = ‘SQL Compiler’ AND dept = ‘SQL Compiler’);

32. Retrieving MAXimum row SELECT name, salary FROM emp WHERE dept = 'SQL Compiler' ORDER BY salary DESC FETCH FIRST ROW ONLY; NameDeptSalary FrankRewrite20000 HarrySQL Compiler18000 JanetSQL Compiler19000 GwenSQL Compiler22000 JasonOptimizer21000

33. Retrieving MAXimum row Access Plan: ------------ RETURN ( 1) | FETCH ( 2) /----+---\ IXSCAN TABLE: EMP ( 3) | INDEX: EMP_IND

34. Delete duplicate rows CREATE TABLE Inventory(Item INTEGER, Quantity INTEGER, InvDate DATE); CREATE UNIQUE INDEX InvIdx ON Inventory(Item ASC, InvDate DESC);

35. Delete duplicate rows ItemQuantityInvDate 13004 Oct 2003 12501 Nov 2003 210001 Nov 2003 3404 Oct 2003 3801 Nov 2003 3618 Dec 2003 41204 Oct 2003 4001 Nov 2003 52818 Dec 2003 61

36. Delete duplicate rows ItemQuantityInvDate 12501 Nov 2003 210001 Nov 2003 3618 Dec 2003 4001 Nov 2003 52818 Dec 2003 61

37. Delete duplicate rows - classic DELETE FROM Inventory AS D WHERE (Item, InvDate) NOT IN (SELECT Item, InvDate FROM Inventory I WHERE D.Item = I.Item ORDER BY InvDate DESC FETCH FIRST ROW ONLY);

38. Delete duplicate rows - improved  Using OLAP DELETE FROM (SELECT row_number() OVER(PARTITION BY Item ORDER BY InvDate DESC) AS rn FROM Inventory) WHERE rn > 1; Tip: Remove ORDER BY clause if it doesn’t matter which duplicates get eliminated.

39. Delete duplicate rows Access Plan: ------------ RETURN ( 1) | DELETE ( 2) /---+---\ FETCH TABLE: INVENTORY ( 3) /---+---\ FILTER TABLE: INVENTORY ( 4) | IXSCAN ( 5) | INDEX: INVIDX

40. Conclusion  Exploit SQL to:  Increase concurrency  Reduce I/O  Reduce code-path  Make the application more readable  SQL provides powerful support Tip: Part II with even meaner examples after the break

41. Serge Rielau IBM srielau@ca.ibm.com SQL on Fire! Part 1