1. SQL on Fire! Part 2 Tips and Tricks around SQL

2. Agenda  Part I  SQL vs. SQL PL example  Error handling  Tables out of nowhere  Pivoting  Aggregation  Deleting duplicate rows  Alternatives to Min and Max  Part II  Mass deletes  Order processing  Moving rows between tables  Recursion  Merge  Calculating nesting levels Easy Difficult

3. Motivation – The OLTP Mantra Reduce Codepath Reduce Logical and Physical I/O Reduce Network Traffic Minimize Lock Contention Avoid Deadlocks High performance starts with the application.

4. Mass deleting of rows  Problem: Delete a large number of rows without excessive usage of log space.  Classic: –Delete using cursor Very slow –EXPORT remaining data, LOAD replace Outage where table is unavailable.  Can do better!

5. Mass deleting of rows CREATE PROCEDURE purgeInventory(IN dt DATE) BEGIN DECLARE SQLCODE INTEGER; loop: LOOP DELETE FROM (SELECT 1 FROM Inventory WHERE InvDate <= dt FETCH FIRST 1000 ROWS ONLY) AS D; IF SQLCODE = 100 THEN LEAVE loop; END IF; COMMIT; END LOOP loop; END CALL purgeInventory(‘2003-10-01’)

6. New order processing - classic "Submit order, provide reference # back" Retrieve next order # SELECT nextnum FROM ordermeta; 1 I/O, S Increment next order # UPDATE ordermeta 1 I/O SET nextnum = nextnum + 1; S->X Insert new order INSERT INTO orders VALUES(nextnum,...) 1 I/O Return nextnum to user Deadlock, 3 SQL Stmts, 3 I/O, single row

7. New order processing - improved Use SEQUENCE/IDENTITY INSERT INTO order VALUES(NEXT VALUE FOR orderseq,...); SET ordernum = PREVIOUS VALUE FOR orderseq; No Deadlock, 2 SQL statements, 1 I/O, single row only Use SELECT FROM INSERT SELECT ordernum INTO :ordernum FROM NEW TABLE(INSERT INTO orders VALUES(NEXT VALUE FOR orderseq,...)); No Deadlock, 1 I/O,1 SQL Stmt, set oriented

8. New order processing – Optimal plan Access Plan: ----------- RETURN ( 1) | 1 INSERT ( 4) 1 /---+---\ 1 180 TBSCAN TABLE: ORDERS ( 5) | 1 TABFNC: GENROW

9. Queue processing – Destructive read "retrieve and delete next in line“ Retrieve "oldest" row in queue SELECT ordernum,... INTO :ordernum 1I/O, S FROM orders ORDER BY ordernum FETCH FIRST ROW ONLY; Delete the row DELETE FROM order 1 I/O, S->X WHERE :ordernum = ordernum; Deadlock, 2 SQL Stmts, single row (or IN list)

10. Destructive read - Improved Delete through ORDER BY SELECT ordernum,... INTO :ordernum,... FROM OLD TABLE(DELETE FROM (SELECT * FROM orders ORDER BY ordernum FETCH FIRST ROW ONLY)); no Deadlock, 1 I/O, set oriented

11. Destructive Read – Optimal plan Access Plan: ----------- RETURN ( 1) | DELETE ( 4) /----+---\ IXSCAN TABLE: ORDERS ( 5) | INDEX: i1

12. Queue processing – 2-phase CREATE TABLE orders(ordernum INTEGER NOT NULL, agentid INTEGER); CREATE UNIQUE INDEX orderind ON orders(ordernum ASC) INCLUDE (agentid ASC); ALTER TABLE orders ADD CONSTRAINT PK PRIMARY KEY (ordernum); Tip 1: Combine unique and non unique index using INCLUDE. Tip 2: Add primary key after creating index to control which index is used (index name and include columns).

13. Queue processing – 2 Phase OrdernumAgentID 103115 10327 103320 1034NULL 1035NULL 1036NULL 1037NULL

14. Queue processing – claim order SET vthisorder = (SELECT ordernum FROM OLD TABLE(UPDATE (SELECT ordernum, status FROM orders WHERE agentid IS NULL ORDER BY ordernum FETCH FIRST ROW ONLY) AS U SET agentid = vagentid)); COMMIT; …; -- Long processing DELETE FROM orders WHERE ordernum = vthisorder; COMMIT;

15. Queue processing – claim order Access Plan: ------------ RETURN ( 1) | TBSCAN ( 2) | SORT ( 3) | UPDATE ( 4) /---+---\ IXSCAN TABLE: ORDERS ( 5) | INDEX: ORDERIND

16. Moving duplicate rows  Task Delete rows from one table and insert into another CREATE TABLE Archive LIKE Inventory; WITH del(Item, Quantity, InvDate) AS (SELECT Item, Quantity, InvDate FROM OLD TABLE (DELETE FROM (SELECT Item, Quantity, InvDate, row_number() OVER(PARTITION BY Item ORDER BY InvDate DESC) AS rn FROM Inventory) WHERE rn > 1)), ins(x) AS (SELECT 1 FROM NEW TABLE(INSERT INTO Archive SELECT * FROM del)) SELECT COUNT(1) FROM ins;

17. Move duplicate rows Do-At-Open Dam (1-row) RETURN ( 1) | TBSCAN ( 2) | SORT ( 3) | GRPBY ( 4) | INSERT ( 5) /---+---\ DELETE TABLE: ARCHIVE ( 6) /---+---\ FETCH TABLE: INVENTORY ( 7) /---+---\ FILTER TABLE: INVENTORY ( 8) | FETCH ( 9) /----+---\ IXSCAN TABLE: INVENTORY ( 10) | INDEX: SRIELAU

18. Recursion  Problem Have table of sales per working day. Need table of sales per calendar day.  DDL CREATE TABLE Sales(day VARCHAR(10), date DATE, amount INTEGER)

19. Recursion DayDateAmount Friday2006-May-1230 Weekend Monday2006-May-1520 Tuesday2006-May-1615 Wednesday2006-May-1725 Thursday2006-May-1831 Friday2006-May-1932 Weekend Monday2006-May-2211 Tuesday2006-May-2318

20. Recursion  Produce a date range CREATE FUNCTION dates(start DATE, end DATE) RETURNS TABLE(dt DATE) RETURN WITH rec(dt) AS (VALUES (start) UNION ALL SELECT dt + 1 DAY FROM rec WHERE dt < end) SELECT dt FROM rec; SELECT DAYNAME(date) AS day, date, COALESCE(sales, 0) AS sales FROM TABLE(dates(DATE('2006-05-12'), DATE('2006-05-23'))) AS dates LEFT OUTER JOIN sales ON dates.dt = sales.date;

21. Recursion DayDateAmount Friday2006-May-1230 Saturday2006-May-130 Sunday2006-May-140 Monday2006-May-1520 Tuesday2006-May-1615 Wednesday2006-May-1725 Thursday2006-May-1831 Friday2006-May-1932 Saturday2006-May-200 Sunday2005-May-210 Monday2006-May-2211 Tuesday2006-May-2318

22. Recursion inside out (a, b, c) (d, e, f) (g, h, i) (j, k, l) Seed (a, b, c) (d, e, f) (g, h, i) (j, k, l) Read Cursor (h, i, j) (k, m, n) (k, l, m) Insert 1Seed -> rec-temp 2For each row in temp execute recursion Append to temp 3Finish when 2. catches up with appends (a, b, c) (d, e, f) (g, h, i) (j, k, l) Read Cursor (h, i, j) (k, m, n) (k, l, m) (z, z, z)

23. RETURN ( 1) | TBSCAN ( 2) | TEMP ( 3) | UNION ( 4) /----+---\ TBSCAN TBSCAN ( 5) ( 6) | | TEMP TABFNC: GENROW ( 3) Rec-Plan

24. Merge Unifies Update, Delete, Insert Procedural statement Set oriented processing per branch Consistency points at each branch SQL Standard

25. Merge Make Up MERGE INTO USING ON {WHEN [NOT] MATCHED [AND ] THEN [UPDATE SET...|DELETE|INSERT VALUES....|SIGNAL...]} [ELSE IGNORE] is any updatable query is whatever query you please partitions into MATCHED and NOT MATCHED Each target-row must only be matched once! WHEN.. [ ] executes THEN for subset of [not] matched rows. each row in is processed once in first qualifying WHEN only

26. Update From CREATE TABLE T(pk INT NOT NULL PRIMARY KEY, c1 INT); CREATE TABLE S(pk INT NOT NULL PRIMARY KEY, c1 INT); Standard pre SQL4 UPDATE T SET c1 = (SELECT c1 FROM S WHERE S.pk = T.pk) WHERE pk IN (SELECT pk FROM S); IBM Informix/MS SQL Server UPDATE T SET c1 = S.c1 FROM T, S WHERE T.pk = S.pk;

27. Merge: Update From RETURN | UPDATE /---+---\ FILTER TABLE: T | NLJOIN /-------+-------\ TBSCAN FETCH | /----+---\ TABLE: S IXSCAN TABLE: T | INDEX: T_PK MERGE INTO T USING S ON T.pk = S.pk WHEN MATCHED THEN UPDATE SET c1 = S.c1;

28. Upsert Standard pre SQL4 (one way of many) FOR m AS SELECT T.pk tpk, S.pk spk, S.c1 FROM T RIGHT JOIN S ON T.pk = S.pk DO IF m.tpk IS NULL THEN INSERT INTO T VALUES(m.spk, m.c1); ELSE UPDATE T SET c1 = m.c1 WHERE pk = tpk; END IF; END FOR;

29. RETURN | INSERT /---+---\ TBSCAN TABLE: T | TEMP | UPDATE /---+---\ NLJOIN TABLE: T /------------------+-----------------\ NLJOIN UNION 25.7489 0.00166765 /-------+-------\ /------+-----\ TBSCAN FETCH FILTER FILTER | /----+---\ | | TABLE: S IXSCAN TABLE: T TBSCAN TBSCAN | | | INDEX: T_PK TABFNC: GENROW TABFNC: GENROW Merge Upsert MERGE INTO T USING S ON T.pk = S.pk WHEN MATCHED THEN UPDATE SET c1 = S.c1 WHEN NOT MATCHED THEN INSERT VALUES(pk, c1);

30. Merge single row Upsert MERGE INTO T USING (VALUES(1, 2)) AS S(pk, c1) ON S.pk = T.pk WHEN MATCHED THEN UPDATE SET c1 = S.c1 WHEN NOT MATCHED THEN INSERT VALUES (pk, c1) RETURN | INSERT /------+------\ UPDATE TABLE: T /------+-------\ NLJOIN TABLE: T /-------------+-------------\ NLJOIN UNION /-----+-----\ /------+-----\ TBSCAN TBSCAN FILTER FILTER | | | | TABFNC: GENROW TABLE: T TBSCAN TBSCAN | | TABFNC: GENROW TABFNC: GENCROW

31. Merge rules of engagement  Place INSERT branch at the end. Otherwise second temp between INSERT and UPDATE  Aim for NLJOIN over MERGE target in OLTP. Other joins result in exclusive lock on target table. E.g. drop optimization level to 3

32. Calculating nesting levels  Problem Given a log table with nested events. How deep is nesting at any given time?  DDL CREATE TABLE log(timestamp TIMESTAMP, event CHAR(3), data VARCHAR(10));

33. Calculating nesting levels TimestampEvent 2006-03-02 04:12:16.124565 In 2006-03-02 04:12:19.134514 In 2006-03-02 04:13:42.424085 Out 2006-03-02 04:15:31.872452 In 2006-03-02 04:16:42.004545 In 2006-03-02 04:17:01.994432 In 2006-03-02 04:17:23.569474 Out 2006-03-02 04:22:25.946465 Out 2006-03-02 04:32:19.543438 In 2006-03-02 04:33:58.172535 Out 2006-03-02 04:42:00.836468 Out 2006-03-02 04:46:51.643544 Out

34. Calculating nesting levels NestingTimestampEvent > 2006-03-02 04:12:16.124565In |> 2006-03-02 04:12:19.134514In |< 2006-03-02 04:13:42.424085Out |> 2006-03-02 04:15:31.872452In ||> 2006-03-02 04:16:42.004545In |||> 2006-03-02 04:17:01.994432In |||< 2006-03-02 04:17:23.569474Out ||< 2006-03-02 04:22:25.946465Out ||> 2006-03-02 04:32:19.543438In ||< 2006-03-02 04:33:58.172535Out |< 2006-03-02 04:42:00.836468Out < 2006-03-02 04:46:51.643544Out

35. Calulating nesting levels SELECT REPEAT('|',SUM(CASE event WHEN 'In' THEN 1 WHEN 'Out' THEN -1 END) OVER (ORDER BY timestamp) + CASE event WHEN 'In' THEN -1 ELSE 0 END) || CASE event WHEN 'In' THEN '>' WHEN 'Out' THEN '<' END AS nesting, timestamp, event FROM Log ORDER BY timestamp;

36. Appendix – V8 feature roadmap  FP2 –MERGE (prefer FP9 for performance optimizations) –ORDER BY and FETCH FIRST in subquery  FP4 (TPC-C) –SELECT FROM UPDATE/DELETE/INSERT –UPDATE/DELETE/INSERT of subquery and order by.  FP7 –CALL in “inline” SQL PL, –“native” SQL Procedures (prefer FP9 for stability) –DROP/SET DEFAULT and identity columns  FP9 –Automatic storage

37. Conclusion Exploit SQL to: Increase concurrency Reduce I/O Reduce code-path Make the application more readable SQL provides powerful support

38. Serge Rielau IBM srielau@ca.ibm.com SQL on Fire! Part 2