1. SQL John.Colquhoun@ncl.ac.uk

2. 2 Introduction Structured Query Language (SQL): the most widely used commercial relational database language Originally developed at IBM in the SEQUEL-XRM and System-R projects (1974-1977). Almost immediately, other vendors introduced DBMS products based on SQL, and it is now a de facto standard. The SQL continues to evolve in response to the changing need. The current ANSI/ISO standard for SQL is SQL-99 –Not all DBMS products support the full SQL-99 yet.

3. 3 The SQL language has several aspects to it: –The Data Definition Language (DDL) The subset of SQL supports the creation, deletion, and modification of definitions for tables and views (chap 3) –The Data Manipulation Language (DML) The subset of SQL allows users to insert, delete, and modify rows (chap 3) and to pose queries. –Triggers and Advanced Integrity Constraints The new SQL-99 standard includes support for triggers, which are action executed by the DBMS whenever changes to the database meet conditions specified in the trigger

4. 4 –Embedded and Dynamic SQL Embedded SQL features allow SQL code to be called from a host language such as C or COBOL. Dynamic SQL features allow a query to be constructed and executed at run-time. –Client-Server Execution and Remote Database Access These commands control –how a client application program can connect to an SQL database server, or –access data from a database over a network. –Transaction Management Transaction: any one execution of a user program in a DBMS Control how a transaction is to be executed. –Security Provides mechanisms to control users’ access to data.

5. 5 Basic SQL Query The basic form of SQL –relation-list A list of relation names (possibly with a range- variable after each name). –target-list A list of attributes of relations in relation-list. SELECT [DISTINCT] target-list FROM relation-list WHERE qualification

6. 6 –qualification Comparisons (Attr op const or Attr1 op Attr2, where op is one of ) combined using AND, OR and NOT. –DISTINCT an optional keyword indicating that the answer should not contain duplicates. Default is that duplicates are not eliminated (why?). SELECT [DISTINCT] target-list FROM relation-list WHERE qualification

7. 7 SELECT clause –specifies columns to be retained in the result. FROM clause –specifies a cross-product of tables. WHERE clause (optional) –specifies selection conditions on the tales mentioned in the FROM clause.

8. 8 An SQL query intuitively corresponds to a relational algebra expression involving selections, projections, and cross-products.  a1, a2, …, an (  P ( R 1 x R 2 x … x R m ) ) SELECT a 1, a 2, …, a n FROM R 1, R 2, …, R m WHERE P

9. 9 Example: find the names of all branches in the Loan relation. SELECT branch_name FROM Loan LoanResult

10. 10 To remove duplications SELECT DISTINCT branch_name FROM Loan LoanResult

11. 11 Conceptual evaluation strategy  Compute the cross-product of relation-list.  Delete rows in the cross-product that fail the qualification.  Delete attributes that are not in target-list.  If DISTINCT is specified, eliminate duplicate rows. (This strategy is probably the least efficient way to compute a query! An optimizer will find more efficient strategies to compute the same answers.) How to get the result without executing a SQL query?

12. 12 Example of conceptual evaluation –find the names of sailors who have reserved boat number 103. sidbidday 2210110/10/96 5810311/12/96 sidsnameratingage 22dustin745.0 31lubber855.5 58rusty1035.0 Instance R3 of Reserves Instance S4 of Sailors

13. 13 SELECT S.sname FROM Sailors S, Reserves R WHERE S.sid=R.sid AND R.bid=103 (sid)snameratingage(sid)bidday 22dustin745.02210110/10/96 22dustin745.05810311/12/96 31lubber855.52210110/10/96 31lubber855.55810311/12/96 58rusty1035.02210110/10/96 58rusty1035.05810311/12/96 S4  R3 Row remains after selection. Result sidbidday 2210110/10/96 5810311/12/96 sidsnameratingage 22dustin745.0 31lubber855.5 58rusty1035.0

14. 14 A Note on Range Variables –Really needed only if the same relation appears twice in the FROM clause. The previous query can also be written as: or SELECT S.sname FROM Sailors S, Reserves R WHERE S.sid=R.sid AND bid=103 SELECT sname FROM Sailors, Reserves WHERE Sailors.sid=Reserves.sid AND bid=103 It is good style, however, to use range variables always!

15. 15 More Examples Given the following schema: Sailors(sid: integer, sname: string, rating:integer, age: real) Boats(bid: integer, bname: string, colour: string) Reserves(sid: integer, bid: integer, day: date) sailors sidsnamerating Boats bidbname age colour Reserves sidbidday

16. 16 Sailors SID Sname Rating age Boats Bid Bname Colour Reserves SID BID Day

17. 17 Example: Find the sids of sailors who have reserved a red boat. Example: Find the names of sailors who have reserved a red boat. SELECT R.sid FROM Boats B, Reserves R WHERE B.bid = R.bid AND B.colour = ‘red’ sailors sidsnamerating Boats bidbname age colour Reserves sidbidday SELECT S.sname FROM Sailors S, Reserves R, Boat B WHERE S.sid = R.sid AND R.bid = B.bid AND B.colour = ‘red’

18. 18 Example: Find the colors of boats reserved by Lubber. (In general, there may be more than one sailor called Lubber. In this case, it will return the colours of boats reserved by some Lubber). sailors sidsnamerating Boats bidbname age color Reserves sidbidday SELECT B.colour FROM Sailors S, Reserves R, Boats B WHERE S.sid = R.sid AND R.bid = B.bid AND S.name = ‘Lubber’.

19. 19 Example: Find the names of sailors who have reserved at least one boat. (If a sailor has not made a reservation, the second step in the conceptual evaluation strategy would eliminate all rows in the cross-product that involve this sailor). sailors sidsnamerating Boats bidbname age color Reserves sidbidday SELECT S.name FROM Sailors S, Reserves R WHERE S.sid = R.sid

20. 20 Expressions and Strings Illustrates use of arithmetic expressions and string pattern matching – to find triples (ages of sailors and two fields defined by expressions) for sailors whose names begin and end with B and contain at least three characters. AS and = are two ways to name fields in result. LIKE is used for string matching. –_ : stands for exactly one, arbitrary, character –%: stands for 0 or more arbitrary characters (i.e., any substring). SELECT S.age, age1=S.age-5, 2*S.age AS age2 FROM Sailors S WHERE S.sname LIKE ‘B_%B’

21. 21 Union, Intersect, and Except SQL provides set-manipulation constructs: –UNION (  ) –INTERSECT (  ) –EXCEPT (  ) By default, duplicates are eliminated in results To retain duplicates, use UNION ALL, INTERSECT ALL, EXCEPT ALL

22. 22 sailors sidsnamerating Boats bidbname age colour Reserves sidbidday Example: Find sid’s of sailors who’ve reserved a red or a green boat UNION : Can be used to compute the union of any two union- compatible sets of tuples (which are themselves the result of SQL queries). SELECT S.sid FROM Sailors S, Boats B, Reserves R WHERE S.sid=R.sid AND R.bid=B.bid AND (B.colour=‘red’ OR B.colour=‘green’) SELECT S.sid FROM Sailors S, Boats B, Reserves R WHERE S.sid=R.sid AND R.bid=B.bid AND B.colour=‘red’ UNION SELECT S.sid FROM Sailors S, Boats B, Reserves R WHERE S.sid=R.sid AND R.bid=B.bid AND B.colour=‘green’

23. 23 sailors sidsnamerating Boats bidbname age colour Reserves sidbidday Example: Find sid’s of sailors who’ve reserved a red and a green boat ? SELECT S.sid FROM Sailors S, Boats B, Reserves R WHERE S.sid=R.sid AND R.bid=B.bid AND (B.colour=‘red’ AND B.colour=‘green’)

24. 24 sailors sidsnamerating Boats bidbname age color Reserves sidbidday Example: Find sid’s of sailors who’ve reserved a red and a green boat INTERSECT: compute the intersection of any two union- compatible sets of tuples. Q: feasibility of using method 2 to find sname of sailors who’ve …? SELECT S.sid FROM Sailors S, Boats B1, Reserves R1, Boats B2, Reserves R2 WHERE S.sid=R1.sid AND R1.bid=B1.bid AND S.sid=R2.sid AND R2.bid=B2.bid AND (B1.colour=‘red’ AND B2.colour=‘green’) SELECT S.sid FROM Sailors S, Boats B, Reserves R WHERE S.sid=R.sid AND R.bid=B.bid AND B.colour=‘red’ INTERSECT SELECT S.sid FROM Sailors S, Boats B, Reserves R WHERE S.sid=R.sid AND R.bid=B.bid AND B.colour=‘green’

25. 25 sailors sidsnamerating Boats bidbname age colour Reserves sidbidday Example: Find sid’s of all sailors who’ve reserved red boat but not green boat. SELECT S.sid FROM Sailors S, Boats B, Reserves R WHERE S.sid=R.sid AND R.bid=B.bid AND B.colour=‘red’ EXCEPT SELECT S.sid FROM Sailors S, Boats B, Reserves R WHERE S.sid=R.sid AND R.bid=B.bid AND B.colour=‘green’ Since the Reserves relation contains sid information, there is no need to look at the Sailors relation. We can have a more efficient version  (This query relies on referential integrity: there’re no reservations for non-existing sailors). SELECT R.sid FROM Boats B, Reserves R WHERE R.bid=B.bid AND B.colour=‘red’ EXCEPT SELECT R.sid FROM Boats B, Reserves R WHERE R.bid=B.bid AND B.colour=‘green’

26. 26 Nested Queries A nested query is a query that has another query embedded within it. –a very powerful feature of SQL: a WHERE clause can itself contain an SQL query! (Actually, so can FROM clause.) –The embedded query is called a subquery. –The embedded query can be a nested query itself. Queries may have very deeply nested structures.

27. 27 Example: Find names of sailors who’ve reserved boat #103: SELECT S.sname FROM Sailors S WHERE S.sid IN ( SELECT R.sid FROM Reserves R WHERE R.bid=103) IN operator: test whether a value is in a given set of elements To understand semantics of nested queries, think of a nested loops evaluation: For each Sailors tuple, check the qualification by computing the subquery. To find sailors who’ve not reserved #103, use NOT IN.

28. 28 Correlated Nested Queries In the previous example, the inner subquery has been completely independent of the outer query. In general, the inner subquery could depend on the row currently being examined in the outer query.

29. 29 Example: Find names of sailors who’ve reserved boat #103: SELECT S.sname FROM Sailors S WHERE EXISTS ( SELECT * FROM Reserves R WHERE R.bid=103 AND S.sid=R.sid) EXISTS (another set comparison operator) allows us to test whether a set is nonempty. For each Sailor row, such a test will be done. The subquery depends on the current Sailor row and must be re-evaluated for each row in Sailors. The occurrence of S.sid in the subquery is called a correlation, and such queries are called correlated queries

30. 30 Example: Find names of sailors who’ve reserved boat #103: SELECT S.sname FROM Sailors S WHERE EXISTS ( SELECT * FROM Reserves R WHERE R.bid=103 AND S.sid=R.sid) The use of special symbol * : we just want to check that a qualifying row exists and don’t want to retrieve any columns from the row By using NOT EXISTS, we can compute names of sailors who haven’t reserved such a boat If EXISTS replaced by UNIQUE and * replaced by R.bid, we get sailors with at most one reservation for boat #103. –UNIQUE returns true if no row appears twice in the answer to the query (i.e., no duplicates) –In particular, it returns true if the answer is empty

31. 31 Set-comparison Operators We’ve already seen IN, EXISTS and UNIQUE. We can also use NOT IN, NOT EXISTS and NOT UNIQUE. Also available: op ANY, op ALL –Where op is one of the arithmetic comparison operator –SOME is also available, but it is just a synonym for ANY. Example: Find sailors whose rating is greater than that of some sailor called Horatio: SELECT * FROM Sailors S WHERE S.rating > ANY ( SELECT S2.rating FROM Sailors S2 WHERE S2.sname=‘Horatio’)

32. 32 sailors sidsnamerating Boats bidbname age color Reserves sidbidday Example: Find sailors whose rating is better than every sailor called Horatio. SELECT * FROM Sailors S WHERE S.rating > ALL ( SELECT S2.rating FROM Sailors S2 WHERE S2.sname=‘Horatio’) Example: Find the sailors with the highest rating. SELECT * FROM Sailors S WHERE S.rating >= ALL ( SELECT S2.rating FROM Sailors S2) What will we get if “>=” replaced with “>”?

33. 33 More examples of nested query Rewriting INTERSECT queries using IN Example: Find sid’s of sailors who’ve reserved both a red and a green boat: = “Find all sailors who have reserved a red boat and, further, have sids that are included in the set of sids of sailors who have reserved a green boat.” To find names of Sailors who’ve reserved both red and green boats, just replace S.sid by S.sname in SELECT clause. Similarly, EXCEPT queries can be re-written using NOT IN. –Replace IN in above by NOT IN  sailors who have reserved red but not green boats SELECT S.sid FROM Sailors S, Boats B, Reserves R WHERE S.sid=R.sid AND R.bid=B.bid AND B.color=‘red’ AND S.sid IN ( SELECT S2.sid FROM Sailors S2, Boats B2, Reserves R2 WHERE S2.sid=R2.sid AND R2.bid=B2.bid AND B2.color=‘green’)

34. 34 Division in SQL Example: Find sailors who’ve reserved all boats. SELECT S.sname FROM Sailors S WHERE NOT EXISTS (( SELECT B.bid FROM Boats B) EXCEPT ( SELECT R.bid FROM Reserves R WHERE R.sid=S.sid)) sailors sidsnamerating Boats bidbname age color Reserves sidbidday Note that this query is correlated – for each sailor S, we check to see that the set of boats reserved by S includes every boat. All boats All boats reserved by S

35. 35 An alternative way to write the previous query without using EXCEPT (Intuitively, for each sailor we check that there is no boat that has not been reserved by this sailor.) sailors sidsnamerating Boats bidbname age color Reserves sidbidday SELECT S.sname FROM Sailors S WHERE NOT EXISTS ( SELECT B.bid FROM Boats B WHERE NOT EXISTS ( SELECT R.bid FROM Reserves R WHERE R.bid=B.bid AND R.sid=S.sid)) Sailors S such that... there is no boat B without... a Reserves tuple showing S reserved B

36. 36 Aggregate Operators Significant extension of relational algebra. –So that we can compute aggregate values such as MIN and SUM. SQL supports five aggregate operations, which can be applied on any column of a relation. COUNT([DISTINCT] A)The number of (unique) value in the A column. SUM ( [DISTINCT] A)The sum of all (unique) values in the A column. AVG ([DISTINCT] A)The average of all (unique) values in the A column. MAX (A)The maximum value in the A column. MIN (A)The minimum value in the A column.

37. 37 sailors sidsnamerating Boats bidbname age color Reserves sidbidday Example: Find the average age of all sailors SELECT AVG (S.age) FROMSailors S Example: Find the average age of sailors with rating of 10 SELECT AVG (S.age) FROMSailors S WHERES.rating = 10

38. 38 sailors sidsnamerating Boats bidbname age color Reserves sidbidday Example: Find the name and age of the oldest sailor SELECT S.sname, MAX (S.age) FROMSailors S SELECT S.sname, S.age FROM Sailors S WHERE S.age = ( SELECT MAX (S2.age) FROM Sailors S2) SELECT S.sname, S.age FROM Sailors S WHERE ( SELECT MAX (S2.age) FROM Sailors S2) = S.age Equivalent to the second query, and is allowed in the SQL/92 standard, but is not supported in some systems. If SELECT uses an aggregate operation, then it must use only agg. operations, unless the query contains a GROUP BY clause.

39. 39 sailors sidsnamerating Boats bidbname age color Reserves sidbidday Example: Count the number of Sailor SELECT COUNT (*) FROMSailors S Example: Count the number of different sailor names SELECT COUNT (DISTINCT S.sname) FROMSailors S

40. 40 sailors sidsnamerating Boats bidbname age color Reserves sidbidday Example: Find the names of sailors who are older than the oldest sailor with a rating of 10. SELECT S.sname FROMSailors S WHERES.age > ( SELECTMAX (S2.age) FROMSailors S2 WHERES2.rating = 10) Aggregate operations offer an alternative to the ANY and ALL constructs.

41. 41 Group by and Having So far, we’ve applied aggregate operators to all (qualifying) tuples. Sometimes, we want to apply them to each of several groups of tuples. Consider: Find the age of the youngest sailor for each rating level. – How many rating levels are there? What are the rating values for these levels? In general, we don’t know! – Suppose we know that rating values go from 1 to 10; we can write 10 queries that look like this (!): For i = 1, 2,..., 10: SELECT MIN (S.age) FROM Sailors S WHERE S.rating = i

42. 42 To write such queries, we need Group BY clause, a major extension to the basic SQL query form. E.g. the previous query (find the age of the youngest sailor for each rating level) can be expressed as follows SELECT S.rating, MIN (S.age) FROM Sailors S GROUP BY S.rating S.rating after ‘group by’ is called a grouping-list Optional HAVING clause can be used to specify qualifications over groups, i.e. group-qualification

43. 43 More examples SELECT branch_name, MAX ( balance ) FROM Account GROUP BY branch_name  For each branch, retrieve the branch name and the highest balance of the accounts in that branch

44. 44 SELECT branch_name, AVG ( balance ) FROM account WHERE acc_no > 100 GROUP BY branch_name HAVING AVG ( balance ) > 1200 account result  For each branch whose average balance for accounts with acc_no larger than 100 is more than $1200, retrieve the branch name and the average balance.

45. 45 SELECT [DISTINCT] target-list FROM relation-list WHERE qualification GROUP BY grouping-list HAVING group-qualification Extended SQL: the general form The target-list may contain (i) attribute names, and (ii) terms with aggregate operations (e.g., MIN (S.age)) Note: Each attribute appearing in (i) must also appear in grouping-list. Why? Understand group first: a collection of rows that have the same value for attributes in grouping-list Each row in the result of the query corresponds to one group, and attributes included in (i) must have the same value in each group If attribute A appears in (i) but not in grouping-list, there can be multiple rows within a group that have different A values, and it’s not clear what value should be assigned to this column in an answer row

46. 46 How to evaluate (conceptually) 1.Compute the cross-product of relation-list 2.Discard rows that fail qualification 3.Delete fields that are not specified by target-list, grouping-list or group-qualification 4.Partition the remaining rows according to the value of attributes in grouping-list 5.Apply the group-qualification to eliminate groups that are not qualified. Note for step 5: 1) Expressions in grp-qlfctn must have a single value per group! -- An attribute appearing in grp-qlfctn must appear as the argument to an aggregate operator, or it must also appear in grouping-list -- SQL does not exploit primary key semantics here! 2). One answer row is generated per qualifying group. SELECT [DISTINCT] target-list FROM relation-list WHERE qualification GROUP BY grouping-list HAVING group-qualification

47. 47 Example: Find the age of the youngest sailor with age 18, for each rating with at least 2 such sailors SELECT S.rating, MIN (S.age) FROM Sailors S WHERE S.age >= 18 GROUP BY S.rating HAVING COUNT (*) > 1 Only S.rating and S.age are mentioned in the SELECT, GROUP BY or HAVING clauses; other attributes are `unnecessary’. 2nd column of result is unnamed. (Use AS or = to name it) Answer relation

48. 48 For each red boat, find the number of reservations for this boat SELECT B.bid, COUNT (*) AS reservationcount FROM Boats B, Reserves R WHERE R.bid=B.bid AND B.colour=‘red’ GROUP BY B.bid SELECT B.bid, COUNT (*) AS reservationcount FROM Boats B, Reserves R WHERE R.bid=B.bid GROUP BY B.bid HAVING B.colour = ‘red’ sailors sidsnamerating Boats bidbname age colour Reserves sidbidday Only columns that appear in the GROUP BY clause can appear in the HAVING clause, unless they appear as arguments to an aggregate operator in the HAVING clause. (Group-qualification B.color = ‘red’ is single-valued per group, since bid is the key for Boats. However, remember “ … not exploit primary key semantics here”?)

49. 49 Example: Find the average age of sailors for each rating level that has at least two sailors. sidsnameratingage 22Dustin745.0 29Brutus133.0 31Lubber855.5 32Andy825.5 58Rusty1035.0 64Horatio735.0 71Zorba1016.0 74Horatio935.0 85Art325.5 95Bob363.5 SELECT S.rating, AVG(S.age) AS avgage FROM Sailor S GROUP BY S.rating HAVING COUNT (*) > 1 Instance S3 of Sailor Ratingavgage 344.5 740.0 840.5 1025.5 Answer SELECT S.rating, AVG(S.age) AS avgage FROM Sailor S GROUP BY S.rating HAVING 1 < (SELECT COUNT (*) FROM Sailors S2 WHERE S.rating = S2.rating) OR S.rating can be used in the subquery, since it has a single value for the current group of sailors HAVING clause can also contain a subquery

50. 50 Example: Find the average age of sailors who are at least 18 years old for each rating level that has at least two sailors. sidsnameratingage 22Dustin745.0 29Brutus133.0 31Lubber855.5 32Andy825.5 58Rusty1035.0 64Horatio735.0 71Zorba1016.0 74Horatio935.0 85Art325.5 95Bo0b363.5 96Frodo325.5 SELECT S.rating, AVG(S.age) AS avgage FROM Sailor S WHERE S.age >=18 GROUP BY S.rating HAVING 1 < (SELECT COUNT (*) FROM Sailors S2 WHERE S.rating = S2.rating) Ratingavgage 344.5 740.0 840.5 1035.0 Answer Instance S3 of Sailor The answer is very similar to the previous one, with the only difference being that for the group 10, we now ignore the sailor with age 16 while computing the average.

51. 51 Example: Find the average age of sailors who are at least 18 years old for each rating level that has at least two such sailors. sidsnameratingage 22Dustin745.0 29Brutus133.0 31Lubber855.5 32Andy825.5 58Rusty1035.0 64Horatio735.0 71Zorba1016.0 74Horatio935.0 85Art325.5 95Bob363.5 96Frodo325.5 SELECT S.rating, AVG(S.age) AS avgage FROM Sailor S WHERE S.age >=18 GROUP BY S.rating HAVING 1 < (SELECT COUNT (*) FROM Sailors S2 WHERE S.rating = S2.rating AND S2.age >=18) Ratingavgage 344.5 740.0 840.5 Answer Instance S3 of Sailor It differs from the answer of the previous question in that there is no tuple for rating 10, since there is only one tuple with rating 10 and age >= 18.

52. 52 Some other ways to write the previous query. SELECT S.rating, AVG(S.age) AS avgage FROM Sailor S WHERE S.age >=18 GROUP BY S.rating HAVING COUNT (*) > 1 SELECT Temp.rating, Temp.avgage FROM (SELECT S.rating, AVG(S.age) AS avgage, count (*) As ratingcount FROM Sailor S WHERE S.age >=18 GROUP BY S.rating) AS Temp WHERE Temp.ratingcount > 1 1.FROM clause can also contain a nested subquery 2.But how does this query work … 3.Same result! HAVING clause can be eliminated; however, it can provide a simpler expression in many cases 4.it’s necessary to give the subquery a name (otherwise we couldn’t express, e.g., the condition Temp.ratingcount > 1

53. 53 Find those ratings for which the average age is the minimum over all ratings sailors sidsnamerating Boats bidbname age colour Reserves sidbidday SELECT S.rating FROM Sailors S WHERE AVG (S.age) = ( SELECT MIN ( AVG (S2.age)) FROM Sailors S2 GROUP BY S2.rating) Aggregate operations cannot be nested! Even if the expression MIN(AVG(S2.age)), which is illegal, is allowed, this query will not work. …why? (homework)

54. 54 Correct solution (in SQL/92): SELECT Temp.rating, Temp.avgage FROM ( SELECT S.rating, AVG (S.age) AS avgage FROM Sailors S GROUP BY S.rating) AS Temp WHERE Temp.avgage = ( SELECT MIN (Temp.avgage) FROM Temp)  First, computes a temporary table containing the average age for each rating value Then, finds the rating(s) for which the average age is the minimum.

55. 55 Security: SQL injection Example query from SQL database: string sql = "SELECT * FROM client WHERE name= ’" + name + "’" Intention: take legitimate input such as Bob into query,  SELECT * FROM client WHERE name= ‘Bob’

56. 56 Security: SQL injection Example query from SQL database: string sql = "SELECT * FROM client WHERE name= ’" + name + "’" Attacker enters ‘ user name ’ : Bob’ OR 1=1 –- The SQL command becomes SELECT * FROM client WHERE name = ‘Bob’ OR 1=1 --’  - - is a comment erasing anything that would follow;  name=‘Bob’ OR 1=1 is always TRUE  the entire client database is selected!