1. CS194-3/CS16x Introduction to Systems Lecture 25 SQL: The Query Language November 28, 2007 Prof. Anthony D. Joseph http://www.cs.berkeley.edu/~adj/cs16x

2. Lec 25.2 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 Two sublanguages: –DDL – Data Definition Language »Define and modify schema (at all 3 levels) –DML – Data Manipulation Language »Queries can be written intuitively The most widely used relational query language –Standardized, although most systems add their own “special sauce” We will study SQL92 – a basic subset Review: The SQL Query Language

3. Lec 25.3 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 Goals for Today Using SQL Database APIs –ODBC –JDBC Note: Some slides and/or pictures in the following are adapted from slides ©2005 Silberschatz, Galvin, and Gagne. Slides courtesy of Kubiatowicz, AJ Shankar, George Necula, Alex Aiken, Eric Brewer, Ras Bodik, Ion Stoica, Doug Tygar, and David Wagner.

4. Lec 25.4 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 Example Database sidsnameratingage 1Fred722 2Jim239 3Nancy827 Sailors sidbidday 11029/12 21029/13 Reserves bidbnamecolor 101Ninared 102Pintablue 103Santa Mariared Boats

5. Lec 25.5 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 The SQL Data Definition Language CREATE TABLE Sailors (sid INTEGER, sname CHAR(20), rating INTEGER, age REAL, PRIMARY KEY sid) CREATE TABLE Boats (bid INTEGER, bname CHAR (20), color CHAR(10) PRIMARY KEY bid) CREATE TABLE Reserves (sid INTEGER, bid INTEGER, day DATE, PRIMARY KEY (sid, bid, day), FOREIGN KEY sid REFERENCES Sailors, FOREIGN KEY bid REFERENCES Boats)

6. Lec 25.6 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 The SQL DML Find all 18-year-old sailors: To find just names and ratings, replace the first line: SELECT * FROM Sailors S WHERE S.age=18 SELECT S.sname, S.rating sidsnameratingage 1Fred722 2Jim239 3Nancy827 Sailors

7. Lec 25.7 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 Querying Multiple Relations SELECT S.sname FROM Sailors S, Reserves R WHERE S.sid=R.sid AND R.bid=102 sidsnameratingage 1Fred722 2Jim239 3Nancy827 Sailors sidbidday 11029/12 21029/13 Reserves

8. Lec 25.8 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 Basic SQL Query relation-list : List of relation names –Possibly with a range variable after each name target-list : List of attributes of tables in relation-list qualification : Comparisons combined using AND, OR and NOT DISTINCT : optional keyword indicating that the answer should not contain duplicates SELECT [DISTINCT] target-list FROM relation-list WHERE qualification

9. Lec 25.9 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 1. FROM : compute cross product of tables 2. WHERE : Check conditions, discard tuples that fail 3. SELECT : Delete unwanted fields 4. DISTINCT (optional) : eliminate duplicate rows Note: Probably the least efficient way to compute a query! –Query optimizer will find more efficient ways to get the same answer Query Semantics

10. Lec 25.10 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 Find Sailors Who’ve Reserved at Least One Boat Would adding DISTINCT to this query make a difference? What is the effect of replacing S.sid by S.sname in the SELECT clause? –Would adding DISTINCT to this variant of the query make a difference? SELECT S.sid FROM Sailors S, Reserves R WHERE S.sid=R.sid

11. Lec 25.11 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 About Range Variables Needed when ambiguity could arise. –e.g., same table used multiple times in FROM (“self-join”) SELECT x.sname, x.age, y.sname, y.age FROM Sailors x, Sailors y WHERE x.age > y.age sidsnameratingage 1Fred722 2Jim239 3Nancy827 Sailors

12. BREAK

13. Lec 25.13 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 Arithmetic Expressions SELECT S.age, S.age-5 AS age1, 2*S.age AS age2 FROM Sailors S WHERE S.sname = ‘dustin’ SELECT S1.sname AS name1, S2.sname AS name2 FROM Sailors S1, Sailors S2 WHERE 2*S1.rating = S2.rating - 1

14. Lec 25.14 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 String Comparisons `_’ stands for any one character and `%’ stands for 0 or more arbitrary characters SELECT S.sname FROM Sailors S WHERE S.sname LIKE ‘B_%B’

15. Lec 25.15 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 Find sid’s of Sailors Who’ve Reserved a red or a green Boat SELECT R.sid FROM Boats B, Reserves R WHERE R.bid=B.bid AND (B.color=‘red’ OR B.color=‘green’) SELECT R.sid FROM Boats B, Reserves R WHERE R.bid=B.bid AND B.color=‘red’ UNION SELECT R.sid FROM Boats B, Reserves R WHERE R.bid=B.bid AND B.color=‘green’... or:

16. Lec 25.16 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 SELECT R.sid FROM Boats B,Reserves R WHERE R.bid=B.bid AND (B.color=‘red’ AND B.color=‘green’) Find sid’s of Sailors Who’ve Reserved a red and a green Boat

17. Lec 25.17 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 Find sid’s of Sailors Who’ve Reserved a red and a green Boat SELECT S.sid FROM Sailors S, Boats B, Reserves R WHERE S.sid=R.sid AND R.bid=B.bid AND B.color=‘red’ INTERSECT SELECT S.sid FROM Sailors S, Boats B, Reserves R WHERE S.sid=R.sid AND R.bid=B.bid AND B.color=‘green’

18. Lec 25.18 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 Could use a self-join: SELECT R1.sid FROM Boats B1, Reserves R1, Boats B2, Reserves R2 WHERE R1.sid=R2.sid AND R1.bid=B1.bid AND R2.bid=B2.bid AND (B1.color=‘red’ AND B2.color=‘green’) Find sid’s of Sailors Who’ve Reserved a red and a green Boat

19. Lec 25.19 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 Find sid’s of sailors who have not reserved a boat SELECT S.sid FROM Sailors S EXCEPT SELECT S.sid FROM Sailors S, Reserves R WHERE S.sid=R.sid

20. Lec 25.20 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 Nested Queries: IN SELECT S.sname FROM Sailors S WHERE S.sid IN (SELECT R.sid FROM Reserves R WHERE R.bid=103) Names of sailors who’ve reserved boat #103:

21. Lec 25.21 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 SELECT S.sname FROM Sailors S WHERE S.sid NOT IN (SELECT R.sid FROM Reserves R WHERE R.bid=103) Names of sailors who’ve not reserved boat #103: Nested Queries: NOT IN

22. Lec 25.22 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 Nested Queries with Correlation Subquery must be recomputed for each Sailors tuple –Think of subquery as a function call that runs a query! Also: NOT EXISTS SELECT S.sname FROM Sailors S WHERE EXISTS (SELECT * FROM Reserves R WHERE R.bid=103 AND S.sid=R.sid) Names of sailors who’ve reserved boat #103:

23. Lec 25.23 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 UNIQUE SELECT S.sname FROM Sailors S WHERE UNIQUE (SELECT * FROM Reserves R WHERE R.bid=103 AND S.sid=R.sid) Names of sailors who’ve reserved boat #103 exactly once:

24. Lec 25.24 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 More on Set-Comparison Operators We’ve seen: IN, EXISTS, UNIQUE Can also have: NOT IN, NOT EXISTS, NOT UNIQUE Other forms: op ANY, op ALL Find sailors whose rating is greater than that of some sailor called Horatio: SELECT * FROM Sailors S WHERE S.rating > ANY (SELECT S2.rating FROM Sailors S2 WHERE S2.sname=‘Horatio’)

25. Lec 25.25 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 A Tough One SELECT S.sname FROM Sailors S WHERE NOT EXISTS ( SELECT B.bid FROM Boats B WHERE NOT EXISTS ( SELECT R.bid FROM Reserves R WHERE R.bid=B.bid AND R.sid=S.sid)) Sailors S such that... there is no boat B without... a Reserves tuple showing S reserved B Find sailors who’ve reserved all boats

26. Lec 25.26 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 SQL Summary Relational model has well-defined query semantics SQL provides functionality close to basic relational model (some differences in duplicate handling, null values, set operators, …) Typically, many ways to write a query –DBMS figures out a fast way to execute a query, regardless of how it is written

27. Lec 25.27 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 SQL is not a general purpose programming language –Tailored for data retrieval and manipulation –Relatively easy to optimize and parallelize –But, can’t write entire apps in SQL alone Options: 1.Make the query language “Turing complete” »Avoids the “impedance mismatch,” but loses advantages of relational language simplicity 2.Allow SQL to be embedded in regular programming languages 3.Provide an API for passing SQL statements to DB and retrieving results Writing Applications with SQL

28. Lec 25.28 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 Cursors SQL relations are (multi-)sets, no a priori bound on the number of records – most languages have no such data structures –SQL provides a cursor to handle this Operations: –Can declare a cursor on a relation or query –Can open a cursor –Can repeatedly fetch a tuple (moving the cursor) –Special return value when all tuples have been retrieved ORDER BY controls the order that tuples are returned –Fields in ORDER BY clause must appear in SELECT clause Can also modify/delete tuple pointed to by a cursor –A “non-relational” way to get a handle to a particular tuple

29. Lec 25.29 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 Database APIs Add a library with database calls (API) –Special procedures/objects –Passes SQL strings from language, presents result sets in a language-friendly way –ODBC a C/C++ standard started on Windows –JDBC a Java equivalent –Most scripting languages have similar things »E.g. For Perl there is DBI, “oraPerl”, other packages Mostly DBMS-neutral –At least try to hide distinctions across different DBMSs

30. Lec 25.30 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 Architecture »A lookup service maps “Data Source Names” (“DSNs”) to drivers Typically handled by OS »Based on the DSN used, a “driver” is linked into the app at runtime »The driver traps calls, translates them into DBMS-specific code »Database can be across a network »ODBC is standard, so the same program can be used (in principle) to access multiple database systems »Data source may not even be an SQL database! Application ODBC driver Data Source

31. Lec 25.31 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 Administrivia Project 3 code due Thursday 12/6 Midterm 3 Exam is Monday 12/10 Optional topics?

32. Lec 25.32 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 Various vendors provide drivers –MS bundles a bunch into Windows Drivers for various data sources –Relational DBMSs (Oracle, DB2, SQL Server, etc.) –“Desktop” DBMSs (Access, Dbase, Paradox, FoxPro) –Spreadsheets (MS Excel, Lotus 1-2-3, etc.) –Delimited text files (.CSV,.TXT, etc.) You can use JDBC/ODBC clients over many data sources –E.g. MS Query comes with many versions of MS Office (msqry32.exe) Can write your own Java or C++ programs against xDBC ODBC/JDBC

33. Lec 25.33 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 Part of Java, very easy to use, can talk to any ODBC data source JDBC tutorial online –http://java.sun.com/docs/books/tutorial/jdbc/http://java.sun.com/docs/books/tutorial/jdbc/ JDBC Basics: Connections –Connection object represents a login to a database // GET CONNECTION Connection con; try { con = DriverManager.getConnection( "jdbc:odbc:sailorsDB", userName,password); } catch(Exception e){ System.out.println(e); } –Eventually you close the connection // CLOSE CONNECTION try { con.close(); } catch (Exception e) { System.out.println(e); } JDBC

34. Lec 25.34 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 JDBC Basics: Statements You need a Statement object for each SQL statement // CREATE STATEMENT Statement stmt; try { stmt = con.createStatement(); } catch (Exception e){ System.out.println(e); } Soon we’ll say stmt.executeQuery(“select …”); CreateStatement cursor behavior (two optional args) –createStatement(ResultSet., ResultSet. ) –Corresponds to SQL cursor features:, »TYPE_FORWARD_ONLY: can’t move cursor backward »TYPE_SCROLL_INSENSITIVE: can move backward, but doesn’t show results of any updates »TYPE_SCROLL_SENSITIVE: can move backward, will show updates made while result set is open »CONCUR_READ_ONLY: this statement doesn’t allow updates »CONCUR_UPDATABLE: this statement allows updates –Defaults: TYPE_FORWARD_ONLY, CONCUR_READ_ONLY

35. Lec 25.35 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 JDBC Basics: ResultSet A ResultSet object serves as a cursor for the statement’s results (stmt.executeQuery()) // EXECUTE QUERY ResultSet results; try { results = stmt.executeQuery( "select * from Sailors") } catch (Exception e){ System.out.println(e); } Obvious handy methods: –results.next() advances cursor to next tuple »Returns “false” when the cursor slides off the table (beginning or end) –“scrollable” cursors: »results.previous(), results.relative(int), results.absolute(int), results.first(), results.last(), results.beforeFirst(), results.afterLast()

36. Lec 25.36 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 ResultSet Metadata Can find out stuff about the ResultSet schema via ResultSetMetaData ResultSetMetaData rsmd = results.getMetaData(); int numCols = rsmd.getColumnCount(); int i, rowcount = 0; // get column header info for (i=1; i <= numCols; i++){ if (i > 1) buf.append(","); buf.append(rsmd.getColumnLabel(i)); } buf.append("\n"); Other ResultSetMetaData methods: –getColumnType(i), isNullable(i), etc.

37. Lec 25.37 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 Getting Values in Current of Cursor getString // break it off at 100 rows max while (results.next() && rowcount < 100){ // Loop through each column, getting the // column data and displaying for (i=1; i <= numCols; i++) { if (i > 1) buf.append(","); buf.append(results.getString(i)); } buf.append("\n"); rowcount++; } Similarly, getFloat, getInt, etc.

38. Lec 25.38 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 Updating Current of Cursor Update fields in current of cursor: result.next(); result.updateInt("Rating", 10); –Also updateString, updateFloat, etc. Or can always submit a full SQL UPDATE statement –Via executeQuery() The original statement must have been CONCUR_UPDATABLE in either case! Cleaning up neatly: try { // CLOSE RESULT SET results.close(); // CLOSE STATEMENT stmt.close(); // CLOSE CONNECTION con.close(); } catch (Exception e) { System.out.println(e); }

39. Lec 25.39 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 Putting it Together (w/o try/catch) Connection con = DriverManager.getConnection("jdbc:odbc:weblog",userName,pas sword); Statement stmt = con.createStatement(); ResultSet results = stmt.executeQuery("select * from Sailors") ResultSetMetaData rsmd = results.getMetaData(); int numCols = rsmd.getColumnCount(), i; StringBuffer buf = new StringBuffer(); while (results.next() && rowcount < 100){ for (i=1; i <= numCols; i++) { if (i > 1) buf.append(","); buf.append(results.getString(i)); } buf.append("\n"); } results.close(); stmt.close(); con.close();

40. Lec 25.40 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 Similar Support for Web Scripting Languages Common scenario today is to have a web client –A web form issues a query to the DB –Results formatted as HTML Many web scripting languages used –jsp, asp, PHP, Ruby, etc. –Most of these are similar, look a lot like JDBC with HTML mixed in

41. Lec 25.41 11/28/07Joseph CS194-3/16x ©UCB Fall 2007 API Summary APIs are needed to interface DBMSs to programming languages Embedded SQL uses “native drivers” and is usually faster but less standard ODBC (used to be Microsoft-specific) for C/C++ JDBC is the standard for Java Scripting languages (PHP, Perl, JSP) are becoming the preferred technique for web-based systems