1. Web Data Management XQuery 1

2. In this lecture Summary of XQuery FLWOR expressions – For, Let, Where, Order by, Return FOR and LET expressions Collections and sorting Resources XQuery: A Query Language for XMLXQuery: A Query Language for XML Chamberlin, Florescu, et al. W3C recommendation: www.w3.org/TR/xquery/ 2

3. XQuery Based on Quilt (which is based on XML-QL) Uses XPath to express more complex queries http://www.w3.org/TR/xquery XML Query data model (of course ) – Ordered ! 3

4. 4 Sample Data for Queries Addison-Wesley Serge Abiteboul Rick Hull Victor Vianu Foundations of Databases 1995 Freeman Jeffrey D. Ullman Principles of Database and Knowledge Base Systems 1998 Addison-Wesley Serge Abiteboul Rick Hull Victor Vianu Foundations of Databases 1995 Freeman Jeffrey D. Ullman Principles of Database and Knowledge Base Systems 1998

5. 5 Data Model for XPath bib book publisherauthor.. Addison-WesleySerge Abiteboul The root The root element

6. FLWOR (“Flower”) Expressions FLWOR expression supports iteration and binding of variables to intermediate results – useful for computing joins between two or more documents and for restructuring data FOR... LET... WHERE...ORDER BY…RETURN... The for and let clauses generate an ordered sequence of tuples of bound variables, called the tuple stream The optional where clause serves to filter the tuple stream The optional order by clause can be used to reorder the tuple stream The return clause constructs the result of the FLWOR expression 6

7. FOR-WHERE-RETURN Find all book titles published after 1995: 7 FOR $x IN document("bib.xml") /bib/book WHERE $x/year/text() > 1995 RETURN $x/title FOR $x IN document("bib.xml") /bib/book WHERE $x/year/text() > 1995 RETURN $x/title Result: abc def ghi

8. 8 FOR-WHERE-RETURN Equivalently (perhaps more geekish) FOR $x IN document("bib.xml")/bib/book[year/text() > 1995] /title RETURN $x FOR $x IN document("bib.xml")/bib/book[year/text() > 1995] /title RETURN $x And even shorter: document("bib.xml")/bib/book[year/text() > 1995] /title

9. 9 FOR-WHERE-RETURN Find all book titles and the year when they were published: FOR $x IN document("bib.xml")/ bib/book RETURN { $x/title/text() } { $x/year/text() } Result: abc 1995 def 2002 ghk 1980

10. 10 FOR-WHERE-RETURN Notice the use of “{“ and “}” What is the result without them ? FOR $x IN document("bib.xml")/bib/book RETURN $x/title/text() $x/year/text()

11. 11 Nesting For each author of a book by Morgan Kaufmann, list all books she published: FOR $b IN document(“bib.xml”)/bib, $a IN $b/book[publisher /text()=“Morgan Kaufmann”]/author RETURN { $a, FOR $t IN $b/book[author/text()=$a/text()]/title RETURN $t } FOR $b IN document(“bib.xml”)/bib, $a IN $b/book[publisher /text()=“Morgan Kaufmann”]/author RETURN { $a, FOR $t IN $b/book[author/text()=$a/text()]/title RETURN $t } In the RETURN clause comma concatenates XML fragments

12. 12 Result Jones abc def Smith ghi Jones abc def Smith ghi

13. 13 Aggregates Find all books with more than 3 authors: count = a function that counts avg = computes the average sum = computes the sum distinct-values = eliminates duplicates FOR $x IN document("bib.xml")/bib/book WHERE count($x/author)>3 RETURN $x

14. 14 Aggregates Same thing: FOR $x IN document("bib.xml")/bib/book[count(author)>3] RETURN $x

15. 15 Aggregates Print all authors who published more than 3 books – be aware of duplicates ! FOR $b IN document("bib.xml")/bib, $a IN distinct-values($b/book/author/text()) WHERE count($b/book[author/text()=$a])>3 RETURN { $a }

16. Aggregates 16 count = a (aggregate) function that returns the number of elms FOR $p IN distinct(document("bib.xml")//publisher) LET $b := document("bib.xml")/book[publisher = $p] WHERE count($b) > 100 RETURN $p FOR $p IN distinct(document("bib.xml")//publisher) LET $b := document("bib.xml")/book[publisher = $p] WHERE count($b) > 100 RETURN $p

17. Aggregates Find books whose price is larger than average: 17 LET $a=avg( document("bib.xml") /bib/book/@price) FOR $b in document("bib.xml") /bib/book WHERE $b/@price > $a RETURN $b LET $a=avg( document("bib.xml") /bib/book/@price) FOR $b in document("bib.xml") /bib/book WHERE $b/@price > $a RETURN $b

18. 18 Flattening “Flatten” the authors, i.e. return a list of (author, title) pairs FOR $b IN document("bib.xml")/bib/book, $x IN $b/title/text(), $y IN $b/author/text() RETURN { $x } { $y } Result: abc efg abc hkj

19. 19 Re-grouping For each author, return all titles of her/his books FOR $b IN document("bib.xml")/bib, $x IN $b/book/author/text() RETURN { $x } { FOR $y IN $b/book[author/text()=$x]/title RETURN $y } What about duplicate authors ? Result: efg abc klm....

20. 20 Re-grouping Same, but eliminate duplicate authors: FOR $b IN document("bib.xml")/bib LET $a := distinct-values($b/book/author/text()) FOR $x IN $a RETURN {$x} { FOR $y IN $b/book[author/text()=$x]/title RETURN $y }

21. 21 Re-grouping Same thing: FOR $b IN document("bib.xml")/bib, $x IN distinct-values($b/book/author/text()) RETURN {$x} { FOR $y IN $b/book[author/text()=$x]/title RETURN $y }

22. 22 Another Example Find book titles by the coauthors of “Database Theory”: FOR $b IN document("bib.xml")/bib, $x IN $b/book[title/text() = “Database Theory”], $y IN $b/book[author/text() = $x/author/text()] RETURN { $y/title/text() } Result: abc def abc ghk Question: Why do we get duplicates ?

23. 23 Distinct-values Same as before, but eliminate duplicates: Result: abc def ghk distinct-values = a function that eliminates duplicates Need to apply to a collection of text values, not of elements – note how query has changed FOR $b IN document("bib.xml")/bib, $x IN $b/book[title/text() = “Database Theory”]/author/text(), $y IN distinct-values($b/book[author/text() = $x] /title/text()) RETURN { $y } FOR $b IN document("bib.xml")/bib, $x IN $b/book[title/text() = “Database Theory”]/author/text(), $y IN distinct-values($b/book[author/text() = $x] /title/text()) RETURN { $y }

24. 24 SQL and XQuery Side-by-side Product(pid, name, maker, price) Find all product names, prices, sort by price SELECT x.name, x.price FROM Product x ORDER BY x.price SQL FOR $x in document(“db.xml”)/db/Product/row ORDER BY $x/price/text() RETURN { $x/name, $x/price } XQuery

25. 25 abc 7 def 23.... Xquery’s Answer Notice: this is NOT a well-formed document ! (WHY ???)

26. 26 Producing a Well-Formed Answer { FOR $x in document(“db.xml”)/db/Product/row ORDER BY $x/price/text() RETURN { $x/name, $x/price } }

27. 27 abc 7 def 23.... Xquery’s Answer Now it is well-formed !

28. 28 SQL and XQuery Side-by-side Product(pid, name, maker, price) Company(cid, name, city, revenues) Find all products made in Seattle SELECT x.name FROM Product x, Company y WHERE x.maker=y.cid and y.city=“Seattle” SQL FOR $r in document(“db.xml”)/db, $x in $r/Product/row, $y in $r/Company/row WHERE $x/maker/text()=$y/cid/text() and $y/city/text() = “Seattle” RETURN { $x/name } XQuery FOR $y in /db/Company/row[city/text()=“Seattle”], $x in /db/Product/row[maker/text()=$y/cid/text()] RETURN { $x/name } Cool XQuery

29. 29 123 abc efg …. ….......

30. 30 SQL and XQuery Side-by-side For each company with revenues < 1M count the products over $100 SELECT y.name, count(*) FROM Product x, Company y WHERE x.price > 100 and x.maker=y.cid and y.revenue < 1000000 GROUP BY y.cid, y.name FOR $r in document(“db.xml”)/db, $y in $r/Company/row[revenue/text() { $y/name/text() } { count($r/Product/row[maker/text()=$y/cid/text()][price/text()>100]) }

31. 31 SQL and XQuery Side-by-side Find companies with at least 30 products, and their average price SELECT y.name, avg(x.price) FROM Product x, Company y WHERE x.maker=y.cid GROUP BY y.cid, y.name HAVING count(*) > 30 FOR $r in document(“db.xml”)/db, $y in $r/Company/row LET $p := $r/Product/row[maker/text()=$y/cid/text()] WHERE count($p) > 30 RETURN { $y/name/text() } avg($p/price/text()) A collection An element

32. 32 Sorting in XQuery FOR $b IN document("bib.xml")//book[publisher = ‘M.K’] ORDER BY $b/price/text() RETURN { $b/title, $b/price } FOR $b IN document("bib.xml")//book[publisher = ‘M.K’] ORDER BY $b/price/text() RETURN { $b/title, $b/price }

33. If-Then-Else 33 FOR $h IN //holding RETURN $h/title, IF $h/@type = "Journal" THEN $h/editor ELSE $h/author SORTBY (title) FOR $h IN //holding RETURN $h/title, IF $h/@type = "Journal" THEN $h/editor ELSE $h/author SORTBY (title)

34. Existential Quantifiers 34 FOR $b IN //book WHERE SOME $p IN $b//para SATISFIES contains($p, "sailing") AND contains($p, "windsurfing") RETURN $b/title FOR $b IN //book WHERE SOME $p IN $b//para SATISFIES contains($p, "sailing") AND contains($p, "windsurfing") RETURN $b/title

35. Universal Quantifiers 35 FOR $b IN //book WHERE EVERY $p IN $b//para SATISFIES contains($p, "sailing") RETURN $b/title FOR $b IN //book WHERE EVERY $p IN $b//para SATISFIES contains($p, "sailing") RETURN $b/title

36. 36 Duplicate Elimination distinct-values(list-of-text-values) How do we eliminate duplicate “tuples” ? 3 100 8 500 3 100 3 200 8 500 3 100 8 500 3 100 3 200 8 500 3 100 8 500 3 200 3 100 8 500 3 200

37. FOR v.s. LET FOR $x in expr -- binds $x to each element in the list expr LET $x = expr -- binds $x to the entire list expr – Useful for common subexpressions and for aggregations 37

38. FOR v.s. LET Summary: FOR-LET-WHERE-RETURN = FLWR 38 FOR/LET Clauses WHERE Clause RETURN Clause List of tuples Instance of Xquery data model

39. FOR v.s. LET FOR Binds node variables  iteration LET Binds collection variables  one value 39

40. FOR v.s. LET 40 FOR $x IN document("bib.xml") /bib/book RETURN $x FOR $x IN document("bib.xml") /bib/book RETURN $x Returns:... LET $x := document("bib.xml") /bib/book RETURN $x LET $x := document("bib.xml") /bib/book RETURN $x Returns:...

41. Collections in XQuery Ordered and unordered collections – /bib/book/author = an ordered collection – Distinct(/bib/book/author) = an unordered collection LET $a = /bib/book  $a is a collection $b/author  a collection (several authors...) 41 RETURN $b/author Returns:...

42. Collections in XQuery What about collections in expressions ? $b/@price  list of n prices $b/@price * 0.7  list of n numbers $b/@price * $b/@quantity  list of n x m numbers ?? $b/@price * ($b/@quant1 + $b/@quant2)  $b/@price * $b/@quant1 + $b/@price * $b/@quant2 !! 42

43. Sorting in XQuery 43 FOR $p IN distinct(document("bib.xml")//publisher) RETURN $p/text(), FOR $b IN document("bib.xml")//book[publisher = $p] RETURN $b/title, $b/@price SORTBY(price DESCENDING) SORTBY(name) FOR $p IN distinct(document("bib.xml")//publisher) RETURN $p/text(), FOR $b IN document("bib.xml")//book[publisher = $p] RETURN $b/title, $b/@price SORTBY(price DESCENDING) SORTBY(name)

44. Sorting in XQuery Sorting arugments: refer to the name space of the RETURN clause, not the FOR clause To sort on an element you don’t want to display, first return it, then remove it with an additional query. 44

45. Other Stuff in XQuery BEFORE and AFTER – for dealing with order in the input FILTER – deletes some edges in the result tree Recursive functions – Currently: arbitrary recursion – Perhaps more restrictions in the future ? 45

46. Group-By in Xquery ?? No GROUPBY currently in XQuery A recent proposal (next) – What do YOU think ? 46

47. Group-By in Xquery ?? 47 FOR $b IN document("http://www.bn.com")/bib/book, $y IN $b/@year WHERE $b/publisher="Morgan Kaufmann" RETURN GROUPBY $y WHERE count($b) > 10 IN $y FOR $b IN document("http://www.bn.com")/bib/book, $y IN $b/@year WHERE $b/publisher="Morgan Kaufmann" RETURN GROUPBY $y WHERE count($b) > 10 IN $y SELECT year FROM Bib WHERE Bib.publisher="Morgan Kaufmann" GROUPBY year HAVING count(*) > 10 SELECT year FROM Bib WHERE Bib.publisher="Morgan Kaufmann" GROUPBY year HAVING count(*) > 10  with GROUPBY Equivalent SQL 

48. Group-By in Xquery ?? 48 FOR $b IN document("http://www.bn.com")/bib/book, $a IN $b/author, $y IN $b/@year RETURN GROUPBY $a, $y IN $a, $y, count($b) FOR $Tup IN distinct(FOR $b IN document("http://www.bn.com")/bib, $a IN $b/author, $y IN $b/@year RETURN $a $y ), $a IN $Tup/a/node(), $y IN $Tup/y/node() LET $b = document("http://www.bn.com")/bib/book[author=$a,@year=$y] RETURN $a, $y, count($b)  with GROUPBY Without GROUPBY 

49. Group-By in Xquery ?? 49 FOR $b IN document("http://www.bn.com")/bib/book, $a IN $b/author, $y IN $b/@year, $t IN $b/title, $p IN $b/publisher RETURN GROUPBY $p, $y IN $p, $y, GROUPBY $a IN $a, GROUPBY $t IN $t  Nested GROUPBY’s