1. Systems Group Dept. Computer Science ETH Zurich - Switzerland XQBench An XQuery Benchmarking Service Peter M. Fischer

2. Systems Group. D-INFK. ETH Zurich2 Motivation and Goal Benchmarks helped RDBMS/SQL to establish themselves: Highlight technical challenges Provide marketing arguments No well-defined and well-accepted benchmarks for XQuery Not another benchmark proposal! (yet?) Provide a service to perform XQuery benchmarks Simplify measurements Compare performance and results Collect interesting use cases

3. Benchmarking Model Model various use cases of XQuery Compare apples to apples Processors (cold start) Databases (ad-hoc loading, preloaded)...? Currently: evaluate individual read queries on static datasets Measure overall time always, details where possible Observe resource consumption and results Allow users to run Existing Benchmarks Contribute Queries Contribute Documents & Schemas Systems Group. D-INFK. ETH Zurich3

4. Usage (Short Demo) Experiment Creation and Submission Choose Engines, Documents, Queries Notification on submission, execution start, exeuction completion Pending Experiments shown to registered users Result Search Direct Link at Notification Complete List Directed GUI search for documents, queries, engines General XQuery for complex searches/correlations Result Presentation For every outcome, multiple views: Experiment description HTML (textual) for human consumption XML for further querying Plots (PNG, PS) in various dimensions, also plot scripts and data Execution Log, Engine Output Systems Group. D-INFK. ETH Zurich4

5. Usage (Short Demo) Public and Private Data/Queries/Experiments Everybody can run and search standard benchmarks Registered users can add own data and queries Private data and outcomes only accessible to owner Private data and outcomes can be made public Managing Documents and Queries Adding new Datasets (including schema) Adding new documents: Files+Metadata Adding new queries: Text+Metadata Access to metadata  Engines: Installation Log, «Adapter», Wiki Page Systems Group. D-INFK. ETH Zurich5

6. Data and Metadata Model Systems Group. D-INFK. ETH Zurich6

7. XQuery Engine Abstraction No common API, no common set of operations among engines Unify at least relevant parts Interaction Model:  Execute (including pre+post operations)  Engine Lifecycle: Start/Stop  Data Lifecycle: Load/Delete Timing:  Measure from outside  Engine reporting: query execution, parsing, serializing,... Tuning currently not supported: we have no good abstractions! (yet?) Running engines (mostly) with out-of-the-box settings Systems Group. D-INFK. ETH Zurich7

8. Abstraction: Adapter Example /dev/null 2>&1&); bin/wait_service_up_port.sh 127.0.0.1 1984]]> #times 2>&1 ]]>... 0.001.*Parsing: (\d+\.\d+) ms #t1... #t5... Systems Group. D-INFK. ETH Zurich8

9. Benchmark Service Architecture Systems Group. D-INFK. ETH Zurich9

10. Supported Engines Policy: cover all stable versions of open-source platforms Development versions change too often  Will update to «intermediate» if this version fixes a significant problem (e.g. BaseX attribute number limitation) Commercial: Cost money to run, restrictions on publishing results  Sponsorship, Permission to publish welcome! Currently:  DB: BaseX, BerkeleyDB, eXist, MonetDB, Sedna  Processor: QizX Free, Saxon B/HE, XQilla, Zorba Systems Group. D-INFK. ETH Zurich10

11. Pre-Installed Benchmarks Policy: provide as many existing workloads as possible The usual suspects  XMark + XPathMark  TPoX Currently collecting more:  The old guard: XMach-1, X007, XBench, Michigan  Recent Proposals: EXRT, XWeb, MeMBer, BooX,...  Interesting «normal» workloads: FDA, Geodata,... Existing Benchmarks secondary goal You should use/contribute your own workloads Contributed data, queries and results stay private until you decide to share them! Systems Group. D-INFK. ETH Zurich11

12. Status ETH provides frontend and several backend nodes (with support from th FLWOR Foundation) Available at „beta“ stage Helper infrastructure Bugtracker Wiki Mailing List Components are/will be open source Systems Group. D-INFK. ETH Zurich12

13. Future Directions Other languages  XSLT  SPARQL ... Improve collaboration features Extend interaction model  Schema  Indexes Extended Workloads  Update/Query mixes  User-defined execution flows... Systems Group. D-INFK. ETH Zurich13

14. Systems Group. D-INFK. ETH Zurich14 xqbench.org