The Random Query Generator The Colonoscopy of Database Software Jim Starkey Philip Stoev MySQL SystemQA Team

Testing Coverage [Slutz, 1998] All Possible SQL Statements and States Customer Usage Scenarios BUGS Tests

What is it? The Random Query Generator: Generates (pseudo-)random data Executes targeted (pseudo-)random queries Validates query results and monitors server status Reports crashes and other deviations

Features Multi-platform - supports Linux, Solaris, Windows (native) Fully automatic - integrated monitoring for unattended runs - provides pass/fail indication via exit() code and XML Highly customizable - tests all types of queries, tables and fields - plug-in components for validating the result Repeatable runs - for non-concurrent tests

The SQL Grammar Describes the queries to generate, YACC-style: query: SELECT _field FROM _table where LIMIT _digit | UPDATE _table SET _field = _digit where ; where: WHERE _field > _digit | WHERE _field IS NULL ; Provides convenience functions _field, _table, _digit, etc. Supports Embedded Perl

Query Generation 1.Walk the grammar tree and selectively take one of the branches at each fork in the road 2.Once a sequence of tokens has been obtained, replace each with an appropriate value query: select: SELECT_fieldFROM_table where: WHERE_field> _digit SELECT`f1`FROM`t1` WHERE`f2`> 9

Sample Generated Query SELECT SUM(DISTINCT OUTR. `varchar_nokey` ) AS X FROM C AS OUTR WHERE OUTR. `pk` IN ( SELECT INNR. `pk` AS Y FROM BB AS INNR2 LEFT JOIN BB AS INNR ON (INNR2.`datetime_nokey` > INNR.`time_nokey`) WHERE INNR. `int_nokey` > INNR. `int_nokey` AND OUTR. `time_key` 1 HAVING X <> '18:18:19 ORDER BY OUTR. `varchar_nokey`, OUTR. `pk`;

Testing home 1.Describe the data you wish to operate on 2.Describe the SQL queries that you will be running in a grammar 3.Run a synthetic workload and monitor server behavior and performance

Testing storage engines Compare query results with a reference storage engine (MyISAM or InnoDB) Ready tests for transactional integrity - Atomicity, Isolation and Consistency - REPEATABLE READ - Durability and recovery Stress testing across a wide range of: - engine configurations - row counts - field sizes - composition of the workload

Testing Recovery Recovery testing is performed at the end of every test - recovery is tested for free every time 1.Kill the server (with kill -9) - or kill at important code points via instrumentation - more sadistic methods forthcoming, e.g. a power cut 2.Restart the server or restore from snapshot/backup 3.Validate the data - verify that the last transactions were durable - check that the database remains consistent - walk the tables back and forth using different methods - issue CHECK/ANALYZE/OPTIMIZE/REPAIR

Questions? Philip Stoev Contact: Code: Documentation: