Mutatis Mutandis Evaluating DBMS Test Adequacy with Mutation Testing Ivan T. Bowman, HANA Product Engineering June 24, 2013 Public

©2013 SAP AG or an SAP affiliate company. All rights reserved.2 Public Agenda Test adequacy and why we want to evaluate it Mutation testing and how to apply it to database systems Conclusions and future work

Test adequacy

©2013 SAP AG or an SAP affiliate company. All rights reserved.4 Public Challenges of testing database servers Self-management features Adapting to current conditions leads to increased internal state that needs to be tested Relational equivalence DML statements can be transformed into different equivalent forms The correct answer may be returned even if a desirable optimization was not considered Concurrent execution Database systems need to work well with concurrent connections and intra-query parallelism Concurrency leads to more primitives that need testing and more interactions to test Performance requirement are not clearly specified Tests of performance need to control many factors, limiting their sensitivity

©2013 SAP AG or an SAP affiliate company. All rights reserved.5 Public Goals of measuring test adequacy We want to make sure that software is thoroughly tested We want to compare advanced test generation techniques: Random query generation (RAGS) Genetic algorithms We want to minimize test suites by removing expensive tests that contribute little We want to prioritize test suites For example, require all developers to run a minimum-acceptance test before submission

©2013 SAP AG or an SAP affiliate company. All rights reserved.6 Public Measuring test adequacy Coverage techniques are commonly used Measure how many statements / basic blocks / code combinations are executed under tests The metric is simple to compute and understand Coverage metrics are necessary but not sufficient for computing test adequacy A direct approach to evaluating adequacy considers how many faults the tests find We could try to use natural faults that result from developer errors found before or after shipping We could manually seed faults in the code to see how many are detected For example, disable the effect of specific types of mutex We could automatically introduce faults using patterns: mutation testing

©2013 SAP AG or an SAP affiliate company. All rights reserved.7 Public Mutation testing Mutation testing evaluates the effectiveness of a test suite for detecting incorrect programs Evaluation focuses on those that are close to the correct version Mutation operators are defined to alter source code For example, change logical and ( && ) to or ( || ) Each operator creates a mutant program A test suite kills the mutant if it passes with the original and fails with the mutant Mutation adequacy score is the ratio of killed mutants to total mutants A particular problem arises with mutants that dont affect program semantics (equivalence)

©2013 SAP AG or an SAP affiliate company. All rights reserved.8 Public Does statement coverage predict ability to kill mutants?

©2013 SAP AG or an SAP affiliate company. All rights reserved.9 Public General approach of mutation testing Generate Mutants P Input Program P Generated Mutants P Input Test Set T P Fails T? Yes: P Killed No: P Lives

©2013 SAP AG or an SAP affiliate company. All rights reserved.10 Public Mutation operators we evaluated Function Applies to all methods and C-style functions. Skip the contents of the function. Condition Applies to the condition in for, while, and if statements. Evaluate a mutated condition. Switch Applies to all switch statements. Add one to the expression. Case Applies to all case statements within a switch. Skip the contents of the case. Default Applies to default statements within a switch statement. Skip the default statements.

©2013 SAP AG or an SAP affiliate company. All rights reserved.11 Public Mutation types and their outcomes (400K of query processing code)

Practical issues

©2013 SAP AG or an SAP affiliate company. All rights reserved.13 Public Large systems generate many mutants In a 400K line subset of SQL Anywhere, about 59K mutants identified Compiling these separately would take too much time and space Running the general mutant algorithm would take 432 days Generate a single meta-mutant where mutants can be enabled at run-time A:if( len > 0 ) B:if(mutation_on(123) ? (len>=0) : (len>0) ) Use the same code editing to track mutation coverage Identify which tests cover mutated code lines No need to evaluate whether a test kills a mutant it does not execute

©2013 SAP AG or an SAP affiliate company. All rights reserved.14 Public Simplifying assumptions

©2013 SAP AG or an SAP affiliate company. All rights reserved.15 Public Proposed Improvements

©2013 SAP AG or an SAP affiliate company. All rights reserved.16 Public Reminder: general approach of mutation testing Generate Mutants P Input Program P Generated Mutants P Input Test Set T P Fails T? Yes: P Killed No: P Lives

©2013 SAP AG or an SAP affiliate company. All rights reserved.17 Public Adjusted approach of mutation testing Input Program P Input Test Set T Meta-Mutant P Prepare Meta-Mutant Measure test time and mutant coverage Find lethal mutants

©2013 SAP AG or an SAP affiliate company. All rights reserved.18 Public Coverage and mutation adequacy as tests execute

Conclusions and Future Work

©2013 SAP AG or an SAP affiliate company. All rights reserved.20 Public Conclusions Mutation testing is practical for large systems such as DBMSs Mutation testing gives a harder adequacy metric than code coverage Our approach required simplifying assumptions that are not generally true Small-scale testing shows these hold sufficiently for some purposes Future work is needed to address simplifying assumptions Tests do not deterministically find faults; in particular, some mutations are non-deterministic Mutants may interact; can we characterize them as independent by analysis? Future work includes comparing test generation frameworks on mutation adequacy

©2013 SAP AG or an SAP affiliate company. All rights reserved.21 Public Acknowledgements Feedback from the SQL Anywhere Query Processing group was invaluable In particular, detailed suggestions from Daniel J. Farrar helped shape this paper Intern J. Devin Papineau provided significant motivation and early discussion for this work The anonymous reviewers provided helpful and insightful suggestions

©2013 SAP AG or an SAP affiliate company. All rights reserved. Thank you Contact information: Ivan T. Bowman Research Manager of SQL Anywhere Query Processing Ivan.Bowman (at) sap.com

©2013 SAP AG or an SAP affiliate company. All rights reserved.23 Public © 2013 SAP AG or an SAP affiliate company. All rights reserved. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. National product specifications may vary. These materials are provided by SAP AG and its affiliated companies ("SAP Group") for informational purposes only, without representation or warranty of any kind, and SAP Group shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty. SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries. Please see for additional trademark information and notices.

©2013 SAP AG or an SAP affiliate company. All rights reserved.24 Public © 2013 SAP AG oder ein SAP-Konzernunternehmen. Alle Rechte vorbehalten. Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und in welcher Form auch immer, ohne die ausdrückliche schriftliche Genehmigung durch SAP AG nicht gestattet. In dieser Publikation enthaltene Informationen können ohne vorherige Ankündigung geändert werden. Einige der von der SAP AG und ihren Distributoren vermarkteten Softwareprodukte enthalten proprietäre Softwarekomponenten anderer Softwareanbieter. Produkte können länderspezifische Unterschiede aufweisen. Die vorliegenden Unterlagen werden von der SAP AG und ihren Konzernunternehmen (SAP-Konzern) bereitgestellt und dienen ausschließlich zu Informationszwecken. Der SAP-Konzern übernimmt keinerlei Haftung oder Gewährleistung für Fehler oder Unvollständigkeiten in dieser Publikation. Der SAP-Konzern steht lediglich für Produkte und Dienstleistungen nach der Maßgabe ein, die in der Vereinbarung über die jeweiligen Produkte und Dienstleistungen ausdrücklich geregelt ist. Keine der hierin enthaltenen Informationen ist als zusätzliche Garantie zu interpretieren. SAP und andere in diesem Dokument erwähnte Produkte und Dienstleistungen von SAP sowie die dazugehörigen Logos sind Marken oder eingetragene Marken der SAP AG in Deutschland und verschiedenen anderen Ländern weltweit. Weitere Hinweise und Informationen zum Markenrecht finden Sie unter

©2013 SAP AG or an SAP affiliate company. All rights reserved.25 Public Drawing area Headline area White space The Grid