We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byAsia Hague
Modified over 2 years ago
Mutatis Mutandis Evaluating DBMS Test Adequacy with Mutation Testing Ivan T. Bowman, HANA Product Engineering June 24, 2013 Public
©2013 SAP AG or an SAP affiliate company. All rights reserved.2 Public Agenda Test adequacy and why we want to evaluate it Mutation testing and how to apply it to database systems Conclusions and future work
©2013 SAP AG or an SAP affiliate company. All rights reserved.4 Public Challenges of testing database servers Self-management features Adapting to current conditions leads to increased internal state that needs to be tested Relational equivalence DML statements can be transformed into different equivalent forms The correct answer may be returned even if a desirable optimization was not considered Concurrent execution Database systems need to work well with concurrent connections and intra-query parallelism Concurrency leads to more primitives that need testing and more interactions to test Performance requirement are not clearly specified Tests of performance need to control many factors, limiting their sensitivity
©2013 SAP AG or an SAP affiliate company. All rights reserved.5 Public Goals of measuring test adequacy We want to make sure that software is thoroughly tested We want to compare advanced test generation techniques: Random query generation (RAGS) Genetic algorithms We want to minimize test suites by removing expensive tests that contribute little We want to prioritize test suites For example, require all developers to run a minimum-acceptance test before submission
©2013 SAP AG or an SAP affiliate company. All rights reserved.6 Public Measuring test adequacy Coverage techniques are commonly used Measure how many statements / basic blocks / code combinations are executed under tests The metric is simple to compute and understand Coverage metrics are necessary but not sufficient for computing test adequacy A direct approach to evaluating adequacy considers how many faults the tests find We could try to use natural faults that result from developer errors found before or after shipping We could manually seed faults in the code to see how many are detected For example, disable the effect of specific types of mutex We could automatically introduce faults using patterns: mutation testing
©2013 SAP AG or an SAP affiliate company. All rights reserved.7 Public Mutation testing Mutation testing evaluates the effectiveness of a test suite for detecting incorrect programs Evaluation focuses on those that are close to the correct version Mutation operators are defined to alter source code For example, change logical and ( && ) to or ( || ) Each operator creates a mutant program A test suite kills the mutant if it passes with the original and fails with the mutant Mutation adequacy score is the ratio of killed mutants to total mutants A particular problem arises with mutants that dont affect program semantics (equivalence)
©2013 SAP AG or an SAP affiliate company. All rights reserved.8 Public Does statement coverage predict ability to kill mutants?
©2013 SAP AG or an SAP affiliate company. All rights reserved.9 Public General approach of mutation testing Generate Mutants P Input Program P Generated Mutants P Input Test Set T P Fails T? Yes: P Killed No: P Lives
©2013 SAP AG or an SAP affiliate company. All rights reserved.10 Public Mutation operators we evaluated Function Applies to all methods and C-style functions. Skip the contents of the function. Condition Applies to the condition in for, while, and if statements. Evaluate a mutated condition. Switch Applies to all switch statements. Add one to the expression. Case Applies to all case statements within a switch. Skip the contents of the case. Default Applies to default statements within a switch statement. Skip the default statements.
©2013 SAP AG or an SAP affiliate company. All rights reserved.11 Public Mutation types and their outcomes (400K of query processing code)
©2013 SAP AG or an SAP affiliate company. All rights reserved.13 Public Large systems generate many mutants In a 400K line subset of SQL Anywhere, about 59K mutants identified Compiling these separately would take too much time and space Running the general mutant algorithm would take 432 days Generate a single meta-mutant where mutants can be enabled at run-time A:if( len > 0 ) B:if(mutation_on(123) ? (len>=0) : (len>0) ) Use the same code editing to track mutation coverage Identify which tests cover mutated code lines No need to evaluate whether a test kills a mutant it does not execute
©2013 SAP AG or an SAP affiliate company. All rights reserved.14 Public Simplifying assumptions
©2013 SAP AG or an SAP affiliate company. All rights reserved.15 Public Proposed Improvements
©2013 SAP AG or an SAP affiliate company. All rights reserved.16 Public Reminder: general approach of mutation testing Generate Mutants P Input Program P Generated Mutants P Input Test Set T P Fails T? Yes: P Killed No: P Lives
©2013 SAP AG or an SAP affiliate company. All rights reserved.17 Public Adjusted approach of mutation testing Input Program P Input Test Set T Meta-Mutant P Prepare Meta-Mutant Measure test time and mutant coverage Find lethal mutants
©2013 SAP AG or an SAP affiliate company. All rights reserved.18 Public Coverage and mutation adequacy as tests execute
Conclusions and Future Work
©2013 SAP AG or an SAP affiliate company. All rights reserved.20 Public Conclusions Mutation testing is practical for large systems such as DBMSs Mutation testing gives a harder adequacy metric than code coverage Our approach required simplifying assumptions that are not generally true Small-scale testing shows these hold sufficiently for some purposes Future work is needed to address simplifying assumptions Tests do not deterministically find faults; in particular, some mutations are non-deterministic Mutants may interact; can we characterize them as independent by analysis? Future work includes comparing test generation frameworks on mutation adequacy
©2013 SAP AG or an SAP affiliate company. All rights reserved.21 Public Acknowledgements Feedback from the SQL Anywhere Query Processing group was invaluable In particular, detailed suggestions from Daniel J. Farrar helped shape this paper Intern J. Devin Papineau provided significant motivation and early discussion for this work The anonymous reviewers provided helpful and insightful suggestions
©2013 SAP AG or an SAP affiliate company. All rights reserved. Thank you Contact information: Ivan T. Bowman Research Manager of SQL Anywhere Query Processing Ivan.Bowman (at) sap.com
©2013 SAP AG or an SAP affiliate company. All rights reserved.23 Public © 2013 SAP AG or an SAP affiliate company. All rights reserved. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. National product specifications may vary. These materials are provided by SAP AG and its affiliated companies ("SAP Group") for informational purposes only, without representation or warranty of any kind, and SAP Group shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty. SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries. Please see http://www.sap.com/corporate-en/legal/copyright/index.epx#trademark for additional trademark information and notices.http://www.sap.com/corporate-en/legal/copyright/index.epx#trademark
©2013 SAP AG or an SAP affiliate company. All rights reserved.24 Public © 2013 SAP AG oder ein SAP-Konzernunternehmen. Alle Rechte vorbehalten. Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und in welcher Form auch immer, ohne die ausdrückliche schriftliche Genehmigung durch SAP AG nicht gestattet. In dieser Publikation enthaltene Informationen können ohne vorherige Ankündigung geändert werden. Einige der von der SAP AG und ihren Distributoren vermarkteten Softwareprodukte enthalten proprietäre Softwarekomponenten anderer Softwareanbieter. Produkte können länderspezifische Unterschiede aufweisen. Die vorliegenden Unterlagen werden von der SAP AG und ihren Konzernunternehmen (SAP-Konzern) bereitgestellt und dienen ausschließlich zu Informationszwecken. Der SAP-Konzern übernimmt keinerlei Haftung oder Gewährleistung für Fehler oder Unvollständigkeiten in dieser Publikation. Der SAP-Konzern steht lediglich für Produkte und Dienstleistungen nach der Maßgabe ein, die in der Vereinbarung über die jeweiligen Produkte und Dienstleistungen ausdrücklich geregelt ist. Keine der hierin enthaltenen Informationen ist als zusätzliche Garantie zu interpretieren. SAP und andere in diesem Dokument erwähnte Produkte und Dienstleistungen von SAP sowie die dazugehörigen Logos sind Marken oder eingetragene Marken der SAP AG in Deutschland und verschiedenen anderen Ländern weltweit. Weitere Hinweise und Informationen zum Markenrecht finden Sie unter http://www.sap.com/corporate-en/legal/copyright/index.epx#trademark.http://www.sap.com/corporate-en/legal/copyright/index.epx#trademark
©2013 SAP AG or an SAP affiliate company. All rights reserved.25 Public Drawing area Headline area White space The Grid
Best Run Global Purchasing Training Documentation: Hubwoo (SRM) Merchandise Catalog Order Process GPO Commodity Marketing April, 2013.
SAP HANA als Entwicklungsplattform
Talent Management with SuccessFactors Erwin F. Schoellkopf / Regional Sales Manager June 11, 2013.
A Framework for Modeling and Execution of Infrastructure Contention Experiments Carmelo Ragusa, Philip Robinson and Sergej Svorobej MERMAT 2013, FIA, 7.
Karsten Kohl / BearingPoint Switzerland AG June 11/12, 2013 Commodity Trading & Treasury Integration.
HANA Persistence Shadow Pages.
What´s New? SAP HANA SPS 07 Text Analysis (Delta from SPS 06 to SPS 07) SAP HANA Product Management November, 2013.
SAP HANA Product Management December, 2013
Yves Brennwald / Stephan Elster Juni, 2013
Jens Lemcke Research Associate, SAP AG Co-Authors: Michael Altenhofen, Egon Börger An Execution Semantics for Mediation Patterns.
SAP Solution Pre-Assembly THE FASTEST WAY TO CONSUME IT INNOVATION Juergen Grimmeisen Senior Program Manager SAP Value Prototyping.
Adaptive Computing Technical Overview
Knowledge, Action, and Context Uwe Riss SAP Research, Karlsruhe.
SAP Best Practices Conversion Tool SAP Best Practices.
Inventory Valuation for Year End Closing
Service Contract with Periodic Billing
General Cost Center Planning
Multiple Components in One Database (MCOD) Dr. Georg Leffers SAP AG.
SAP Sybase License Keys Learning Map SAP Support Portal.
Definition Usage Target Group Availability Benefits More info Cost
Internal Order for Marketing and Other Overhead Actual SAP Best Practices.
Platform Development Accelerator for SAP HANA Terms & Conditions DRAFT.
Physical Inventory / Inventory Count & Adjustment SAP Best Practices.
Serial Number Management SAP Best Practices. ©2013 SAP AG. All rights reserved.2 Purpose, Benefits, and Key Process Steps Purpose To explain the business.
Use this title slide only with an image Transforming Business for the Digital Era Jay Thoden van Velzen, P&I IoT & Customer Innovation February 11, 2016.
Create Engineering Change Number SAP Best Practices.
Rapid database migration to Sybase Adaptive Server Enterprise
Internal Order R&D Planning SAP Best Practices. ©2013 SAP AG. All rights reserved.2 Purpose, Benefits, and Key Process Steps Purpose This process makes.
Service with Time & Material Based Billing SAP Best Practices Baseline package SAP Best Practices.
Multiple Components in One Database (MCOD)
Overhead Cost Accounting – Actual
Travel Management SAP Best Practices. ©2013 SAP AG. All rights reserved.2 Purpose, Benefits, and Key Process Steps Purpose To provide fully integrated.
Reference and Simulation Costing
Quality Management for Procurement with Vendor Evaluation
Asset Acquisition through Direct Capitalization
Standard Cost Calculation
Stock Handling: Scrap and Blocked Stock
Internal Projects SAP Best Practices. ©2013 SAP AG. All rights reserved.2 Purpose, Benefits, and Key Process Steps Purpose In any professional service.
Advanced Shipping Notification via EDI - Inbound SAP Best Practices.
Maintain Material Master Extensions SAP Best Practices.
Sales Quotation SAP Best Practices.
Web SAP Peter McNulty, SAP Labs, LLC Sam Tingleff, SAP Labs, LLC Sinisa Zimek, SAP Labs, LLC.
Quarterly Plan - Sales Quantity Forecast with CO-PA
Sales Order Processing with Customer Down Payment SAP Best Practices.
Spot Consulting with Fixed Price Billing SAP Best Practices.
Purchased Material Price Planning SAP Best Practices.
© 2017 SlidePlayer.com Inc. All rights reserved.