We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byAsia Hague
Modified over 2 years ago
Mutatis Mutandis Evaluating DBMS Test Adequacy with Mutation Testing Ivan T. Bowman, HANA Product Engineering June 24, 2013 Public
©2013 SAP AG or an SAP affiliate company. All rights reserved.2 Public Agenda Test adequacy and why we want to evaluate it Mutation testing and how to apply it to database systems Conclusions and future work
©2013 SAP AG or an SAP affiliate company. All rights reserved.4 Public Challenges of testing database servers Self-management features Adapting to current conditions leads to increased internal state that needs to be tested Relational equivalence DML statements can be transformed into different equivalent forms The correct answer may be returned even if a desirable optimization was not considered Concurrent execution Database systems need to work well with concurrent connections and intra-query parallelism Concurrency leads to more primitives that need testing and more interactions to test Performance requirement are not clearly specified Tests of performance need to control many factors, limiting their sensitivity
©2013 SAP AG or an SAP affiliate company. All rights reserved.5 Public Goals of measuring test adequacy We want to make sure that software is thoroughly tested We want to compare advanced test generation techniques: Random query generation (RAGS) Genetic algorithms We want to minimize test suites by removing expensive tests that contribute little We want to prioritize test suites For example, require all developers to run a minimum-acceptance test before submission
©2013 SAP AG or an SAP affiliate company. All rights reserved.6 Public Measuring test adequacy Coverage techniques are commonly used Measure how many statements / basic blocks / code combinations are executed under tests The metric is simple to compute and understand Coverage metrics are necessary but not sufficient for computing test adequacy A direct approach to evaluating adequacy considers how many faults the tests find We could try to use natural faults that result from developer errors found before or after shipping We could manually seed faults in the code to see how many are detected For example, disable the effect of specific types of mutex We could automatically introduce faults using patterns: mutation testing
©2013 SAP AG or an SAP affiliate company. All rights reserved.7 Public Mutation testing Mutation testing evaluates the effectiveness of a test suite for detecting incorrect programs Evaluation focuses on those that are close to the correct version Mutation operators are defined to alter source code For example, change logical and ( && ) to or ( || ) Each operator creates a mutant program A test suite kills the mutant if it passes with the original and fails with the mutant Mutation adequacy score is the ratio of killed mutants to total mutants A particular problem arises with mutants that dont affect program semantics (equivalence)
©2013 SAP AG or an SAP affiliate company. All rights reserved.8 Public Does statement coverage predict ability to kill mutants?
©2013 SAP AG or an SAP affiliate company. All rights reserved.9 Public General approach of mutation testing Generate Mutants P Input Program P Generated Mutants P Input Test Set T P Fails T? Yes: P Killed No: P Lives
©2013 SAP AG or an SAP affiliate company. All rights reserved.10 Public Mutation operators we evaluated Function Applies to all methods and C-style functions. Skip the contents of the function. Condition Applies to the condition in for, while, and if statements. Evaluate a mutated condition. Switch Applies to all switch statements. Add one to the expression. Case Applies to all case statements within a switch. Skip the contents of the case. Default Applies to default statements within a switch statement. Skip the default statements.
©2013 SAP AG or an SAP affiliate company. All rights reserved.11 Public Mutation types and their outcomes (400K of query processing code)
©2013 SAP AG or an SAP affiliate company. All rights reserved.13 Public Large systems generate many mutants In a 400K line subset of SQL Anywhere, about 59K mutants identified Compiling these separately would take too much time and space Running the general mutant algorithm would take 432 days Generate a single meta-mutant where mutants can be enabled at run-time A:if( len > 0 ) B:if(mutation_on(123) ? (len>=0) : (len>0) ) Use the same code editing to track mutation coverage Identify which tests cover mutated code lines No need to evaluate whether a test kills a mutant it does not execute
©2013 SAP AG or an SAP affiliate company. All rights reserved.14 Public Simplifying assumptions
©2013 SAP AG or an SAP affiliate company. All rights reserved.15 Public Proposed Improvements
©2013 SAP AG or an SAP affiliate company. All rights reserved.16 Public Reminder: general approach of mutation testing Generate Mutants P Input Program P Generated Mutants P Input Test Set T P Fails T? Yes: P Killed No: P Lives
©2013 SAP AG or an SAP affiliate company. All rights reserved.17 Public Adjusted approach of mutation testing Input Program P Input Test Set T Meta-Mutant P Prepare Meta-Mutant Measure test time and mutant coverage Find lethal mutants
©2013 SAP AG or an SAP affiliate company. All rights reserved.18 Public Coverage and mutation adequacy as tests execute
Conclusions and Future Work
©2013 SAP AG or an SAP affiliate company. All rights reserved.20 Public Conclusions Mutation testing is practical for large systems such as DBMSs Mutation testing gives a harder adequacy metric than code coverage Our approach required simplifying assumptions that are not generally true Small-scale testing shows these hold sufficiently for some purposes Future work is needed to address simplifying assumptions Tests do not deterministically find faults; in particular, some mutations are non-deterministic Mutants may interact; can we characterize them as independent by analysis? Future work includes comparing test generation frameworks on mutation adequacy
©2013 SAP AG or an SAP affiliate company. All rights reserved.21 Public Acknowledgements Feedback from the SQL Anywhere Query Processing group was invaluable In particular, detailed suggestions from Daniel J. Farrar helped shape this paper Intern J. Devin Papineau provided significant motivation and early discussion for this work The anonymous reviewers provided helpful and insightful suggestions
©2013 SAP AG or an SAP affiliate company. All rights reserved. Thank you Contact information: Ivan T. Bowman Research Manager of SQL Anywhere Query Processing Ivan.Bowman (at) sap.com
©2013 SAP AG or an SAP affiliate company. All rights reserved.23 Public © 2013 SAP AG or an SAP affiliate company. All rights reserved. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. National product specifications may vary. These materials are provided by SAP AG and its affiliated companies ("SAP Group") for informational purposes only, without representation or warranty of any kind, and SAP Group shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty. SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries. Please see for additional trademark information and notices.http://www.sap.com/corporate-en/legal/copyright/index.epx#trademark
©2013 SAP AG or an SAP affiliate company. All rights reserved.24 Public © 2013 SAP AG oder ein SAP-Konzernunternehmen. Alle Rechte vorbehalten. Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und in welcher Form auch immer, ohne die ausdrückliche schriftliche Genehmigung durch SAP AG nicht gestattet. In dieser Publikation enthaltene Informationen können ohne vorherige Ankündigung geändert werden. Einige der von der SAP AG und ihren Distributoren vermarkteten Softwareprodukte enthalten proprietäre Softwarekomponenten anderer Softwareanbieter. Produkte können länderspezifische Unterschiede aufweisen. Die vorliegenden Unterlagen werden von der SAP AG und ihren Konzernunternehmen (SAP-Konzern) bereitgestellt und dienen ausschließlich zu Informationszwecken. Der SAP-Konzern übernimmt keinerlei Haftung oder Gewährleistung für Fehler oder Unvollständigkeiten in dieser Publikation. Der SAP-Konzern steht lediglich für Produkte und Dienstleistungen nach der Maßgabe ein, die in der Vereinbarung über die jeweiligen Produkte und Dienstleistungen ausdrücklich geregelt ist. Keine der hierin enthaltenen Informationen ist als zusätzliche Garantie zu interpretieren. SAP und andere in diesem Dokument erwähnte Produkte und Dienstleistungen von SAP sowie die dazugehörigen Logos sind Marken oder eingetragene Marken der SAP AG in Deutschland und verschiedenen anderen Ländern weltweit. Weitere Hinweise und Informationen zum Markenrecht finden Sie unter
©2013 SAP AG or an SAP affiliate company. All rights reserved.25 Public Drawing area Headline area White space The Grid
A Framework for Modeling and Execution of Infrastructure Contention Experiments Carmelo Ragusa, Philip Robinson and Sergej Svorobej MERMAT 2013, FIA, 7.
SAP HANA als Entwicklungsplattform Matthias Kupczak HANA Center of Excellence (CoE) Switzerland SAP Forum Juni 2013.
Adaptive Computing Technical Overview. The Adaptive Computing Controller Summary General information Building blocks and characteristics Adaptive Computing.
SAP Best Practices for Mining V1.606 for Australia Whats New.
Knowledge, Action, and Context Uwe Riss SAP Research, Karlsruhe.
Rapid database migration to Sybase Adaptive Server Enterprise Solution Details.
ECR Compliant Procurement Processes SAP Best Practices for Retail (US) SAP Best Practices.
SYSTEMATIC THOUGHT LEADERSHIP FOR INNOVATIVE BUSINESS Blending Complex Event Processing with the RETE Algorithm 2008 Kay-Uwe Schmidt, SAP Research.
SAP SOA Introduction to SAP SOA. Disclaimer This presentation outlines our general product direction and should not be relied on in making a purchase.
AL_Tables Jim Kirk August, ©2011 SAP AG. All rights reserved.2 Agenda – metadata tables 1) AL_Statistics 2) AL_History 3) AL_Lang & AL_Langxmlltext.
Solution Details SAP® Deposits Management rapid-deployment solution for transaction history analysis with SAP HANA.
Metrics and Techniques for Quantifying Performance Isolation in Cloud Environments Rouven Krebs (SAP AG), Christof Momm (SAP AG), Samuel Kounev (KIT) SPEC.
SAP for Utilities Enabling the Energy Supply of the Future SAP AMI & EAM, Perfect Together! Automated Meter Infrastructure & Enterprise Asset Management.
ECATT VSDK 3.0. SAP AG 2005, Title of Presentation / Speaker Name / 2 Target Group and Prerequisites Target Group Implementation and integration teams.
Oliver Mainka GIS Program Manager Office of the CEO Business Process Renovation Group SAP Labs, Palo Alto, CA SAP Location Intelligence Location Intelligence.
Inventory Valuation for Year End Closing SAP Best Practices.
Insights, Challenges and Opportunities for CIOs in Capital Intensive Industries Kieran OConnor, SAP – Moderator Vivek Bapat, SAP Pat Kennedy, OSIsoft Paul.
Title of Presentation, Speaker Name / 1 Content: SAP Best Practices Rapid implementation leveraging pre-packaged know-how Definition Benefits More info.
Solution Details SAP BusinessObjects Net Margin Analysis rapid deployment solution.
CITGO Refined Products Supply Chain Solution Combining R/3, IS-Oil, TSW and BW for an end-to-end solution Kathy Miller and Whitney Macormic, CITGO Petroleum.
Public Customer Feedback Getting Feedback – Getting Direction SAP Labs Canada, Knowledge Management Jon Druker November 2009 For the STC Montreal Chapter.
Computing Higher - SD Process – Topic 2 St Andrew’s High School Unit 2 Software Development Process.
©Ian Sommerville 2000Software Engineering, 6th edition. Chapter 29Slide 1 Chapter 29 Configuration Management.
1 Computer Systems & Architecture Lesson 3 5. Designing the Architecture.
2 Welcome To Defect Management Training Objective: The objective of this course is to learn about standards that emphasize a best practice approach for.
SAP Best Practices para Manufatura Discreta Brasil V1.604 Novidades.
1 Note content copyright © 2004 Ian Sommerville. NU-specific content copyright © 2004 M. E. Kabay. All rights reserved. Quality Management IS301 – Software.
Of An Expert System. Introduction What is AI? Intelligent in Human & Machine? What is Expert System? How are Expert System used? Elements of ES Who are.
Testing 1…2…3... Gail Kaiser, Columbia University April 18, 2013.
© 2016 SlidePlayer.com Inc. All rights reserved.