Presentation is loading. Please wait.

Presentation is loading. Please wait.

Sept 2008 1 Data Mutation Testing Applied to a Modelling Tool Hong Zhu Dept. of Computing, Oxford Brookes Univ., Oxford, UK

Similar presentations

Presentation on theme: "Sept 2008 1 Data Mutation Testing Applied to a Modelling Tool Hong Zhu Dept. of Computing, Oxford Brookes Univ., Oxford, UK"— Presentation transcript:

1 Sept 2008 1 Data Mutation Testing Applied to a Modelling Tool Hong Zhu Dept. of Computing, Oxford Brookes Univ., Oxford, UK Email: Lijun Shan Dept of Computer Science, National Univ. of Defence Technology, Changsha, China

2 Sept 2008 2 MOTES 2008 Outline Motivation Overview of existing work on software test case generation The challenges to software testing The Data Mutation Testing Method Basic ideas Process Measurements A Case study of a modelling tool Subject software under test The mutation operators Experiment process Main results Perspectives and future works Potential applications Integration with other black box testing methods

3 Sept 2008 3 MOTES 2008 Motivation Test case generation One of the most labour intensive tasks in practices Huge impact on test effectiveness and efficiency Need to meet multiple goals Reality: to represent real operation of the system Coverage: functions, program code, input/output data space, and their combinations Efficiency: not to overkill, easy to execute, etc. Effective: capable of detecting faults, which implies easy to check the correctness of programs output Externally useful: help with debugging, reliability estimation, etc.

4 Sept 2008 4 MOTES 2008 Existing work Program-based approach Derive test cases from program code Specification-based approach Derive test cases from either formal or semi- formal specifications of the required functions and/or the designs Random testing Through random sampling over input domain based on probabilistic models of the operation of the software under test. Domain specific techniques

5 Sept 2008 5 MOTES 2008 Existing Work 1: Program-based Static: analysis of code without execution, e.g. symbolic execution Path oriented Howden, W. E. (1975, 1977, 1978); Ramamoorthy, C., Ho, S. and Chen, W. (1976) ; King, J. (1975) ; Clarke, L. (1976) ; Xie T., Marinov, D., and Notkin, D. (2004); Zhang. J. (2004), Xu, Z. and Zhang, J. (2006) Goal oriented DeMillo, R. A., Guindi, D. S., McCracken, W. M., Offutt, A. J. and King, K. N. (1988) ; Pargas, R. P., Harrold, M. J. and Peck, R. R. (1999); Gupta, N., Mathur, A. P. and Soffa, M. L. (2000); Dynamic: through execution of the program Korel, B. (1990), Beydeda, S. and Gruhn, V. (2003) Hybrid: combination of dynamic execution with symbolic execution, e.g. concolic techniques Godefroid, P., Klarlund, N., and Sen, K.. (2005); Techniques: Constraint solver, Heuristic search, e.g. genetic algorithms: McMinn, P. and Holcombe, M. (2003), Survey: McMinn, P. (2004)

6 Sept 2008 6 MOTES 2008 Formal specification-based: First order logic, Z spec and Logic programs: Tai, K.-C. (1993); Stocks, P. A. and Carrington, D. A. (1993) ; Ammann, P. and Offutt, J. (1994) ; Denney, R. (1991) Algebraic specification: Bouge, L., Choquet, N., Fribourg, L. and Gaudel, M.-C. (1986) ; Doong, R. K. and Frankl, P. G. (1994) ; Chen, H. Y., Tse, T. H. and Chen, T. Y. (2001) ; Zhu (2007); Finite state machines: Fujiwara, S., et al.. (1991) ; Lee, D. and Yannakakis, M. (1996) ; Hierons, R. M. (2001) ; Zhu, H., Jin, L. & Diaper, D. (1999) ; Petri nets: Morasca, S. and Pezze, M. (eds). (1990) ; Zhu, H. and He, X. (2002) Model-based: SSADM models: Zhu, H., Jin, L. and Diaper, D. (1999, 2001); UML models: Offutt, J. and Abdurazik, A. (2000) ; Tahat, L. H., et al. (2001); Hartman, A. and Nagin, K. (2004); Li, S., Wang, J. and Qi, Z.-C. (2004) ; Techniques: Constraint solving; Theorem prover; Model-checker Existing Work 2: Specification-based

7 Sept 2008 7 MOTES 2008 Profile-based: Sampling over an existing operation profile at random Stochastic model based: Use a probabilistic model of software usages Markov chain: Avritzer, A. Larson, B. (1993) ; Avritzer, A. Weyuker, E. J. (1994) ; Whittaker, J. A. and Poore, J. H. (1993) ; Guen, H. L., Marie, R. and Thelin, T. (2004) ; Prowell, S. J. (2005) Stochastic automata networks: Farina, A. G., Fernandes, P. and Oliveira, F. M. (2002, 2004) ; Bayesian networks: Fine, S. and Ziv, A. (2003) Adaptive random testing: Even spread of randomly test cases Chen, T. Y., Leung, H. and Mak, I. K. (2004), etc. Variants: Mirror, Restricted, and Probabilistic ART Existing Work 3: Random testing

8 Sept 2008 8 MOTES 2008 Database applications Zhang, J., Xu, C. and Cheung, S. C. (2001) Spreadsheets: Fisher, M., Cao, M., Rothermel, G., Cook, C. and Burnett, M. (2002) Erwig, M., Abraham, R., Cooperstein, I., and Kollmansberger S. (2005) XML Scheme: Lee, S. C. and Offutt, J. (2001) ; Li, J. B. and Miller, J. (2005) Compiler: See Boujarwah, A. S. and Saleh, K. (1997) for a survey. Existing Work 3: Domain-specific techniques

9 Sept 2008 9 MOTES 2008 The Challenge How to generate adequate test cases of high reality for programs that process structurally complex inputs? Structural complexity: A large number of elements Explicitly represented relationships between the elements A number of constraints imposed on the relationships Meaning of the data depends on not only the values of the elements, but also the relationships and thus their processing Reality: Likely or close to be a correct real input in the operation of the system Likely or close to be an input that contains errors that a user inputs to the system in operation Examples : CAD, Word processor, Web browser, Spreadsheets, Powerpoint, Software modelling tools, Language processor, Theorem provers, Model-checkers, Speech recognition, Hand writing recognition, Search engine,…

10 Sept 2008 10 MOTES 2008 Basic Ideas of Data Mutation Testing 1.Preparing the seeds, i.e. a small set of test cases Contain various types of elements and relationships between them Highly close to the real input data Easy to check their correctness 2.Generating mutant test cases by modifying the seeds slightly Preserve the validity of the input Change at one place a time unless imposed by the constraints (but may use second order even higher order mutants) Make as many different mutants as possible 3.Executing the software under test on both seeds and their mutants What to observe: programs correctness on both seeds and mutants the differences of the programs behaviours on seed and their mutants Uses of metrics and measurements seeds are sufficient mutations are effective and/or sufficient Feedback to step 1 and 2 if necessary, or to improve the observation.

11 Sept 2008 11 MOTES 2008 Illustrative Example Triangle classification Input: x, y, z: Natural Numbers; Output: {equilateral, isosceles, scalene, non-triangle} Seeds: The lengths of the sides The type of triangles Non-triangle(x=3, y=5, z=9)t4t4 Scalene(x=5, y=7, z=9)t3t3 Isosceles(x=5, y=5, z=7)t2t2 Equilateral(x=5, y=5, z=5)t1t1 Expected outputInputID

12 Sept 2008 12 MOTES 2008 Mutation operators IVP: Increase the value of a parameter by 1; DVP: Decrease the value of a parameter by 1; SPL: Set the value of a parameter to a very large number, say 1000000; SPZ: Set the value of a parameter to 0; SPN: Set the value of a parameter to a negative number, say - 2; WXY: Swap the values of parameters x and y; WXZ: Swap the values of parameters x and z; WYZ: Swap the values of parameters y and z; RPL: Rotate the values of parameters towards left; RPR: Rotate the values of parameters towards right.

13 Sept 2008 13 MOTES 2008 Generation of mutant test cases For example, by applying the mutation operator IVP to test case t 1 on parameter x, we can obtain the following test case t 5. IVP(t 1, x) = t 5 = Input: (x=6, y=5, z=5). Total number of mutants: (5*3 +5)*4 = 80 Covering all sorts of combinations of data elements Systematically produced from the four seeds

14 Sept 2008 14 MOTES 2008 Execution of program and classification of mutants A mutant is classified as dead, if the execution of the software under test on the mutant is different from the execution on the seed test case. Otherwise, the mutant is classified as alive. For example For a correctly implemented Triangle Classification program, the execution on the mutant test case t 5 will output isosceles while the execution on its seed t 1 will output equilateral. TrC(t 5 ) TrC(t 1 ) t 5 is dead It depends on how you observe the behaviour!

15 Sept 2008 15 MOTES 2008 Analyse test effectiveness Reasons why a mutant can remain alive: The mutant is equivalent to the original with respect to the functionality or property of the software under test. RPL(t 1 )=t 1 The observation on the behaviour and output of the software under test is not sufficient to detect the difference RPL(t 2 )= t 6 = Input: (x=5, y=7, z=5). The software is incorrectly designed and/or implemented so that it is unable to differentiate the mutants from the original. Same output, but different execution paths for a correct program.

16 Sept 2008 16 MOTES 2008 Measurements of Data Mutation Equivalent mutant score EMS: A high equivalent mutant score EMS indicates that the mutation operators have not been well designed to achieve variety in the test cases. Live mutant score LMS: A high LMS indicates that the observation on the behaviour and output of the software under test is insufficient. Typed live mutant score LMS, where is a type of mutation operators A high LMS reveals that the program is not sensitive to the type of mutation probably because a fault in design or implementation. Number of equivalent mutants Total number of mutants Number of life mutants

17 Sept 2008 17 MOTES 2008 Process of Data Mutation Testing

18 Sept 2008 18 MOTES 2008 Analysis of Program Correctness Can data mutation testing be helpful to the analysis of program correctness? Consider the examples in Triangle Classification: IVP or DVP to test case t 1, we can expect the output to be isosceles. For the RPL, RPR, WXY, WYZ, and WYZ mutation operators, we can expect that the program should output the same classification on a seed and its mutant test cases. If the softwares behaviour on a mutant is not as expected, an error in the software under test can be detected.

19 Sept 2008 19 MOTES 2008 Case study of a modelling tool The subject CAMLE: Caste-centric Agent-oriented Modelling Language and Environment Automated modelling tool for agent-oriented methodology Developed at NUDT of China Potential threats to the validity of the case study Subject is developed by the tester The developer is not professional software developer Validation of the case study against the potential threats The test method is black box testing. The knowledge of the code and program structure affect the outcomes. The subject was developed before the case study and no change at all was made during the course to enable the case study to be carried out. In software testing practice, systems are often tested by the developers. The developer is a capable master degree student with sufficient training at least equivalent to an average programmer. The correctness of the programs output can be judges objectively.

20 Sept 2008 20 MOTES 2008 Complexity of the Input Data Input: models in CAMLE language Multiple views: a caste diagram that describes the static structure of a multi-agent system, a set of collaboration diagrams that describe how agents collaborate with each other, a set of scenario diagrams that describe typical scenarios namely situations in the operation of the system, and a set of behaviour diagrams that define the behaviour rules of the agents in the context of various scenarios. Well-formedness constraints Each diagram has a number of different types of nodes and arcs, etc. Each diagram and the whole model must satisfy a set of well- formedness conditions to be considered as a valid input (e.g. the types of nodes and arcs must match with each other)

21 Sept 2008 21 MOTES 2008 The Function to Be Tested Consistency checker Consistency constraints are formally defined in first order logic Potential threat to the validity The program is not representative. Validation of the case study The programs input is structurally complex The program is non-trivial 414Inter-model 88Inter-diagram 10Intra-diagram Intra- model GlobalLocal Vertical Consistency Horizontal Consistency Table 1. Summary of CAMLEs Consistency Constraints

22 Sept 2008 22 MOTES 2008 Types of Data Mutation Operators Delete an existing env node in a sub-collaboration diagram Delete env node12 Generate a sub-collaboration diagram for an existing node Add sub diagram11 Replace an existing node with a new node of another type Change node type10 Rename an existing node in a diagramRename node9 Delete an existing node in a diagramDelete node8 Replicate an existing node in a diagramReplicate node7 Add an edge of some type to a diagramAdd edge6 Add a node and link it to an existing nodeAdd node with edge5 Add a node of some type to a diagramAdd node4 Change the title of an existing diagramRename diagram3 Delete an existing diagramDelete diagram2 Add a collaboration or behaviour or scenario diagramAdd diagram1 DescriptionOperator typeNo.

23 Sept 2008 23 MOTES 2008 13Rename env nodeRename an existing environment node in a sub- collaboration diagram 14Delete node annotationRemove an annotation on an existing node 15Replicate edgeReplicate an existing non-interaction edge 16Delete edgeDelete an existing edge in a diagram 17Change edge associationChange the Start or End node of an existing edge 18Change edge directionReverse the direction of an existing edge 19Change edge typeReplace an existing edge in a diagram with a new edge of another type 20Replicate interaction edge Replicate an existing interaction edge without Action List 21Replicate interactionReplicate an existing interaction edge with Action List 22Change edge annotationChange the Action List annotated to an existing interaction edge 23Delete edge annotationDelete the Action List of an existing interaction edge 24Change edge end to envChange the Start or End node of an existing edge to an env node

24 Sept 2008 24 MOTES 2008 The Seed Test Cases Models developed in previous case studies of agent- oriented software development methodology The evolutionary multi-agent Internet information retrieval system Amalthaea (originally developed at MIT media lab); Online auction web service; The agent-oriented model of the United Nations Security Council on the organisational structure and the work procedure to pass resolutions at UNSC. All seeds passed consistency check before the case study started No change was made to these seeds in this case study

25 Sept 2008 25 MOTES 2008 The Seed Test Cases and Their Mutants

26 Sept 2008 26 MOTES 2008 The Results: Fault Detecting Ability 5114 (97%)61 (52%)118Total 019 (100%)12 (63%)19 Transposition of statements 114 (93%)9 (60%)15 Incorrect expression 031 (100%)13 (42%)31 Omission of statements 021 (88%)14 (58%)24 Incorrect variable Computation 217 (100%)8 (47%)17Path selection 212 (100%)5 (42%)12Missing path Domain IndigenousInserted By mutants By seeds No. of Detected Faults No. of Inserted Faults Fault Type

27 Sept 2008 27 MOTES 2008 Detecting Design Errors In the case study, we found that a large number of mutants remain alive 16.47%652212867808Total 15.43%9151671082UNSC 12.94%28384223260Auction 20.11%27696973466Amalthaea %Dead#Alive#Dead#MutantSeed Table. The numbers of alive and dead mutants Review: Three possible reasons: (a) improper design of data mutation operators, (b) insufficient observation on the behaviour and output (c) defects in the software under test.

28 Sept 2008 28 MOTES 2008 Statistics on Amalthaea test suite Some typed mutation score is very low Design of consistency checker has errors! Especially, the consistency constraints are weak.

29 Sept 2008 29 MOTES 2008 Results: Detecting Design Errors Hypothesis Design of the tool is weak in detecting certain types of inconsistency or incompleteness Validation of the hypothesis Strengthening the well-formedness constraints Strengthening the consistency constraints: 3 constraints modified Introducing new completeness constraints: 13 new constraints introduced Test again using the same seeds and the same mutation operators A significant change in the statistic data is observed. 85.18% 106060927152Total 82.76% 171821992UNSC 83.33% 51625793095Auction 87.83% 37326923065Amalthaea %Dead#Alive#Dead#MutantSeed Table. The statistics of alive and dead mutants after modification

30 Sept 2008 30 MOTES 2008 Test Adequacy Our experiments show that high test adequacy can be achieved through data mutation. Coverage of input data space Measured by the coverage of various kinds of mutants Coverage of program structure Measured by code coverage (equivalent to the branches covered) Coverage of the functions of the requirements Measured by the consistency constraints used in checking Two factors the determines the test adequacy: the seeds the mutation operators

31 Sept 2008 31 MOTES 2008 Coverage of scenario diagram variants 4000 17 11011016 1002814 10028 2404209 2404208 2404207 2400 6 1703145 2107144 30123 30122 31111 TotalUNSCAuctionAmalthaea Mutation operator type

32 Sept 2008 32 MOTES 2008 Coverage of Program Structure and Functions The test data achieved 100% coverage of the functions of the consistency checker and 100% of the branches in the code.

33 Sept 2008 33 MOTES 2008 Test Cost Source of costAmount in case study Design and implementation of data mutation operators 1.5 man-month Development of seed test cases0 man-month Analysis of program correctness on each test case 2 man-month (estimated) The seeds were readily available from previous case studies of the tool. Table. Summary of the test cost spent in the case study

34 Sept 2008 34 MOTES 2008 Analysis Programs Correctness The experiment took the black-box approach The output on a test case consists of Whether the input (a model) is consistent and complete The error message(s) and/or warning message(s), if any The expected output on a mutant is specified 14, (Interaction edges in the main collaboration diagram) E0165 6, (Caste nodes in the main collaboration diagram) E0042 5, (Agent nodes in the main collaboration diagram) E0031 Add a new Collaboration diagram / Top of model 1 #Messages, Message Content Message ID Violated Constraint Expected Output Operator /Location Mutant No.

35 Sept 2008 35 MOTES 2008 Experiments The experiments Mutants are selected at random The programs correctness on each mutant is checked manually Time is measured for how long it needs to check the correctness of the program on each test case Two experiments were conducted Experiment 1 1 mutant selected at random from each set of the mutants generated by one type of mutation operator (24 mutants in total) Detected 2 faults in the checker and 1 fault in other parts of the tool Experiment 2 22 live mutants from the Amalthaea suite selected at random Detected 2 faults in the other parts of the tool

36 Sept 2008 36 MOTES 2008 The Experiment Data 211DeadNon-equivalent 11AliveNon-equivalent 00DeadEquivalent 234AliveEquivalent #Detected Faults#MutantsAlivenessType of Mutant Results: Checking correctness on dead mutants: 3 minute/per mutant Checking correctness on live mutants: 1 minute/per mutant

37 Sept 2008 37 MOTES 2008 Related Works Mutation testing Program or specification is modified Used as a criteria to measure test adequacy Data mutation testing adopted the idea of mutation operators, but applied to test cases to generate test case, rather than to measure adequacy. Meek and Siu (1989) Randomisation in error seeding into programs to test compiler Adaptive Random Testing (Chen, et al. 2003, 2004) Random test cases as far apart as possible Not yet applied to structurally complex input space Data perturbation testing (Offutt, 2001) Test XML message for web services As a application specific technique and applicable to XML files Metamorphic testing (Chen, Tse, et al. 2003) As a test oracle automation technique and focus on the metamorphic relations rather than to generate test cases Could be integrated with data mutation method

38 Sept 2008 38 MOTES 2008 Future Work More case studies with potential applications Security control software: Role-Base Access Control Input: Role model, User assignments P User assignments: Users P(Roles) Virus detection Input: files infected by virus Virus are programs in assembly/binary code format One virus may have many variants obtained by equivalent transformation of the code. Spreadsheet processing software and spreadsheets applications Input: spreadsheets

39 Sept 2008 39 MOTES 2008 Perspectives and Future Work Integration of data mutation testing, metamorphic testing and algebraic testing methods Letbe the program under test Data mutation testing generates test cases using a set of data mutation operators Metamorphic testing used a set of metamorphic relations to check output correctness We can use i to define metamorphic relations as follows:

40 Sept 2008 40 MOTES 2008 Example Consider the Triangle Classification program P The following is a metamorphic relation P(t)= equilateral P(IPV(t))= isosceles For each of the data mutation operators = WXY, WXZ, WYZ, RPL, or RPR, the following is a metamorphic relation P( (t))=P(t) We observed in case study that data mutation operators are very helpful to find metamorphic relations.

41 Sept 2008 41 MOTES 2008 Integration with Algebraic Testing In algebraic software testing, axioms are written in the form of T 1 =T 1 ^ T 2 =T 2 ^ … ^ T n =T n => T=T, Where T i, T i are terms constructed from variables and function/procedure/methods of the program under test. The integration of data mutation testing, metamorphic testing and algebraic testing by developing A black box software testing specification language An automated tool to check metamorphic relations Using observation context to check if a relation is true To allow user defined data mutation operators to be invoked To allow metamorphic relations to be specified

42 Sept 2008 42 MOTES 2008 Screen Snapshot of Algebraic Testing Tool CASCAT

43 Sept 2008 43 MOTES 2008 References Liang Kong, Hong Zhu and Bin Zhou, Automated Testing EJB Components Based on Algebraic Specifications, Proc. of TEST 2007, IEEE CS Press 2007. Lijun Shan and Hong Zhu, Generating Structurally Complex Test Cases by Data Mutation: A Case Study of Testing an Automated Modelling Tool, Special Issue on Automation of Software Test, the Computer Journal, (accepted). Shan, L. and Zhu, H., Testing Software Modelling Tools Using Data Mutation, Proc. of AST06, ACM Press, 2006, pp43-49. Zhu, H., A Note on Test Oracles and Semantics of Algebraic Specifications, Proc. of QSIC'03, IEEE CS Press, 2003, pp91-99. Zhu, H. and Shan, L., Caste-Centric Modelling of Multi-Agent Systems: The CAMLE Modelling Language and Automated Tools, in Beydeda, S. and Gruhn, V. (eds) Model-driven Software Development, Research and Practice in Software Engineering, Vol. II, Springer, 2005, pp57-89.

Download ppt "Sept 2008 1 Data Mutation Testing Applied to a Modelling Tool Hong Zhu Dept. of Computing, Oxford Brookes Univ., Oxford, UK"

Similar presentations

Ads by Google