Presentation is loading. Please wait.

Presentation is loading. Please wait.

Software Testing Methods and Techniques Philippe CHARMAN Last update: 21-05-2014.

Similar presentations

Presentation on theme: "Software Testing Methods and Techniques Philippe CHARMAN Last update: 21-05-2014."— Presentation transcript:

1 Software Testing Methods and Techniques Philippe CHARMAN Last update:

2 Summary The box approach –Black Box Testing –White Box Testing –Gray Box Testing Preliminary Testing –Smoke Testing –Sanity Testsing Informal Testing Methods –Monkey Testing –Exploratory Testing Reducing the input data set –Equivalence partitioning –Boundary Value Analysis Test Case Generation –Pairwise testing Fault injection methods –Error Handling Testing –Bebugging –Mutation Testing

3 About the Testing Terminolgy Software Testing has developed many redundant or overlapping terminologies over the last decade For instance black box testing is equivalent to specification-based testing or functional Testing The reference is the ISTQB glossary

4 Black Box Testing Testing software based on output requirements and without any knowledge of the internal structure or coding in the program Input -> -> Output Black-box testing is focused on results Synonyms: specification-based testing, functional testing Program

5 Black Box Testing Pros –Quick understanding of the system behaviour –Simplicity: design the input and get the output –Unaffiliated opinion Cons –Too many redundant test cases may be created –Missing test cases

6 Black Box Test Design Techniques boundary value analysis cause-effect graphing classification tree method decision table testing domain analysis elementary comparison testing equivalence partitioning n-wise testing pairwise testing process cycle test random testing state transition testing syntax testing use case testing user story testing

7 White Box Testing White box testing is when the tester has access to the internal data structures and algorithms including the code that implement these Some types: –Code review –Code coverage –Performance –etc. Usually performed with dedicated QA tools (static or dynamic tools) Synonyms: clear-box testing, glass box testing, code-based testing, logic-coverage testing, logic-driven testing, structural testing

8 White Box Testing Pro –May detect missing test cases with code coverage –May detect bugs with dedicated tools (memory debuggers, static analysis code tools, etc.) –Unit tests can be easily automated Cons –Programming skills required –Need to understand the implementation that may be complicated –Time-expensive activity

9 White Box Test Design Techniques branch testing condition testing data flow testing decision condition testing decision testing LCSAJ testing modified condition decision testing multiple condition testing: path testing statement testing:

10 Gray Box Testing Combination of black box testing and white box testing Idea: if one knows something about how the product works on the inside, one can test it better, even from the outside

11 Gray Box Testing Pros –Better test suite with less redundant and more relevant test cases Cons –The tester may loose his/her unaffiliated opinion

12 Testing Levels Who Tests What Component Testing Integration Testing System Testing Low Level Code High Level Code Module 1 Low Level Code High Level Code Module 2 Customer DB Acceptance Testing Developer Tester Consultant Customer App Server

13 Smoke Testing In electronics jargon: when a new electrical system is to be tested, plug it into the power –If there’s smoke, stop the tests –If no smoke, more tests can be executed In software engineering: –Preliminary tests before advanced tests, for instance when assembling the differents modules –A smoke test is a quick test done at integration time Good practice: perform smoke tests after every build in continuous integration system A smoke test should not be confused with a stress test!

14 Sanity Testing Quick, broad, and shallow testing 2 possible goals: –Determine whether it is reasonable to proceed with further testing –Determine whether the very last release candidate can be shipped (provided more advanced and detailed tests have been performed on previous release candidates) Example when testing a GUI: checking all the actions from the menus work for at least one test case

15 Smoke Testing vs. Sanity Testing According to a unfortunately shared opinion: –A smoke test determines whether it is possible to continue testing e.g. “File > Open” doesn’t work at all whatever the format of the data -> useless to perform more tests –A sanity test determines whether it is reasonable to continue testing e.g. “File > Open” works only for a subset of the formats of the data -> we should be able to perform more tests According to ISTQB: –smoke testing = sanity testing

16 Monkey Testing Monkey testing is random testing performed by –a person testing an application on the fly with no idea in mind –or an automated test tool that generate random input Depending on the characteristics of the random input, we can distinguish: –Dumb Monkey Testing: uniform probability distribution –Smart Monkey Testing: sophisticated probability distribution

17 Monkey Testing Pro: –May find severe bugs not found by developers that make the application crash Con: –Less efficient than directed testing

18 Exploratory Testing Not well-know technique that consists of –Learning the product –Designing the test plan –Executig the test –Interpreting the results in parallel Exploratory testing is particularly suitable if requirements and specifications are incomplete, or if there is lack of time

19 Exploratory Testing Pros: –less preparation is needed –important bugs are found quickly –and at execution time, the approach tends to be more intellectually stimulating than execution of scripted tests. Cons: –skill testers are needed –tests can't be reviewed in advance –difficult trace which tests have been run

20 Equivalence Partitioning A process of selecting test cases/data by identifying the boundaries that separate valid and invalid conditions Goal: reduce the total number of test cases to a finite set of testable test cases, still covering maximum requirements Example: values of months as integer: | | invalid partition 1 valid partition invalid partition 2 Three selected values: –A negative value: eg -2 –A valid value: eg 5 –A value greater than 12, eg 15

21 Boundary Value Analysis A process of selecting test cases/data by identifying the boundaries that separate valid and invalid conditions. Goal: reduce the total number of test cases to a finite set of testable test cases, still covering maximum requirements. Example: values of months as integer (1=Jan, 12=Dec): | | invalid partition 1 valid partition invalid partition 2 Four selected values: 0,1 and 12,13 Alternative: six selected values: 0,1,2 and 11, 12,13

22 Pairwise Testing Effective test case generation technique that is based on the observation that most faults are caused by interactions of at most two factors Pairwise-generated test suites cover all combinations of two therefore are much smaller than exhaustive ones yet still very effective in finding defects Can be extended to t-wise testing Synonym: All-pair testing

23 Pairwise Testing 3 variables x,y,z with 2 possibles values 1 and 2 Exhaustive test cases: 2 3 = 8 One solution among others: xyz Test #1121 Test #2112 Test #3222 Test #4211

24 Pairwise Testing

25 Basis Path Testing One testing strategy, called Basis Path Testing by McCabe who first proposed it, is to test each linearly independent path through the program; in this case, the number of test cases will equal the cyclomatic complexity of the program.[1]testing[1]

26 Error Handling Testing Objective: to ensure that an application is capable to handling incorrect input data Usually neglected by developers in unit tests Benefit: improving the code coverage by introducing invalid input data to test error handling code paths

27 Error Handling Testing What kind of bugs can be found Severe –application frozen or crash Major: –Undo doesn’t work Minor: –Missing button Cancel in dialog box –Cryptic error messages –Non-localized error messages

28 Bebugging The process of intentionally adding known faults to those already in a computer program for the purpose of monitoring the rate of detection and removal, and estimating the number of faults remaining in the program. Bebugging is a type of fault injection Synonym: fault seeding

29 Mutation Testing Suppose a test suite has been written to check a feature or a function Some questions that may arise: –What is the quality of this test suite ? –Are the relevant input values well tested ? –How to be sure the test suite covers all possible test cases ? Code coverage and pairwise test set generation can help Another less known technique is mutation testing

30 Mutation Testing Idea: bugs are intentionally introduced into source code Each new version of the source code is called a mutant and contains a single bug The mutants are executed with the test suite If a mutant successfully pass the test suite: –either the test suite needs to be completed –or the execution of the mutant code is equivalent to the original code A test suite which does reject the mutant code is considered defective

31 Examples of Non Equivalent Mutant Code Original CodeMutant Code if (a && b) c = 1; else c = 0; if (a || b) c = 1; else c = 0; if (a && b) c = 1; else c = 0; if (a && b) c = -1; else c = 0;

32 Example of Equivalent Mutant Code Original CodeEquivalent Mutant Code int index=0; while (...) {...; index++; if (index == 10) break; } int index=0; while (...) {...; index++; if (index >= 10) break; }

33 Mutation Testing Pros –Strenghten the quality of test data New test cases can be found Code coverage is increased Cons –Rather expensive technique for augmenting an existing test suite –The choice of mutation tool and operator set can play an important role in determining how efficient mutation analysis is for producing new test cases

34 References –Myers, Glenford J. (1979). The Art of Software Testing. John Wiley and Sons. ISBN –Kaner, Cem; Falk, Jack and Nguyen, Hung Quoc (1999). Testing Computer Software, 2nd Ed.. New York, et al: John Wiley and Sons, Inc.. pp. 480 pages. ISBN –Ben Smith, Laurie Williams: Should Software Testers use Mutation Analysis to Augment a Test Set? Journal of Systems and Software, 2009 –Kaner, Cem; Bach, James; Pettichord, Bret (2001). Lessons Learned in Software Testing. John Wiley & Sons. ISBN

35 Further Reading

Download ppt "Software Testing Methods and Techniques Philippe CHARMAN Last update: 21-05-2014."

Similar presentations

Ads by Google