Software Testing

Software testing is the execution of software with test data from the problem domain. Software testing is the execution of software with test data from the problem domain. Static analysis (or static testing) involves analyzing the source code to determine errors. Static analysis (or static testing) involves analyzing the source code to determine errors. A test case is a specific input and the output that it should produce. A test case is a specific input and the output that it should produce.

Software Testing Exhaustive testing is the execution of every possible test cases in the problem domain (The set of all possible inputs is called the domain space). Exhaustive testing is the execution of every possible test cases in the problem domain (The set of all possible inputs is called the domain space). Generally a subset of the set of all possible test cases is chosen. Generally a subset of the set of all possible test cases is chosen. The subset should be chosen so it is representative of the set of all test cases. The subset should be chosen so it is representative of the set of all test cases.

Software Testing The two basic concerns in software testing are: The two basic concerns in software testing are: 1. What test cases to use (test case selection) 2. How many test cases are necessary (stopping criterion). A test case should always include the expected output. A test case should always include the expected output.

Software Testing Test case selection can be based on: Test case selection can be based on: 1. Specifications 2. The structure of the code 3. Data flow 4. Random selection

Test Coverage Criteria A test coverage criteria is a rule about how to select test cases and when to stop testing. A test coverage criteria is a rule about how to select test cases and when to stop testing. Test criterion A subsumes test criterion B if any test that satisfies criterion A also satisfies criterion B. Test criterion A subsumes test criterion B if any test that satisfies criterion A also satisfies criterion B.

Functional Testing In functional testing, the specification of the software is used to identify subdomains that should be tested. In functional testing, the specification of the software is used to identify subdomains that should be tested. One way is to generate a test case for every distinct ouput type. One way is to generate a test case for every distinct ouput type.

Functional Testing Example: The lengths of three sides of a triangle are entered and the type of triangle the lengths represent is output (either scalene, isosceles or equilateral). Example: The lengths of three sides of a triangle are entered and the type of triangle the lengths represent is output (either scalene, isosceles or equilateral). What is a “good” set of test cases? What is a “good” set of test cases?

Functional Testing Scalene: Scalene: Increasing size3, 4, 5 Decreasing size4, 3, 5 Largest as second4, 5, 3 Isosceles Isosceles a=b and c larger5, 5, 8 a=c and b larger5, 8, 5 b=c and a larger8, 5, 5 a=b and c smaller8, 8, 5 a=c and b smaller8, 5, 8 b=c and a smaller5, 8, 8 Not a triangle Not a triangle largest first 6, 4, 2 largest second 4, 6, 2 largest third 1, 2, 3 Bad Inputs Bad Inputs One bad-1, 2, 4 Two bad3, -2, -5 Three bad0, 0, 0 Equilateral Equilateral All sides equal 5, 5, 5

Test Matrices One way to formally identify subdomains is to build a matrix with specifications (conditions) in the rows and subdomains in the columns One way to formally identify subdomains is to build a matrix with specifications (conditions) in the rows and subdomains in the columns

Test Matrices Conditions a=b or a=c or b=c TTTTTFFF a=b and b=c TTFFFFFF a<=b+c Or b<=a+c Or c<=a+b TFTTFTTF a>=0 or b>=0 or c>=0 TFTFFTFF Sample test case 0,0,03,3,30,4,03,8,35,8,50,5,63,4,83,4,5 Expected Output Bad InputsEquilateralBad InputsNot triangle IsoscelesBad InputsNot triangle Scalene

Structural Testing Structural testing is based on the structure of the source code. Structural testing is based on the structure of the source code. The criterion is that every statement of source code should be executed by some test case. The criterion is that every statement of source code should be executed by some test case. The normal approach is to select test cases until a coverage tool indicates that all statements in the code have been executed (Every statement coverage) The normal approach is to select test cases until a coverage tool indicates that all statements in the code have been executed (Every statement coverage)

Structural Testing NodeSource Line3,4,53,5,30,1,04,4,4 ARead a, b, c**** BType=”scalene”**** Cif (a==b||b==c||a==c)**** DType=”isosceles”*** Eif (a===b && b==c)**** FType=”equilateral”* GIf (a>=b+c ||b>=a+c||c>=a+b)**** HType=”not a triangle”* Iif (a<=0||b<=0||c<=0)**** JType=”bad inputs”* KPrint type****

Structural Testing A more thorough test criterion is every branch covering (called the C1 test coverage). A more thorough test criterion is every branch covering (called the C1 test coverage). In C1 test coverage, the goal is to go each way out of every test decision. In C1 test coverage, the goal is to go each way out of every test decision.

Structural Testing Control Flow Graph for the previous code Control Flow Graph for the previous code

Structural Testing Arcs3,4,53,5,30,1,04,4,4 ABC-D*** ABC-E* D-E*** E-F* E-G*** F-G* G-H* G-I** H-I* I-J* I-K*** J-K*

Every Path Testing A more thorough test is every path testing which tests all possible paths through the control flow graph. A more thorough test is every path testing which tests all possible paths through the control flow graph. A path that cannot be taken is called an infeasible path. A path that cannot be taken is called an infeasible path.

Multiple Condition Coverage A multiple condition testing criterion requires that each primitive relation condition is evaluated both true and false. A multiple condition testing criterion requires that each primitive relation condition is evaluated both true and false. Lazy evaluation of expressions can be used to eliminate cases Lazy evaluation of expressions can be used to eliminate cases

Multiple Condition Coverage if (a==b||b==c||a==c) if (a==b||b==c||a==c) CombinationsPossible Test Cases Branch TXX3,3,4ABC-D FTX4,3,3ABC-D FFT3,4,3ABC-D FFF3,4,5ABC-E

Test Plans A test plan states: What the items to be tested are At what level they will be tested What sequence they are to be tested in How the test strategy will be applied to the testing of each item A description of the test environment.

What to consider for the Test Plan From IEEE 829 Standard for Software Test Documentation: 1. Test Plan Identifier 2. References 3. Introduction 4. Test Items

What to consider for the Test Plan 5. Software Risk Issues 6. Features to be Tested 7. Features not to be Tested 8. Approach 9. Item Pass/Fail Criteria 10. Suspension Criteria and Resumption Requirements

What to consider for the Test Plan 11. Test Deliverables 12. Remaining Test Tasks 13. Environmental Needs 14. Staffing and Training Needs 15. Responsibilities 16. Schedule 17. Planning Risks and Contingencies 18. Approvals 19. Glossary

What I want included in the test plan List of responsibilities (and who does them) – –Who runs the test – –Who approves the results Unit tests to be run with expected results – –Objectives – –Approach – –Include a description of the type of data used for unit testing and why that data is being used – –Pass/fail criteria – –When the tests are to be run (timeline)

What I want included in the test plan Integration tests to be run with expected results – –Objectives – –Approach – –Include a description of the type of data used for integration testing and why that data is being used – –Pass/fail criteria – –When the tests are to be run (timeline)

What I want included in the test plan Systems/Acceptance tests to be run with expected results – –Objectives – –Approach – –Include a description of the type of data used for systems/acceptance testing and why that data is being used – –Pass/fail criteria – –When the tests are to be run (timeline)

What I want included in the test plan Alpha testing to be done – –Objectives – –Approach – –Features to be tested – –Pass/fail criteria – –When the tests are to be run (timeline)

What I want included in the test plan Beta testing – –Objectives – –Approach – –Who is participating in the beta testing – –Pass/fail criteria – –What I want included in the test plan

Deliverables Test Design Specification: detailing test conditions and the expected results as well as test pass criteria. Test Case Specification: specifying the test data for use in running the test conditions identified in the Test Design Specification

Deliverables Test Procedure Specification: detailing how to run each test, including any set-up preconditions and the steps that need to be followed Test Item Transmittal Report: reporting on when tested software components have progressed from one stage of testing to the next

Deliverables Test Log: recording which tests cases were run, who ran them, in what order, and whether each test passed or failed

Deliverables Test Incident Report: detailing, for any test that failed, the actual versus expected result, and other information intended to throw light on why a test has failed. These include the expected results being wrong, the test being run wrongly, or inconsistency in the requirements meaning that more than one interpretation could be made.

Deliverables- Test Incident Report The report consists of all details of the incident such as actual and expected results, when it failed, and any supporting evidence that will help in its resolution. The report will also include, if possible, an assessment of the impact of an incident upon testing.

Deliverables- Test Summary Report Test Summary Report: A management report providing any important information uncovered by the tests accomplished, and including: Assessments of the quality of the testing effort The quality of the software system under test, and statistics derived from Incident Reports.

Deliverables- Test Summary Report The report also records: What testing was done How long it took in order to improve any future test planning. This final document is used to indicate whether the software system under test is fit for purpose according to whether or not it has met acceptance criteria defined by project stakeholders

IEEE Standards IEEE Standard for Software Test Documentation IEEE Standard for Software Unit Testing IEEE Standard for Software Verification & Validation Plans IEEE Guide for Software Verification & Validation Plans

Sample Test Plans solutions/testing-a-sample-test-plan solutions/testing-a-sample-test-plan-3283