1. Practical Testing Techniques

2. Verification and Validation Validation –does the software do what was wanted? “Are we building the right system?” –This is difficult to determine and involves subjective judgments (reviews, etc.) Verification –does the software meet its specification? “Are we building the system right?” –This can only be objective if the specifications are sufficiently precise –Implications? Everything must be verified –…including the verification process itself

3. Testing Testing is a verification technique –critically important for quality software Industry averages: –30-85 errors per 1000 lines of code; –0.5-3 errors per 1000 lines of code not detected before delivery. The ability to test a system depends on a thorough, competent requirements document.

4. Goals of Testing Goal: show a program meets its specification –But: testing can never be complete for non- trivial programs What is a successful test? –One in which no errors were found? –One in which one or more errors were found?

5. Goals of Testing - 2 Testing should be: –repeatable if you find an error, you want to repeat the test to show others if you correct an error, you want to repeat the test to check that you fixed it –systematic random testing is not enough select test sets that are representative of real uses select test sets that cover the range of behaviors of the program –documented keep track of what tests were performed, and what the results were

6. Goals of Testing - 3. Therefore you need a way to document test cases showing: –the input –the expected output –the actual result These test plans/scripts are critical to project success!

7. Errors/Bugs Errors of all kinds are known as “bugs”. Bugs come in two main types: –compile-time (e.g., syntax errors) which are cheap to fix –run-time (usually logical errors) which are expensive to fix.

8. Testing Strategies Never possible for designer to anticipate every possible use of system. Systematic testing is therefore essential. Offline strategies: 1.syntax checking & “lint” testers; 2.walkthroughs (“dry runs”); 3.inspections Online strategies: 1.black box testing; 2.white box testing.

9. Syntax Checking Detecting errors at compile time is preferable to having them occur at run time! Syntax checking will simply determine whether a program “looks” acceptable “lint” programs try to do deeper tests on code: –will detect “this line will never be executed” –“this variable may not be initialized” Compilers do a lot of this in the form of “warnings”.

10. Inspections Formal procedure, where a team of programmers read through code, explaining what it does. Inspectors play “devils advocate”, trying to find bugs. Time consuming process! Can be divisive/lead to interpersonal problems. Often used only for critical code.

11. Walkthroughs Similar to inspections, except that inspectors “mentally execute” the code using simple test data. Expensive in terms of human resources. Impossible for many systems. Usually used as discussion aid. Inspections/walkthroughs typically take 90- 120 minutes. Can find 30%-70% of errors.

12. Black Box Testing Generate test cases from the specification –i.e. don’t look at the code Advantages: –avoids making the same assumptions as the programmer –test data is independent of the implementation –results can be interpreted without knowing implementation details

13. Consider this Function function max_element ($in_array) { /* Returns the largest element in $in_array */ … return $max_elem; } function max_element ($in_array) { /* Returns the largest element in $in_array */ … return $max_elem; }

14. A Test Set Is this enough testing?

15. “Black Box” Testing In black box testing, we ignore the internals of the system, and focus on the relationship between inputs and outputs. Exhaustive testing would mean examining system output for every conceivable input. –Clearly not practical for any real system! Instead, we use equivalence partitioning and boundary analysis to identify characteristic inputs.

16. Black Box Testing Three ways of selecting test cases: –Paths through the specification e.g. choose test cases that cover each part of the preconditions and postconditions –Boundary conditions choose test cases that are at or close to boundaries for ranges of inputs –Off-nominal cases choose test cases that try out every type of invalid input (the program should degrade gracefully, without loss of data)

17. Equivalence Partitioning Suppose system asks for “a number between 100 and 999 inclusive”. This gives three equivalence classes of input: –less than 100 –100 to 999 –greater than 999 We thus test the system against characteristic values from each equivalence class. Example: 50 (invalid), 500 (valid), 1500 (invalid).

18. Boundary Analysis Arises from the observation that most programs fail at input boundaries. Suppose system asks for “a number between 100 and 999 inclusive”. The boundaries are 100 and 999. We therefore test for values: 99 100 101998 999 1000 lower boundaryupper boundary

19. White (Clear) Box Testing In white box testing, we use knowledge of the internal structure to guide development of tests. The ideal: examine every possible run of a system. –Not possible in practice! Instead: aim to test every statement at least once

20. White Box Testing Examine the code and test all paths …because black box testing can never guarantee we exercised all the code Path completeness: –A test set is path complete if each path through the code is exercised by at least one case in the test set

21. White Box Testing - Example if ($signal > 5) { echo “hello”; } else { echo ‘‘goodbye’’; } There are two possible paths through this code, corresponding to $signal > 5 and $signal <= 5. Aim to execute each one.

22. White Box Testing - Example 2 Consider an extremely simplified version of the code to authorize withdrawals from an ATM: $balance = get_account_balance($user); if ($amount_requested <= 300) { if ($amount_requested < $balance) { dispense_cash($amount_requested); } else { echo “You don’t have enough to withdraw $amount_requested”; } } else { echo “You can not withdraw more than $300”; } Mini-exercise: Document the white-box tests you need to create to test this code snippet.

23. Weaknesses of Path Completeness Path completeness is usually infeasible –e.g. –there are 2100 paths through this program segment (!) … and even if you test every path, it doesn’t mean your program is correct for ($j=0, $i=0; $i<100; $i++) if ($a[i]==True) $j=$j+1; for ($j=0, $i=0; $i<100; $i++) if ($a[i]==True) $j=$j+1;

24. Loop Testing Another kind of boundary analysis: 1.skip the loop entirely 2.only one iteration through the loop 3.two iterations through the loop 4.m iterations through the loop (m < n) 5.(n-1), n, and (n+1) iterations where n is the maximum number of iterations through the loop

25. Test Planning

26. Testing must be taken seriously, and rigorous test plans and test scripts developed. These are generated from requirements analysis document (for black box) and program code (for white box). Distinguish between: 1.unit tests; 2.integration tests; 3.system tests.

27. Alpha and Beta Testing In-house testing is usually called alpha testing. For software products, there is usually an additional stage of testing, called beta testing. Involves distributing tested code to “beta test sites” (usually prospective/friendly customers) for evaluation and use. Typically involves a formal procedure for reporting bugs.

28. Review: Triangle Classification Question: How many test cases are enough? Answer: A lot! expected cases (one for each type of triangle): (3,4,5), (4,4,5), (5,5,5) boundary cases (only just not a triangle): (1,2,3) off-nominal cases (not valid triangle): (4,5,100) vary the order of inputs for expected cases: (4,5,4), (5,4,4), … vary the order of inputs for the boundary case: (1,3,2), (2,1,3), (2,3,1), (3,2,1), (3,1,2) vary the order of inputs for the off-nominal case: (100,4,5), (4,100,5), … choose two equal parameters for the off-nominal case: (100,4,4), … bad input: (2, 3, q) vary the order of bad inputs: …

29. Documenting Test Cases Describe how to test a system/module/function Description must identify –short description (optional) –system state before executing the test –function to be tested –input (parameter) values for the test –expected outcome of the test

30. Test Automation Testing is time consuming and repetitive Software testing has to be repeated after every change (regression testing) Write test drivers that can run automatically and produce a test report

31. Exercise: Create a Test Plan What are you going to test? –functions, features, subsystems, the entire system What approach are you going to use? –white box, black box, in-house, outsourced, inspections, walkthroughs, etc. –what variant of each will you employ? When will you test? –after a new module is added? after a change is made? nightly? …. –what is your testing schedule?

32. Exercise: Create a Test Plan - 2 What pass/fail criteria will you use? –Provide some sample test cases. How will you report bugs? –what form/system will you use? –how will you track them and resolve them? What are the team roles/responsibilities? –who will do what?