1. October 21, 2010COMS W41561 COMS W4156: Advanced Software Engineering Prof. Gail Kaiser Kaiser+4156@cs.columbia.edu http://bank.cs.columbia.edu/classes/cs4156/

2. October 21, 2010COMS W41562 Topics covered in this lecture Unit Testing –Black box –White box –Automating unit testing

3. October 21, 2010COMS W41563 Unit Testing

4. October 21, 2010COMS W41564 Unit Testing Overview Unit testing is testing some program unit in isolation from the rest of the system (which may not exist yet) Usually the programmer is responsible for testing a unit during its implementation (even though this violates the rule about a programmer not testing own software) Easier to debug when a test finds a bug (compared to full-system testing)

5. October 21, 2010COMS W41565 Unit Testing Strategies Black box testing, aka specification-based or partition testing White box testing, aka glass box, program- based or structural testing Normally perform both (not alternatives!)

6. October 21, 2010COMS W41566 Unit Testing: Black Box

7. October 21, 2010COMS W41567 Black Box Testing Test suite constructed by inspecting the specification (requirements, design, etc.), not the source code Tests unit against functional and, sometimes, extra-functional specifications (e.g., resource utilization, performance, security) Attempts to force behavior (outcome) that doesn't match specification Really “grey box”, since probably need to look at code to implement test harness

8. October 21, 2010COMS W41568

9. October 21, 2010COMS W41569 Test-Driven Development Applied in agile programming (and increasingly other software processes) Lay out criteria for passing test(s) before coding the unit Often reveals lack of clarity in specification Write small pieces of code that return true when the unit passes that test, and false when it doesn’t Effort goes into determining and expressing these two sets of conditions

10. October 21, 2010COMS W415610 Test-Driven Development Forces bottom-up design – considering how the code will be invoked (e.g., method signature) Determines when code-writing subtask successfully completed – write the simplest code possible to pass all the tests

11. October 21, 2010COMS W415611 Blackbox Approaches to Unit Testing Functional testing – exercise code with valid or nearly valid input for which the expected outcome is known (outcome includes global state, out parameters, exceptions raised, any externally visible results as well as return value) Exhaustive testing usually infeasible, so need way(s) to select test cases and determine when “done” testing

12. October 21, 2010COMS W415612 Blackbox Approaches to Unit Testing Choose test cases to attempt to find different flaws –Equivalence partitioning –Boundary value analysis

13. October 21, 2010COMS W415613 Equivalence Partitioning Equivalence class is a related set of valid or invalid values or states Only one or a few examples are selected to represent an entire equivalence class Assumes similar inputs will evoke similar responses Good for basic functionality testing

14. October 21, 2010COMS W415614 Equivalence Partitioning Divide input domain into equivalence classes Ideally, also divide outcome domain into equivalence classes –Need to determine inputs to cover each output equivalence class –Also attempt to cover classes of errors, exceptions and external state changes

15. October 21, 2010COMS W415615 Range equivalence classes 1.Arbitrary value below range 2.Arbitrary value within range 3.Arbitrary value above range

16. October 21, 2010COMS W415616 Example A student must be registered for at least 12 points to be considered full-time (but there are various “max” points depending on registration category) –Full-time: some number between 12 and “max” –Not full-time: some number less than 12 –Error cases: above “max”, negative number, otherwise invalid number (3.1415926…), non- number

17. October 21, 2010COMS W415617 Set equivalence classes Member of set Non-member of set

18. October 21, 2010COMS W415618 Example A student must have a valid CID –Valid CID: C00xxxxxxx that has been assigned to some student –Invalid CID: C00yyyyyyy that has never been assigned to any student –Error cases: C00zzz, ABCxxxxxxx, 57

19. October 21, 2010COMS W415619 More Partition Examples negative, positive, and zero numbers numeric values within size limits and beyond size limits strings within size limits, beyond size limits, and the empty string strings containing printable vs. non-printable characters combinations of input values that are consistent, and combinations that are not consistent

20. More Complex Partition Examples Table pre-sorted or not pre-sorted by key Table with and without “missing” values Table with and without “repeated” values Empty table, less than minimum size, within expected size range, beyond expected size range, humongous  Consider other data structures October 21, 2010COMS W415620

21. October 21, 2010COMS W415621 Also Consider External State input files that exist or not, are readable or not, are correct format or not output files that exist or not, are writeable or not, are correct format or not (if modifying pre- existing file) network sockets, database connections, GUI, …

22. October 21, 2010COMS W415622 Boundary Value Analysis Extends equivalence partitioning In practice, more (often subtle) errors found at or near equivalence class boundaries than in the middle of the classes Aka corner, edge, fencepost cases Divide input domain into equivalence classes Divide outcome domain into equivalence classes (and derive inputs to attempt to force those outcomes) Tends to catch initialization errors, bounds-checking and out- of-range problems

23. October 21, 2010COMS W415623 Range boundary values 1.Value immediately below range 2.First value of range 3.Second value of range 4.Next to last value of range 5.Last value of range 6.Value immediately above range 7.Minimum and maximum values of range's basic type (e.g., integer, double)

24. October 21, 2010COMS W415624 Example A student must be registered for at least 12 points to be considered full-time (but there are various “max” points depending on registration category) –Full-time: some number between 12 and “max” –Not full-time: some number less than 12 –Error cases: above “max”, negative number, otherwise invalid number (3.1415926…), non- number

25. October 21, 2010COMS W415625 Example A student must be registered for at least 12 points to be considered full-time – not full-time case 1.Value immediately below range: -1 2.First value of range: 0 3.Second value of range: 1 4.Next to last value of range: 10 5.Last value of range: 11 6.Value immediately above range: 12 7.Minimum and maximum values of range's basic type: MinInt, MaxInt

26. October 21, 2010COMS W415626 Example A student must be registered for at least 12 points to be considered full-time – full-time case 1.Value immediately below range: 11 2.First value of range: 12 3.Second value of range: 13 4.Next to last value of range: max-1 5.Last value of range: max 6.Value immediately above range: max+1 7.Minimum and maximum values of range's basic type: MinInt, MaxInt

27. October 21, 2010COMS W415627 Scalar set boundary values Member of set Values immediately above and below each member of set Minimum and maximum values of set element's basic type

28. October 21, 2010COMS W415628 Non-scalar set boundary values Member of set Null value (if not in set) Other non-member value(s) of correct type ("above" and "below" each set member, "minimum" and "maximum" of type, if such concepts are applicable) Value of incorrect type, structurally equivalent (if not detected by compiler) Value of incorrect type, not structurally equivalent (if not detected by compiler)

29. October 21, 2010COMS W415629 Example A student must have a valid CID –Valid CID: C00xxxxxxx that has been assigned to some student –Invalid CID: C00yyyyyyy that has never been assigned to some student –Error cases: C00zzz, ABCxxxxxxx, 57

30. October 21, 2010COMS W415630 Example A student must have a valid CID –Member of set: C00xxxxxxx –Null value (if not in set): [blank] –Other non-member value(s) of correct type: C00yyyyyyy, C01xxxxxxx, C00zzz –Value of incorrect type, structurally equivalent (if not detected by compiler) A12xxxxxxx –Value of incorrect type, not structurally equivalent (if not detected by compiler) 57 [as a number rather than an alphanumeric string]

31. October 21, 2010COMS W415631 More Boundary Case Examples minimum negative (and +1), maximum positive (and -1), and zero (and +/- 1) inputs or outputs input or output strings at size limits, one character above and below size limits, the empty string, and strings of one character empty input files, files with one bit, byte or binary word in them, files just below or at maximum file size

32. October 21, 2010COMS W415632 DVR Example Consider embedded software for a digitally controlled video recorder That has a function that sets the time to automatically record a program boolean set_timed_program( input Integer start_hour, input Integer start_min, input Integer end_hour, input Integer end_min, input Integer channel)

33. October 21, 2010COMS W415633 Example, cont. Start and end times expressed as an (hour, minute) pair, with hour expressed using 24hr time The channel ranges from 1 to 999 The function returns true if the program was successfully set, false if it was not set To record a program starting at 8PM, ending at 9:30PM, on channel 9, the function call would be: set_timed_program(20, 00, 21, 30, 9);

34. October 21, 2010COMS W415634 What are the equivalence classes and boundary values?

35. October 21, 2010COMS W415635 Unit Testing: White Box

36. October 21, 2010COMS W415636 Unit Testing Strategies Black box testing, aka specification-based or partition testing White box testing, aka glass box, program- based or structural testing Normally perform both (not alternatives!)

37. October 21, 2010COMS W415637 White Box Testing Test suite constructed by inspecting the program (code) Look at specification (requirements, design, etc.) only to determine what is an error Attempt to exercise all statements, all branches, or all paths (control flow and/or data flow within each method, state transition sequences across methods) Referred to as adequacy or coverage criteria Intuition: If you never tested that part of the code, or that path to executing the code, how can you have any reason to believe that it works?

38. October 21, 2010COMS W415638

39. October 21, 2010COMS W415639 Whitebox Approaches to Unit Testing 1.Execute all (reachable) statements 2.Execute all branches of logical decisions, including boundaries of loops 3.Execute all (feasible) control flow paths in combination 4.Execute all data flow paths (from each variable definition to all its uses)  Usually applied only to individual methods rather than larger units due to combinatorics, more practical to consider sequencing across methods within a class or component  May be combined with black box testing if tracked

40. October 21, 2010COMS W415640 Example Consider a function that: Takes as input a string assumed to be a URL and Checks to see if it contains any characters that are illegal Illegal URL characters are control characters (ascii 0-31, 127) and delimiter characters (">", "<", "#", "%", and " [the double quote character]) The function returns true if the URL is valid (does not contain an illegal character), and false if the URL is invalid (contains an illegal character)

41. October 21, 2010COMS W415641 is_illegal (input String url, output Boolean valid) begin Integer i; Char c; valid = TRUE; i = 1; while (i <= length(url) AND valid == TRUE) begin c = url[i]; if (c >= 0 AND c <= 32) then valid = false; else switch (c) case ">": valid = false; case "<": valid = false; case "#": valid = false; case "%": valid = false; case "\"": valid = false; end switch; end if; end while; end;

42. October 21, 2010COMS W415642 What are the statements, conditions, etc. that should be tested?

43. October 21, 2010COMS W415643 Automated Unit Testing

44. October 21, 2010COMS W415644 Automated Testing Testing by hand is tedious, slow, error-prone and not fun Computers are much less easily bored than people So write code to test your code

45. October 21, 2010COMS W415645 Automated Testing Write code to set up the unit, call its methods with test inputs, and compare the results to the known correct answers Once tests are written, they are easy to run, so you are much more likely to run them Every time you find a bug not caught by an existing test case, write a new unit test to catch it! JUnit, NUnit, various C++ unit testing, xUnit for various xJUnitNUnitvarious C++ unit testing

46. October 21, 2010COMS W415646 Mutation Testing Testing the tests Develop black box test suite Program executes test suite correctly Make small change in code Retain syntactic correctness but not semantics Run test suite and see if there are any changes in results Repeat

47. October 21, 2010COMS W415647 Mutations Change sense of a comparison operator Change an arithmetic operator Change a constant Change which variable or array index is used Delete a statement Delete/insert a return statement

48. October 21, 2010COMS W415648 is_illegal (input String url, output Boolean valid) begin Integer i; Char c; valid = TRUE; i = 1; while (i <= length(url) AND valid == TRUE) begin c = url[i]; if (c >= 0 AND c <= 32) then valid = false; else switch (c) case ">": valid = false; case "<": valid = false; case "#": valid = false; case "%": valid = false; case "\"": valid = false; end switch; end if; end while; end;

49. October 21, 2010COMS W415649 Intuition If a mutation doesn't matter, then something is wrong! –Either mutated code is unnecessary –Or there's a problem with the test suite itself Expensive: n 2 mutants for n statements, but automatable Often basis for validation suites

50. Upcoming Assignments October 21, 2010COMS W415650

51. Upcoming Assignments First Iteration Progress Report due Tuesday 26 October. First Iteration Progress Report First Iteration Demo Wednesday 3 November – Thursday 11 November. First Iteration Demo First Iteration Final Report due Friday 12 November. First Iteration Final Report October 21, 2010COMS W415651

52. COMS W4156: Advanced Software Engineering Prof. Gail Kaiser Kaiser+4156@cs.columbia.edu http://bank.cs.columbia.edu/classes/cs4156/ October 21, 2010COMS W415652