1. Exceptions and Testing

2. Review Problems: IntMap Specification The Overview: /** An IntMap is a mapping from integers to integers. * It implements a subset of the functionality of Map. * All operations are exactly as specified in the documentation * for Map. * * IntMap can be thought of as a set of key-value pairs: * * @specfield pairs = {,,,... } */ Fall 15 CSCI 2600, A Milanova (spec by Michael Ernst) 2

3. Review Problems: IntMap Specification interface IntMap { /** Associates specified value with specified key in pairs. */ bool put(int key, int value); /** Removes the mapping for key from pairs if it is present. */ void remove(int key); /** Returns true if pairs contains a mapping for the specified key. */ bool containsKey(int key); /** Returns the value to which specified key is mapped, or 0 if this map contains no mapping for the key. */ int get(int key); } 3 Fall 15 CSCI 2600, A Milanova (spec by Michael Ernst)

4. Review Problems: IntStack Specification /** * An IntStack represents a stack of ints. * It implements a subset of the functionality of Stack. * All operations are exactly as specified in the documentation * for Stack. * * IntStack can be thought of as an ordered list of ints: * * @specfield stack = [a_0, a_1, a_2,..., a_k] */ 4 Fall 15 CSCI 2600, A Milanova (spec by Michael Ernst)

5. Review Problems: IntStack Specification interface IntStack { /** Pushes an item onto the top of this stack. * If stack_pre = [a_0, a_1, a_2,..., a_(k-1), a_k] * then stack_post = [a_0, a_1, a_2,..., a_(k-1), a_k, val ]. */ void push(int val); /** * Removes the int at the top of this stack and returns that int. * If stack_pre = [a_0, a_1, a_2,..., a_(k-1), a_k] * then stack_post = [a_0, a_1, a_2,..., a_(k-1)] * and the return value is a_k. */ int pop(); } 5

6. Review Problems: Rep Invariants and Abstraction Functions Willy Wazoo wants to write an IntMap but only knows how to use an IntStack ! So he starts like this before he gets stuck class WillysIntMap implements IntMap { private IntStack theRep; … Help Willy write the rep invariant and abstraction function Fall 15 CSCI 2600, A Milanova (problem due to Mike Ernst) 6

7. Review Problems Help Willy implement an IntStack with an IntMap class WillysIntStack implements IntStack { private IntMap theRep; int size; … Write a rep invariant and abstraction function Fall 15 CSCI 2600, A Milanova 7

8. Review Problem: Willy’s IntStack class IntStack { // Rep invariant: |theRep| = size // and theRep.keySet = {i | 1 ≤ i ≤ size} private IntMap theRep = new IntMap(); private int size = 0; public void push(int val) { size = size+1; theRep.put(size,val); } public int pop() { int val = theRep.get(size); theRep.remove(size); size = size–1; return val; } 8

9. Review Problem: Willy’s IntStack Base case Prove rep invariant holds on exit of constructor Inductive step Prove that if rep invariant holds on entry of method, it holds on exit of method push Pop For brevity, ignore popping an empty stack Fall 15 CSCI 2600, A Milanova 9

10. Practice Defensive Programming Check Precondition Postcondition Rep invariant Other properties we know must hold Check statically via reasoning “Statically” means before execution Works in simpler cases (the examples we saw), can be difficult in general Motivates us to simplify and/or decompose our code! 10

11. Practice Defensive Programming Check dynamically via assertions What do we mean by “dynamically”? At run time Assertions, supported by Java since 1.4 assert index >= 0; assert coeffs.length-1 == degree : “Bad rep” assert coeffs[degree] != 0 : “Bad rep” Write assertions, as you write code Aside: not to be confused with JUnit method such as assertEquals! 11 Fall 15 CSCI 2600, A Milanova

12. Assertions java runs with assertions disabled (default) java –ea runs Java with assertions enabled Always enable assertions during development. Turn off in rare circumstances assert (index >= 0) && (index < names.length); Fall 15 CSCI 2600, A Milanova 12 If assertion fails, program exits: Exception in thread "main" java.lang.AssertionError at Main.main(Main.java:34)

13. When NOT to Use Assertions Useless: x = y+1; assert x == y+1; When there are side effects assert list.remove(x); // Better: boolean found = list.remove(x); assert found; How can you test at runtime whether assertions are enabled? 13

14. Failure Some causes of failure 1. Misuse of your code Precondition violation 2. Errors in your code Bugs, rep exposure, many more 3. Unpredictable external problems Out of memory Missing file Memory corruption Fall 15 CSCI 2600, A Milanova 14 Which one is it: A)Failure of a subcomponent B) Division by zero

15. What to Do When Something Goes Wrong? Fail friendly, fail early to prevent harm Goal 1: Give information To the programmer, to the client code Goal 2: Prevent harm Abort: inform a human, cleanup, log error, etc. Retry: problem might be temporary Skip subcomputation: permit rest of program to continue Fix the problem (usually infeasible) Fall 15 CSCI 2600, A Milanova 15

16. Preconditions vs. Exceptions A precondition prohibits misuse of your code Adding a preconditions weakens the spec A precondition ducks the problem Behavior of your code when precondition is violated is unspecified! Does not help clients violating precondition of your code Removing the precondition requires specifying the behavior. Strengthens the spec Example: specify that an exception is thrown Fall 15 CSCI 2600, A Milanova 16

17. Which One Is Better? Choice 1: // modifies: this // effects: removes element at index from this // throws: IndexOutOfBoundsException if index < 0 || // index >= this.size public void remove(int index) { if (index >= size() || index < 0) throw new IndexOutOfBoundsException(“Info…”); else // remove element at index from collection } Choice 2: // requires: 0 <= index < this.size // modifies: this // effects: removes element at index from this public void remove(int index) { // no check, remove element at index } 17

18. Preconditions vs. Exceptions In certain cases, a precondition is the right choice When checking would be expensive. E.g., array is sorted In private methods, usually used in local context Whenever possible, remove preconditions from public methods and specify behavior Usually, this entails throwing an Exception Stronger spec, easier to use by client Fall 15 CSCI 2600, A Milanova 18

19. Square Root, With Precondition and Assertions // requires: x >= 0 // returns: approximation to square root of x public double sqrt(double x) { assert x >= 0 : “Input must be >=0”; double result; … // compute result assert(Math.abs(result*result – x) <.0001); return result; } Fall 15 CSCI 2600, A Milanova 19

20. Better: Square root, Specified for All Inputs // throws: IllegalArgumentException if x < 0 // returns: approximation to square root of x public double sqrt(double x) throws IllegalArgumentException { double result; if (x < 0) throw new IllegalArgumentException(“…”); … // compute result return result; } Fall 15 CSCI 2600, A Milanova 20

21. Better: Square root, Specified for All Inputs Client code: try { y = sqrt(-1); } catch (IllegalArgumentException e) { e.printStackTrace(); // or take same other } Exception is handled by catch block associated with nearest dynamically enclosing try Top-level handler: print stack trace, terminate program Fall 15 CSCI 2600, A Milanova 21

22. Throwing and Catching Java maintains a call stack of methods that are currently executing When an exception is thrown, control transfers to the nearest method with a matching catch block If none found, top-level handler Exceptions allow non-local error handling A method far down the call stack can handle a deep error! 22 main readFile readLine readChar decodeChar...... Fall 15 CSCI 2600, A Milanova

23. The finally Block finally is always executed No matter whether exception is thrown or not Useful for clean-up code FileWriter out = null; try { out = new FileWriter(…); … write to out; may throw IOException } finally { if (out != null) { out.close(); } Fall 15 CSCI 2600, A Milanova 23

24. Propagating an Exception up the Call Chain // throws: IllegalArgumentException if no real // solution exists // returns: x such that ax^2 + bx + c = 0 double solveQuad(double a, double b, double c) throws IllegalArgumentException { … // exception thrown by sqrt is declared, // no need to catch it here return (-b + sqrt(b*b – 4*a*c))/(2*a); } Fall 15 CSCI 2600, A Milanova 24

25. Informing the Client of a Problem Special value null – Map.get(x) -1 – List.indexOf(x) NaN – sqrt of negative number Problems with using special value Hard to distinguish from real values Error-prone: programmer forgets to check result? The value is illegal and will cause problems later Ugly Better solution: exceptions 25

26. Two Distinct Uses of Exceptions (External) failures (e.g., file not found) Unexpected by your code Usually unrecoverable. If condition is left unchecked, exception propagates up the stack Special results Expected by your code Unknowable for the client of your code Always check and handle locally. Take special action and continue computing Fall 15 CSCI 2600, A Milanova 26

27. Java Exceptions: Checked vs. Unchecked Exceptions Checked exceptions. For special results Library: must declare in signature Client: must either catch or declare in signature It is guaranteed there is a dynamically enclosing catch Unchecked exceptions. For failures Library: no need to declare Client: no need to catch RuntimeException and Error Fall 15 CSCI 2600, A Milanova 27 Throwable Exception Error Checked Exception Runtime Exception

28. Java Exception Hierarchy (Part of) Fall 15 CSCI 2600, A Milanova 28 Throwable ClassNotFound Exception DataFormat Exception IOExceptionRuntimeException … FileNotFound Exception MalformedURL Exception SocketException ArithmeticExceptionClassCastException IndexOutOfBounds Exception …

29. Don’t Ignore Exceptions An empty catch block is poor style! Often done to hide an error or get to compile try { readFile(filename); } catch (IOException e) {} // do nothing on error At a minimum, print the exception } catch (IOException e) { e.printStackTrace(); } Fall 15 CSCI 2600, A Milanova 29

30. Exceptions, review Use an exception when Checking the condition is feasible Used in a broad or unpredictable context Use a precondition when Checking would be prohibitive E.g., requiring that a list is sorted Used in a narrow context in which calls can be checked Fall 15 CSCI 2600, A Milanova 30

31. Exceptions, review Avoid preconditions because Caller may violate precondition Program can fail in an uninformative or dangerous way Want program to fail as early as possible Use checked exceptions most of the time Handle exceptions sooner than later Fall 15 CSCI 2600, A Milanova 31

32. Outline Testing Introduction Strategies for choosing tests suites Black-box testing White-box testing Fall 15 CSCI 2600, A Milanova 32

33. What is Testing? Testing: the process of executing software with the intent of finding errors Good testing: a high probability of finding yet- undiscovered errors Successful testing: discovers unknown errors “Program testing can be used to show the presence of bugs, but never to show their absence.” Edsger Dijkstra 1970 Fall 15 CSCI 2600, A Milanova 33

34. Quality Assurance (QA) The process of uncovering problems and improving the quality of software. Testing is the major part of QA QA is testing plus other activities: Static analysis (finding bugs without execution) Proofs of correctness (theorems) Code reviews (people reading each other’s code) Software process (development methodology) No single activity or approach can guarantee software quality 34

35. Famous Software Bugs Ariane 5 rocket’s first launch in 1996 The rocket exploded 37 seconds after launch Reason: a bug in control software Cost: over $1 billion Therac-25 radiation therapy machine Excessive radiation killed patients Reason: software bug linked to a race condition, missed during testing Fall 15 CSCI 2600, A Milanova 35

36. Famous Software Bugs Mars Polar Lander Legs deployed after sensor falsely indicated craft had touched down 130 feet above surface Reason: one bad line of software Cost: $110 million And many more… Northeast blackout (2003) Toyota Prius breaks and engine stalling (2005) And many many more… Fall 15 CSCI 2600, A Milanova 36

37. Cost to Society (Source: NIST Planning Report 2002) Inadequate testing infrastructure costs the US $22-60 billion annually Testing accounts for 50% of software development cost Program understanding and debugging accounts for up to 70% of time to ship a software product Maintenance (bug fixes and upgrades) accounts for up to 95% of total software cost Improvement in testing infrastructure can save one third of the cost 37

38. Scope (Phases) of Testing Unit testing Does each module do what it is supposed to do? Integration testing Do the parts, when put together, produce the right result? System testing Does program satisfy functional requirements? Does it work within overall system? Behavior under increased loads, failure behavior, etc. Fall 15 CSCI 2600, A Milanova 38

39. Unit Testing Our focus will be on unit testing Tests a single unit in isolation from all others In object-oriented programming, unit testing mostly means class testing Tests a single class in isolation from others Fall 15 CSCI 2600, A Milanova 39

40. Why Is Testing So Hard? // requires: 1 <= x,y,z <= 10000 // returns: computes some f(x,y,z) int proc(int x, int y, int z) Exhaustive testing would require 1 trillion runs! And this is a trivially small problem The key problem: choosing set of inputs (i.e., test suite) Small enough to finish quickly Large enough to validate program Fall 15 CSCI 2600, A Milanova (based on slide by Michael Ernst) 40

41. sqrt Example // throws: IllegalArgumentException if x < 0 // returns: approximation to square root of x public double sqrt(double x) What are some values of x worth trying? x < 0 (exception thrown) x >= 0 (returns normally) around 0 (boundary conditions) Perfect squares, non-perfect squares x x in this case), x = 1, x > 1 Fall 15 CSCI 2600, A Milanova (based on slide by Michael Ernst) 41

42. Outline Testing Introduction Strategies for choosing tests suites Black box testing White box testing Catch up: exceptions Fall 15 CSCI 2600, A Milanova 42

43. Testing Strategies Test case: specifies Inputs + pre-test state of the software Expected result (outputs and post-test state) Black box testing: We ignore the code of the program. We look at the specification (roughly, given some input, was the produced output correct according to the spec?) Choose inputs without looking at the code White box (clear box, glass box) testing: We use knowledge of the code of the program (roughly, we write tests to “cover” internal paths) Choose inputs with knowledge of implementation Fall 15 CSCI 2600, A Milanova 43

44. Black Box Testing Advantages Robust with respect to changes in implementation (independent of implementation) Test data need not be changed when code is changed Allows for independent testers Testers need not be familiar with implementation Tests can be developed before code based on specifications. 44

45. Black Box Testing Heuristic Choose test inputs based on paths in specification // returns: a if a > b // b if b > a // a if a = b int max(int a, int b) 3 paths, 3 test cases: (4,3) => 4 (input along path a > b) (3,4) => 4 (input along path b > a) (3,3) => 3 (input along path a = b) Fall 15 CSCI 2600, A Milanova 45

46. Black Box Testing Heuristic Choose test inputs based on paths in specification // returns: index of first occurrence of value in a // or -1 if value does not occur in a int find(int[] a, int value) What are good test cases? ([4,3,5,6], 5) => 2 ([4,3,5,6], 7) => -1 ([4,5,3,5], 5) => 1 Fall 15 CSCI 2600, A Milanova 46

47. sqrt Example // throws: IllegalArgumentException if x < 0 // returns: approximation to square root of x public double sqrt(double x) What are some values of x worth trying? We used this heuristic in sqrt example. It tells us to try a value of x = 0 (returns normally) are worth trying Fall 15 CSCI 2600, A Milanova (based on slide by Michael Ernst) 47

48. Black Box Heuristics “Paths in specification” heuristic is a form of equivalence partitioning Equivalence partitioning partitions input and output domains into equivalence classes Intuition: values from different classes drive program through different paths Intuition: values from the same equivalence class drive program through “same path”, program will likely behave “equivalently” 48

49. Black Box Heuristics Choose test inputs from each equiv. class // returns: 0 <= result <= 10 // throws: SomeException if arg 10 int proc(int arg) There are three equivalence classes: “ arg < 0”, “0 <= arg <= 10” and “10 < arg ”. We write tests with values of arg from each class Stronger vs. weaker spec. What if the spec said // requires: 0 <= arg <= 10? 49

50. Equivalence Partitioning Examples of equivalence classes Valid input x in interval [a..b]: this defines three classes “ x <a”, “a<= x <=b”, “b< x ” Input x is boolean: classes “true” and “false” Choosing test values Choose a typical value in the middle of the “main” class (the one that represents valid input) Also choose values at the boundaries of all classes: e.g., use a-1,a, a+1, b-1,b,b+1 50

51. Black Box Testing Heuristic: Boundary Value Analysis Idea: choose test inputs at the edges of the equivalence classes Why? Off-by-one bugs, forgot to handle empty container, overflow errors in arithmetic Cases at the edges of the “main” class have high probability of revealing these common errors Complements equivalence partitioning 51

52. Equivalence Partitioning and Boundary Values Suppose our specification says that valid input is an array of 4 to 24 numbers, and each number is a 3-digit positive integer One dimension: partition size of array Classes are “n<4”, “4<=n<=24”, “24<n” Chosen values: 3,4,5, 14, 23,24,25 Another dimension: partition integer values Classes are “x<100”, “100<=x<=999”, “999<x” Chosen values: 99,100,101, 500, 998,999,1000 52

53. Equivalence Partitioning and Boundary Values Equivalence partitioning and boundary value analysis apply to output domain as well Suppose that the spec says “the output is an array of 3 to 6 numbers, each one an integer in the range 1000 - 2500” Test with inputs that produce (for example): 3 outputs with value 1000 3 outputs with value 2500 6 outputs with value 1000 6 outputs with value 2500 More tests… Fall 15 CSCI 2600, A Milanova 53

54. Equivalence Partitioning and Boundary Values // returns: index of first occurrence of value in a, or -1 if value does not occur in a int find(int[] a, int value) What is a good partition of the input domain? One dimension: size of the array People often make errors for arrays of size 1, we decide to create a separate equivalence class Classes are “empty array”, “array with one element”, “array with many elements” Fall 15 CSCI 2600, A Milanova 54

55. Equivalence Partitioning and Boundary Values We can also partition the output domain: the location of the value Four classes: “first element”, “last element”, “middle element”, “not found” ArrayValueOutput Empty 5 -1 [7] 7 0 [7] 2 -1 [1,6,4,7,2] 1 0 (boundary, start) [1,6,4,7,2] 4 2 (mid array) [1,6,4,7,2] 2 4 (boundary, end) [1,6,4,7,2] 3 -1 Fall 15 CSCI 2600, A Milanova 55

56. Other Boundary Cases Arithmetic Smallest/largest values Zero Objects Null Circular list Same object passed to multiple arguments (aliasing) Fall 15 CSCI 2600, A Milanova (based on slide by Michael Ernst) 56

57. Boundary Value Analysis: Arithmetic Overflow // returns: |x| public int abs(int x) What are some values worth trying? Equivalence classes are x = 0 x = -1, x = 1, x = 0 (boundary condition) How about x = Integer.MIN_VALUE? // this is -2147483648 = -2 31 // System.out.println(Math.abs(x) < 0) prints true! Fall 15 CSCI 2600, A Milanova (based on slide by Mike Ernst) 57

58. Boundary Value Analysis: Aliasing // modifies: src, dest // effects: removes all elements of src and appends them in reverse order to the end of dest void appendList(List src, List dst) { while (src.size() > 0) { Integer elt = src.remove(src.size()- 1); dest.add(elt); } What happens if we run appendList(list,list) ? Aliasing. 58

59. Summary So Far Testing is hard. We cannot run all inputs Key problem: choose test suites such that Small enough to finish in reasonable time Large enough to validate the program (reveal bugs, or build confidence in absence of bugs) All we have is heuristics! We saw black box testing heuristics: run paths in spec, partition input/output into equivalence classes, run with input values at boundaries of these classes There are also white box testing heuristics 59

60. White Box Testing Ensure test suite covers (covers means executes) all of the program Measure quality of test suite with % coverage Assumption: high coverage implies few errors in program Focus: features not described in specification Control-flow details Performance optimizations Alternate algorithms (paths) for different cases Fall 15 CSCI 2600, A Milanova (based on slide by Michael Ernst) 60

61. White Box Testing: Control-flow-based Testing Control-flow-based white box testing: Extract a control flow graph (CFG) Test suite must cover (execute) certain elements of this control-flow graph graph Idea: Define a coverage target and ensure test suite covers target Targets: nodes, branch edges, paths Coverage target approximates “all of the program” Fall 15 CSCI 2600, A Milanova 61

62. Aside: Control-flow Graph (CFG) Assignment x=y+z => node in CFG: If-then-else if (b) S1 else S2 => Fall 15 CSCI 2600, A Milanova 62 x=y+z CFG for S1 b b TrueFalse CFG for S2

63. Aside: Control-flow Graph (CFG) Loop while (b) S => Fall 15 CSCI 2600, A Milanova 63 CFG for S b b TrueFalse

64. Aside: Control Flow Graph (CFG) Draw the CFG for the code below: 64 1 s:= 0; 2 x:= 0; 3 while (x<y) { 4 x:=x+3; 5 y:=y+2; 6 if (x+y<10) 7 s:=s+x+y; else 8 s:=s+x-y; }

65. Statement Coverage Traditional target: statement coverage. Write test suite that covers all statements, or in other words, all nodes in the CFG Motivation: code that has never been executed during testing may contain errors Often this is the “low-probability” code Fall 15 CSCI 2600, A Milanova 65

66. Suppose that we write and execute two test cases Test case #1: follows path 1-2-exit (e.g., we never take the loop) Test case #2: 1-2-3-4-5-7- 8-2-3-4-5-7-8-2-exit (loop twice, and both times take the true branch) Problems? 1 2 3 5 6 7 8 TF Example 4 F T

67. Branch Coverage Target: write test cases that cover all branch edges at predicate nodes True and false branch edges of each if-then-else The two branch edges corresponding to the condition of a loop All alternatives in a SWITCH statement In modern languages, branch coverage implies statement coverage Fall 15 CSCI 2600, A Milanova 67

68. Branch Coverage Motivation for branch coverage: experience shows that many errors occur in “decision making” (i.e., branching). Plus, it implies statement coverage Statement coverage does not imply branch coverage I.e., a suite that achieves 100% statement coverage does not necessarily achieve 100% branch coverage Can you think of an example? 68

69. Example static int min(int a, int b) { int r = b; if (a <= b) r = a; return r; } Let’s test with min(1,2) What is the statement coverage? What is the branch coverage? Fall 15 CSCI 2600, A Milanova 69

70. We need to cover the red branch edges Test case #1: follows path 1-2-exit Test case #2: 1-2-3-4-5-7- 8-2-3-4-5-7-8-2-exit What is % branch coverage? 1 3 5 6 7 8 TF Example T F 2 4

71. Code Coverage in Eclipse Fall 15 CSCI 2600, A Milanova 71

72. Rules of Testing First rule of testing: Do it early and do it often Best to catch bugs soon, before they hide Automate the process Regression testing will save time Second rule of testing: Be systematic Writing tests is a good way to understand the spec Specs can be buggy too! When you find a bug, write a test first, then fix Fall 15 CSCI 2600, A Milanova 72