1. CEN 4072 Software Testing Chapter 1: How Failures Come To Be

2. Golden Rules Software Testing begins with careful design. Faulty software can cost stakeholders more than millions, or even risk lives. Testing is an important step in the development cycle. Software that is poorly designed or put together is difficult to test. Software that is difficult to test or cannot be tested is useless.

3. Failures & Damage Caused Knight Capital Group (2012) – Faulty stock trading algorithm caused company to lose $440M in 30 minutes. NASA Mars Climate Orbiter (1998) – Unit conversion failure, caused $125M spacecraft to fail to stabilize and lost in space. Patriot Missile Error (1991) – Software’s accuracy to detect enemy attacks became worse the longer software ran, eventually failing to detect an attack which killed 28 American soldiers.

4. Debugging Facts Software bugs are costing the U.S. economy an estimated $59.5 Billion per year [RTI (2002)]. Developers estimate that improvements in debugging & testing could reduce the cost of software by $22.5 Billion. [RTI (2002)] 50% of labor to develop a working program is spent on testing & debugging activities. [Beizer (1990)] Validation Activities (Debugging, Testing & Verification) can range from 50%-75% of development costs. [Hailpern & Santhanam (2002)].

5. Defects Piece of code that can cause program to enter an infected state. Can be a miscalculation, a piece of code that did not consider future or possible scenarios, etc. Not necessarily a program failure.

6. From Defect(s) to Failure Sequence of events cause defect to become failure. Programmer creates defect – a piece of code is created that did not predict future requirements/scenarios. The defect causes an infection – the program runs, and program enters a state different from what is intended. The infection propagates – Functions dependent on the results of the now infected state are likely to also become infected (unless overwritten, hidden or corrected). The infection causes a failure – the program produces a visible error, or acts incorrectly, and is thus, now a failure.

7. TRAFFIC Principal Track the problem. Reproduce the failure. Automate & simplify the test case. Find possible infection origins. Focus on likely origins. Isolate the infection chain. Correct the defect.

8. Searching in Time & Space We must find out which part of the state is infected (search in space). We also must find when the state becomes infected (search in time). This can be quite difficult, imagine trying to search in time & space for a program with 1000’s of variables & references between them!

9. Sample Program - Sorting int main(int argc, char *argv[]) { int *a; int i; a = (int *)malloc((argc - 1) * sizeof(int)); for (i = 0; i < argc - 1; i++) a[i] = atoi(argv[i + 1]); shell_sort(a, argc); printf("Output: "); for (i = 0; i < argc - 1; i++) printf("%d ", a[i]); printf("\n"); free(a); return 0; }

10. Sample Program (Cont’d) static void shell_sort(int a[], int size) { int i, j; int h = 1; do { h = h * 3 + 1; } while (h <= size); do { h /= 3; for (i = h; i < size; i++) { int v = a[i]; for (j = i; j >= h && a[j - h] > v; j -= h) a[j] = a[j - h]; if (i != j) a[j] = v; } } while (h != 1); } There is a defect somewhere in this program, find it using the principles described below.

11. Sample Exercise When ran, notice the input & output: $./sample 11 14 Output: 0 11 Expected Output: 11 14 What’s going on here? Let’s Debug!

12. Sample Exercise In this sample, the first three principles are straight forward. Track – We have observed the problem, an unknown 0 appears on each run. Reproduce – The sample program produces incorrect data with any run. Automate – The sample program is not very complex, and does not need automation.

13. Find Origins We observe that the program has two methods – main & shell_sort. main – Allocates variables, reads input, sorts input using shell_sort then prints result. shell_sort – Sorts inputted array. When the output is printed, we notice that the value of a[0] is incorrectly set to 0. Was the array incorrectly initialized or did something happen in shell_sort?

14. Observing Run/Search in Time By printing to console using fprintf, we can note the values of the array at each step of the program../sample 11 14 Input: 11 14 Shell Sort Input: 11 14 0 Shell Sort Size: 3 Shell Sort Sorted: 0 11 14 Output: 0 11 Shell Sort is called with a 3 element array?

15. Focus on Likely Origins We hypothesize that shell_sort runs correctly. The infection must be happening before shell_sort runs. We notice that the size passed in shell sort is not equal to that of the array. This is likely related to our defect. Looking through shell sort, we notice that it accesses no local variables. We also notice the 3 element array outputted by shell_short is correctly sorted.

16. Finding Causes Array is correctly initialized with 2 elements at the start. However, in main, before shell_sort is called, our array is of size 2 while the passed value for size in shell_sort is 3. shell_sort takes the 3 element array it gets [11, 14, 0] and in fact correct sorts it as [0, 11, 14]. The input of size 3 causes shell sort to access uninitialized space in a[], a[2] which happens to hold a value of 0. During the sort, a[2] is swapped with a[0], setting a[0] = 0, thus infecting the state. We speculate this incorrectly passed size is to blame!

17. Correct Issue We used argc-1 as the size of our array previously to initialize it, however argc is passed into shell_sort, giving it the incorrect size. shell_sort(a, argc); We fix this by replacing this line with: shell_sort(a, argc-1); We recompile the program & run it with the same input: $./sample 11 14 Output: 11 14 The failure no longer occurs, issue is resolved!

18. Efficiency of Testing We were able to debug this sample manually. All debugging problems could be solved manually. However, manually debugging may take large amounts of time. We may miss defects. Automated Debugging techniques make the task of testing quicker & easier.

19. Automated Debugging: Program Slices Separate part of the program that is relevant to the failure. Deduce from code design what could or could not possibly in different parts of the program. Slice the program into parts relevant to the failure.

20. Automated Debugging: Observing the State Stop the program to observe the entire state, line by line. Allows programmer to view entire state at any given time. This allows us to view values of variables at any given time. No need to modify or recompile the program each time.

21. Automated Debugging: Watching State Watch smaller part of a state to find changes during execution. More focused than observing the state. Watch a particular variable to find the precise line of code where its changed from sane to infected value.

22. Automated Debugging: Assertions Specify expected values and check for them. Especially before/after function call, we assert that the current value of a particular variable is equal to the expected value. If all assertions hold true, we can consider a chunk of code sane. Thus allows us to focus on other potential infections. // Return Sum of squares from 1 to n int sumOfSquares(int n) { ASSERT(n == 3); // Do Math // … ASSERT(sum == 14); return sum; } In Example: We assert input is 3 as expected, and that output is 14 as expected.

23. Automated Debugging: Anomalies Generally, we assume a program works fine most of the time. If a program fails, we observe key differences between the passing runs & failing run. These difference are called anomalies. // Raises n to k’th power. int pow(int n, int k) { int result = n; for (int i = 0; i < k; i++) result *= n; return result; } This method generally works fine. It works for 2^4, 5^2 and any positive integer for k. However, what happens if we use a negative integer or zero for k?

24. Automated Debugging: Cause-Effect Chains Identify which variable or assignment in each state transition contributed to failure. Although causes are not necessarily defects/failures, they can help narrow down which variables may be related to the failure.

25. Automated Debugging: Simplified Input Simplify the input so that each part contributes to the failure. Using a technique called delta debugging, which automatically narrows down what’s different between passing & failing run. For simplified input, delta debugging is used to find only inputs that contributed to the failure. For Example: Suppose we have some program that transitions to infected each time when a sequence of even numbers in inputted. Manually, we may be have tested this using a more complex input like: 0 1 2 4 5 7 -1 2 4 6 8 1 1 10 Simplified Input may take this & a passing set of input & create: 0 2 4 6 8

26. Recap Importance of Software Design & Testing Cost of Software Failures Defects, Infections & Failures TRAFFIC Principle Manual Debugging Automated Debugging

27. Bibliography Zeller, A. (2009). Why Programs Fail: A Guide to Systematic Debugging(2 nd ed.). Burlington, MA: Morgan Kaufmann. Harley, N. (2014, May 29). 10 of the most costly software errors in history. Retrieved September 1, 2015.