1 CEN 4072 Software Testing PPT12: Fixing the defect

2 From defect to failure How does a defect in a program become a failure? When a defect causes an infection (erroneous program state) that is visible to the user Not all defects lead to failures –Nevertheless a defect is an erroneous piece of code and should be fixed How to locate the defect that leads to a given failure? 1.Start with the infected value that defines the failure 2.Determine the possible origins of the infected value, following dependencies in the code 3.Observe each origin to determine whether it is infected or not –Restart at step 2 with the infected origin(s) The above process will loop until an infection is found that has all sane origins The code that produces this infection is the defect that caused the failure

3 Techniques for locating defects Infections If an assertion fails Dependences Identify what the value of a variable is dependent on –Example: average = sum / count The value of average is dependent on sum and count Causes Identify causes through experimentation –The likelihood of finding a defect near a cause is very high Anomalies Identify states that only occur in failing runs –Example: if function getSum() only run on failing runs the likelihood that the defect is closely related to this anomaly is high Code Smells Identify constructs in the code that are known to cause errors –Example: initialized variables These do not necessarily cause errors, but they do at a very high rate

4 Focusing on errors Reduce defect search time Focus search on origins that are more likely to cause errors Assertions Guarantee the sanity of data over time on any program run –If an assertion fails then an infection has been identified –Else anything covered by the assertion can be ignored in the search Because it is known to be in a sane state Anomalies Aspects of the program that correlate with failure –Example: a certain function only runs on failing runs –This correlation to failure means that it is wise to focus on these Causes Not only correlate to failure, but have been experimentally proven to cause the failure –These origins are even more likely than anomalies to indicate the defect

5 Focusing on errors Focus on Infections If an origin is known to be faulty then focus on it first –Check whether or not the infection causes the failure Focus on Causes If some form of experimentation has shown some state or input to be a failure cause –Check if the cause is infected Focus on Anomalies Any origin that is associated with an anomaly has a good chance to contain errors –Check if these origins are infected Focus on Code Smells If code smells have been identified and the origin is one of these code smells –Check if the code smell causes the infection and/or error Focus on Dependences If all the above fail to find the defect, then the defect must be found in some dependency –For all of the possible origins, check the backward slice for an infection Start with the origins nearest the failure

6 Techniques: dependences Starting from the failure, follow dependences to find possible origins of the defect

7 Techniques: observation Observe the state of an origin and decide whether or not it is infected

8 Techniques: assertion Use assertions to determine whether a given origin is sane or not Note that a single assertion can cover multiple origins as seen below

9 Techniques: anomaly Here there are two origins to choose from One of them is observed to be an anomaly so it is focused

10 Techniques: cause transition Here a cause transition is identified This is known because of the fact that the target is infected but the origin is sane

11 Techniques: defect found The defect has now been identified

12 Techniques: TRAFFIC strategy The techniques explained above follow the TRAFFIC strategy Specifically, the “focus on likely origins” step TRAFFIC strategy Track the problem Reproduce Automate Find origins Focus on likely origins Isolate Correct

13 Techniques: ordering The order in which to apply the aforementioned techniques is as follows 1.Focus on infections 2.Focus on causes 3.Focus on anomalies 4.Focus on code smells 5.Focus on dependences This ordering is chosen so that the techniques that are more likely to find a defect are applied before the techniques that are less likely to find a defect

14 Is the error a cause? Just because an error has been found does not mean that the error causes the failure When tracing an infection chain it must be shown at each step that 1.The origin is infected 2.The origin causes the infection chain Example: a = compute_value(); printf("a = %d\n", a); If the output of the program is always “a = 0” Then it may be assumed that the defect is related to the compute_value() function This can be shown to be false by manually setting a = 1 right before the print –If the program still outputs “a = 0” when a is known to not be 0 Then the problem must be with the print statement If this manual check had not occurred –Then the tester would search through origins where the cause does not exist Some error may be found in this search; however, this error is not the cause

15 Does making symptoms disappear fix the error? Simply making the symptoms of an error disappear does not mean that the error has been fixed Example: balance[account] = 0.0; for (int i = 0; i < n; i++) balance[account] += deposit[i] Assume that account 123 has an incorrect balance The following makes the symptom disappear but does not fix the true error if (account == 123) balance[123] += 45.67

16 Debugging into existence Changing the code to make it work without understanding the cause Example: static void shell_sort(int a[], int size) { int i, j; int h = 1; do { h = h * 3 + 1; } while (h <= size); do { h /= 3; for (i = h; i < size; i++) { int v = a[i]; for (j = i; j >= h && a[j - h] > v; j -= h) a[j] = a[j - h]; if (i != j) a[j] = v; } } while (h != 1); } Assume that the fix for this program was to change the highlighted line to instead be the following for (i = h; i < size - 1; i++) This change was made based off a guess that the loop bound was off by one. There was no testing to prove this to be the case. This is wrong because it is unknown whether or not the true error was fixed. This technique of debugging with guesses is dangerous because it can introduce new bugs into the program.

17 Devil’s guide to debugging Find the defect by guessing Scatter debugging statements throughout the program Try changing code until something works Do not back up old versions of the code Do not bother understanding what the program should do Do not waste time understand the problem Most problems are trivial anyways Use the most obvious fix Just fix what you see Example: x = compute(y) // compute(17) is wrong – fix it if (y == 17) x = x = Why bother tracing the error through compute()? Just fix it ! These are all techniques that should be fully avoided

18 Correcting the defect Before applying the correction to the code you must always first BACKUP THE CODE After correcting the code there are four problems that must be looked at 1.Does the failure no longer occur? –If the failure no longer occurs then the correction was successful –If the failure is still present then there are two possible reasons 1.The failure was caused by more than one defect and only one was fixed 2.Your understanding of the infection chain was wrong 2.Did the correction introduce new problems? –A few practices that are useful for this question 1.Have corrections peer reviewed 2.Have a regression test ready to detect unwanted changes

19 Correcting the defect 3.Was the same mistake made elsewhere? –Check for other defects that are caused by the same mistake Focus on –Code from the same developer –Code involving the same API 4.Did I commit the change? –Assign resolution to the problem (such as FIXED) –Integrate the fix into the production code

20 Workarounds Why might workarounds be needed? 1.Unable to change –The source code may not be available to be changed 2.Risks –The correction may induce huge risks Such as if the fix implies large changes throughout the system 3.Flaw –The problem may not be in the code but the overall design of the program In which case the system design must undergo a major revision Under what circumstances are workarounds allowed? If no better solution is known –Workarounds are meant to be temporary fixes –Be sure to keep the problem open so that a true fix may one day be found

21 Famous workarounds 1.Spam filters –Workaround for solving a flaw in the system Anyone can forge messages and conceal their true identity –Proper solution would involve reworking the entire system 2.Virus scanners –Privilege level flaw in operating systems By default a user has admin rights Means that any download can gain complete control of the machine –Proper solution would be to limit user rights Problem is that too many regular programs assume admin privileges 3.Date windowing –Inability of many legacy systems to deal with four-digit years –Resolves ambiguity of two-digit year by defining a 100-year window –Proper solution would require adapting legacy systems Too much cost and risk involved

22 Learning from mistakes Improve quality assurance –Introduce systematic processes to obtain program requirement and design –Check specifications early Find differences in what the developers think the program should do and what the clients want the program to do Resolving these issues early leads to requirements being more precise Improve test suite –If the test suite failed to find the defect Once the defect has been found the test suite should be extended to test for the new defect Setup assertions –Can narrow down the infection chain of a failure –Will catch similar infections in the future Improve the software process –Plan the software carefully before starting Improve analysis tools –Verify whether common tools could have found the defect early Particularly tools that detect code smells or verify system assertions

23 Facts on fixes The size of fixes with respect to other changes –Fixes are typically two to three times smaller than other changes Fixes and failures –Fixes are more likely to induce failures than other changes Fixes and older modules –A module that is one year older than another has 30 percent less errors Newly written code versus old code –Newly written code is 2.5 times as likely to contain a defect than old code

24 References Zeller, Andreas. Why Programs Fail. Second edition. Chapters 15 and 16.