Scientific Debugging

Errors in Software Errors are unexpected behaviors or outputs in programs As long as software is developed by humans, it will contain potential errors

Typical methods of debugging Typically our approach to debugging is very scattered, and we learn primarily from experience. The more bugs we encounter, the more quickly we can debug. For most programmers, sourcing and systematically solving problems is challenging Debugging intuition is not easy for many developers, and can’t be relied on

Our approach to errors We can choose to see errors as independent frustrations We can assume everyone has the same intuition regarding debugging, but this proves unrealistic A smarter technique is approaching all errors from a scientific perspective Scientists approach problems with the scientific method Programmers are capable of approaching errors with a similar approach To do this we have to consider errors a natural phenomena

Scientific Debugging method - simplified approach 1. Observe a failure 2. Create a hypothesis for cause of failure based on observations 3. Use hypothesis to make predictions 4. If the experiment satisfies the predictions, refine or limit the hypothesis 5. If the experiment does not satisfy predictions, create a new hypothesis 6. Repeat 3,4,5 as needed until further refining is not possible Eventually we will arrive at a cause for the software failure

What is a theory? In informal speech, the word theory is sometimes confused with a “good guess” In a scientific context, a theory is a rigorously tested hypothesis attempts to explain past observations makes testable claims about future observations In scientific debugging, theories are proven causes of software failure

Shell Sort revisited

Shell Sort debugging example - 1 For shell sort, we will note the following: Hypothesis - The Sample program works Prediction - Attempting to sort the collection “11 14” will result in “11 14” Experiment - Run sample on given input Observation - the rejected output is “0 11” Conclusion - rejected hypothesis

Shell Sort debugging example - 2 For our new hypothesis, let’s try restricting to a simpler case. Hypothesis - our array start index, a[0] will stay 0 Prediction - If we insert a breakpoint in our debugger at 37, a[0] is 0. Experiment - Run sample with a[0] as 0 Observation - the correct output is 0 Conclusion - supported hypothesis Continuing down this trend, we can observe more stringent hypotheses and reach our final theory Maintaining this approach of systematic checking will ensure consistent debugging

Explicit Debugging Scientific Debugging tests hypotheses which are explicitly stated Explicit in this case means problems are well defined and documented When debugging, developers often keep error notes in their mind of the problems This is an enormous mistake because we are human, and humans are forgetful much harder to see patterns emerge coworkers can’t help you nearly well enough Organizing thoughts in and of itself is useful It becomes vital once we have multi-layered issues “Writing bugs” only useful when organized

Explicit Debugging One solution: keep a logbook Useful because: we avoid relying on own memory there is a permanent record of bug solutions, in case similar bugs occur in other software it’s easier finding related errors (multiple errors with a single fix)

Algorithmic Debugging Try to automate the process of debugging Walk through execution of a program Observe current state and values Follow valid input until it “breaks” Stop when we transition from valid -> invalid Can be thought of as a “tree” of execution states

Insertion Sort Example - Algorithmic

Example: Algorithmic Debugging Sort([2,1,3]) == [3,1,2]? no → first check left execution path

Example: Algorithmic Debugging Sort([3]) == [3] ? yes → mark OK, check other path

Example: Algorithmic Debugging Insert(1,[3]) != [3,1] → algorithmic debugger marks invalid

Scientific Debugging - Pros and Cons Scientific Debugging is very helpful when: the software is designed as small modular parts for easier separate testing our program performs a linear procedure of instructions it is easy to tell the difference between “unfinished” and “bad” answers our output is easy to parse and test mission critical software needs testing Not as helpful when: “unfinished” and “bad” output are very similar, and confuse our debugging method we have very convoluted data structures we perform several datatype conversions which break many types of testing we combine multiple languages in a single piece of software

Obtaining a Hypothesis - Understanding the Problem In order to use scientific debugging, we have to arrive at meaningful hypotheses There are a handful of tools to do this to help narrow our effective results There are several key ingredients of debugging:

Hypothesis Formation - Problem Description A problem you cannot clearly describe is a problem you cannot clearly solve “Shell sort doesn’t work on input [2,1,3]” is not a useful description Observing that shell sort fails once we call insert() is useful Problem reports and problem tracking are vital to narrowing problem scope

Hypothesis Formation - Program Code It is intuitive that programmers should know their code in order to debug it However this becomes very difficult for large projects reliant on others’ work Important to keep in mind when relying on libraries and external projects as well For our shell sort example, would have been impossible to see insert() problem

Hypothesis Formation - Earlier Hypotheses One other crucial rule to keep in mind is hypothesis refinement When we update a hypothesis: Include earlier hypotheses that passed Exclude earlier hypotheses that failed Future hypotheses must also keep track of prior observations

Reasoning about Programs - using Deduction Deduction is reasoning that moves from the general to the particular These take the form of mathematical proofs If these abstractions of our program are true, so are properties we can deduce This technique is useful because it doesn’t require running the program If we don’t run the program, it is static analysis If we use deduction but also test against execution, it is a dynamic analysis In general static attempts to predict the future dynamic holds onto values from the past It’s also difficult because abstracting our program can be challenging

Reasoning about Programs - using Observation Observation is a dynamic method dynamic: requires execution of program for testing take in facts from execution majority of these methods also look at program code so they usually include deduction There are multiple different kinds of observational methods including: Inductive methods Experimental methods Deductive methods (if using dynamic deduction) All use information about the past to use for debugging solutions

Reasoning about Programs: using Induction Reasoning from particular to general (unlike deduction) Attempt to summarize multiple runs through formal inductive proof 1. Start with base case for program input (for shell sort, sort a single element) 2. Attempt to show that if it works for n inputs, 3. It will also work for n+1 inputs This requires multiple executions to test, so induction is also dynamic All inductive methods are also observational because they are dependent on observing output of multiple runs

Reasoning about Programs - using Experimentation Searching for a failure through experiments constantly refining and updating a hypothesis this reduces our hypotheses to a theory Based on multiple program runs, so also dynamic each run is guided by a specific experiment to narrow the hypotheses In general, experimentation techniques are so named if: they generate findings from multiple runs these runs are controlled by some hypothesis or test