PPT 4: Reproducing the Problem CEN Software Testing

First Things First: Reproduce the Problem Reproducing the problem is important for two reasons Observe the problem: if unable to reproduce the problem the error cannot be deduced Check whether the problem is fixed: Recreate the original scenario with the design change to demonstrate Initial Debugging Process Step 1 - Reproduce the problem environment The Setting in which the problem originally occurred Step 2 - Reproduce the problem history Prior steps to create the problem

First Things First: Reproduce the Problem Usually debugging in the problem environment leads to the best chance of reproducing the problem, however may not be possible due to: Privacy: the user may not wish for others to operate their machines Ease of development: User may not have a complete development environment (diagnostic software) Cost of maintenance: User may require that the machine is operational and in their possession Travel cost: Cost of either travelling to the machine or being sent the machine Risk of Experiment: Debugging involves experimenting and may cause damage to machine

Iterative Reproduction 1. Attempt to reproduce the problem in your environment, if the problem occurs you are lucky 2. If initial problem does not occur then add more circumstances from the problems environment, (Config Files, Drivers, Hardware) Start with circumstances That are likely to cause problems Easy to change 3. Adopt additional circumstances until Problem is reproduced Your environment is identical to problem environment

Iterative Reproduction If the development and problem environment are identical there exists two alternatives: The problem could not occur in the problem environment and the problem report may be incomplete or wrong The Problem report is accurate but there still exists a difference between environments that was unstated by user

Iterative Reproduction If reproducing environment was insufficient then reproducing the execution may lead to success The execution of a program is determined by code (assumed constant) and user input To reproduce user input, programmer must Observe the input Control the input Input may be Deterministic or Nondeterministic Deterministic: input controllable by the user Nondeterministic: input not controlled by user

Program Inputs Inputs for problem reproduction Data: stored in files and databases, easily reproducible User Interaction: Can differ in minor details, difference may be important Communication: data between threads, processes, or machines Time: Can influence program execution Randomness: By definition make every execution different Operating System: Provides services beyond the above and can influence program execution Schedules: Normally doesn’t interfere but if the program has a defect it may Debugging Tools: Interferes with programs execution may uncover but also mask problems Physics: Unintended electrical discharges and radiation may affect program execution.

Data Reproducing Data: data stored in files/databases, seldom an issue Only three issues to be aware of: Get all the data you need: reproduce all data the application accesses and all data under user’s control. Includes configuration data such as registries or configuration files Get only the data you need: Some programs use enormous amounts of data, so simplifying input data may be prudent Take care of privacy issues: User’s data may contain private or confidential information.

User Interaction Capturing interaction: Program operate normally, interacting environment but all inputs from environment are recorded to a script before program uses it Replaying interaction: Program executes under control of the tool Tool redirects program input so the program no longer receives input from the environment

Communication Record since last checkpoint Not all data is required. If a program goes into a reproducible state while operating only capture during that state Record last transaction Failures are more likely caused by later communications than earlier. Suffices to capture just the latest communications.

Randomness Program behavior in different runs Individual runs may be impossible Randomness must be made controlled and reproducible Check using random input If utilizing pseudo random numbers, capture initial seed of the random number generator If utilizing Cryptographic, then record all sources of randomness User Input Network Events Thermal Noise Audio

Operating System Record and replay OS interactions Rather than setting up a layer between a program and the OS, isolate arbitrary program units from their environment Traced interaction Works by diverting the calls to the OS to wrapper functions that log the incoming and outgoing data Challenges of replaying traced interactions The primary problem is that every interaction must be replayed which eats a lot of memory

Schedules Thread changes Debugging here is potentially the most difficult When multiple threads attempt to change the same file sourcing the error may be impossible To solve accessing threads must “lock” the file, thus protecting it. Or check last update time before accessing and double checking before saving Reproducing schedules Like reproducing communications, performance may become an issue by reproducing all inputs A program should behave identically so one may attempt to uncover difference in execution as induced by schedule differences

Debugging Tools Code may fail outside the debugger only Simply observing or examining the problem may cause the problem to disappear, or be replaced by another otherwise known as a Heisenbug Analogy to Heisenberg’s uncertainty principle May be caused by uninitialized variables Debugging tools may also introduce timing issues Extended run time by routing commands to the debugging environment Bugs within the debugging tools may cause additional issues

Debugging Tools Types of bugs: Bohr bug: Repeatable bug - one that manifests reliably under unknown well-defined set of conditions Heisenbug: Bug that disappears or alters its behavior when one attempts to probe or isolate it Mandelbug: Bug whose underlying cause are so complex and obscure as to make its behavior appear chaotic or non-deterministic. Schroedinbug: Design or implementation bug that does not manifest until someone reading source or using the program unusually notices it never should have worked In which the program stops working until everything is fixed

Isolating Units The general premise is not to reproduce the execution of the entire program but rather the execution of a specific unit May be easier than controlling entire program However only this unit will provide information Capture and replay units Create mock object that would do nothing but simulate the original object using recorded interactions. Unit control layer For bottom-level units simply record and replay the interaction at the unit boundary

Isolated Units Databases: To reproduce problem, record and replay the SQL transactions given by an application - rather than executing the entire program Compilers: To reproduce problem, record and restore the intermediate data structures - rather than the entire front end Networking: To reproduce problem, record and replay the basic communication calls as issued by the application - rather than the entire application.

Log as a Program Aim is to create a control layer that logs all input and output of a mapping from string to integers Simple way of creating such a map is to create a log as a stand-alone program file These log files can be compiled and executed, thus reproducing the interaction of a map object and its environment

Logging Interaction with the OS clog: a C++ buffered output stream generally used for error / debug messages The code example on the right is a simple program attempting to open a file and if it fails displays an Error message

Other Interaction Types Variables: Some units provide variables that applications can access to alter unit’s behavior, or retrieve query results These implicit communications via variables must be controlled and require a lot of work Unless capture/replay mock tools are utilized Time: Some units depend on a specific amount of time to elapse between function calls Record and replay these time intervals for debugging Other Units: Some units depend on other unit, which may also be controlled by the application Be sure to capture and restore the states of these units Breaking dependency of the units may be needed

Tools WINRUNNER Records and replays user input for Windows and UNIX systems, CHECKPOINT Allow the “freezing” of a running process so it can be resumed later Even on a different machine, ANDROID Open-source GUI testing tool, REVIRT Records and replays interaction of entire machine,