What is Software Testing? And Why is it So Hard J. Whittaker paper (IEEE Software – Jan/Feb 2000) Summarized by F. Tsui

Common Reasons of “Bugs” Escaped to Customers User executed an “untested” segment of code. User executed the sequence of code differently in actual use from any of the test cases User applied a different combination of input values from any of the test cases User’s operating environment is different from any of the tested environment --- or an environment that was conscientiously untested due to cost. Note: These reasons give us a “hint” of where and what to test.

Proposed: 4 Major Testing “Phases” or “Activities” 1.Model the User’s Software Environment 2.Generating & Selecting the Test Scenarios 3.Running and Evaluating the Test Scenarios 4.Record the Test Results and Measure the Progress

1. Modeling the Software Environment Simulate the interactions between the software and its environment; mostly in the interfaces that the software uses: – Human Interface (e.g. GUI with mouth clicks, key board inputs, other devices) – Software Interface (e.g. software interfaces to Op System or other modules or apps, especially in the areas of “error” return messages) – File System interfaces (e.g. data written out to or read from external files or DB) – Communications interface (e.g. protocols to the network, both valid and invalid ones) In testing the interface the tester must consider 2 things: 1) the actual data value for the test inputs and 2) the sequencing of the test inputs e.g. -1 in lock-stock-barrel

Interface Testing Suggestions Boundary Value Partitioning (combination of what we called boundary value and equivalence class) is often used for input value choices The sequencing of inputs must consider the “dependencies” among the inputs. (our “strong normal” or “worst case” case testing) – Exercise both the “legal” combination and sequence along with the “illegal” combination and sequence. Use “Models” to help set up sequences of inputs and transitions: – Graphs – State Diagrams, – Logic Tables, etc.

2. Selecting Test Scenarios There are often too many test cases (possibly infinite) to run, but schedule and resource forces us to select a subset. – Consider “Coverage” as criteria: 1.Code statements coverage – how much of the source code are executed at least once? 2.Input coverage – how many of the inputs and how much of the variations for each input are covered? (boundary value, robust testing, -----, decision table) 3.Output coverage - how many of the outputs are covered and to what depths? – Consider the execution paths: Sequences of inputs to execute different paths of the source code – Major user paths in terms of “typical usage” scenarios – Less well –defined (thus less well coded) and minor functional paths only used by a minority of users. my addition A bit different from just plain path testing ---- needs to know the usage & users and application domain

Execution Paths Testing as Discriminating “Criteria”: For Control flow Testing – Every source line executed – Every branch (case, while, if-then-else) For Data flow testing – Data “define” to data “usage” path – Data structure initialization to usage Think about our lecture on paths testing: - code coverage - branch coverage - linearly independent paths - all logical combinations how far would you go? How much of the D-U paths would you use ---- just the D to C-use, D-to P-use, or all D-U paths? Would you include program slice testing to focus on certain data?

Fault Seeding and Discovery as “criteria” Insert defects into the code Design test scenarios to detect seeded defects, but the seeded defects are not necessarily known to the testers. Use the amount of seeded defects uncovered to predict how effective the testing methodology is e.g. : - put in 10 seeded defects - set the goal to detect, say 90%, of the seeded bugs - but, say, found only 6 seeded bugs and 25 “real” bugs - 6/10 = 25/x - x = 250/6 ; which implies there are 42 total “real” bugs - so, we believe 42-25= 17 more real bugs left - decision would be keep on testing until we discover 90%seeded defect

Input Domain Testing as “criteria” Select test cases to cover all physical input Select test cases that causes each input interface control to be stimulated (e.g. window menu, radio button, drop downs, etc.) Discriminating Criteria for these above: – Statistically equivalent to covering the complete set (equivalence partitioning) – As stated before, typical or most likely to be executed by users

3. Running and Evaluating Test Scenarios Executing the designed test scenarios: manually executing test scenarios are expensive ---so often testing includes a tool that aids in automating some aspect of the test execution Evaluating test results : comparing the actual test result against the expected test result --- needs human oracle to compare results ?. – Use formal specification to state the expected result and then compare against the actual results, also specified in formal terms. (my example: use pre & post conditions with “assert”.) – Embedding some code to show the executing results (e.g. out- putting variable values to be checked by an external person or automated program in some strategically picked program slices my words

Regression Testing – just re-running old test cases? Regression test is a form of testing changes (& fixes) Often an application software may go through several versions before final testing and release, each version may contain corrected code, modified code, and/or additional functional code: – We need to test to see that the newer version with these potential changes did not “regress” what was working before in the previous versions. This is called regression testing. How much testing do we do? 1.Test only the modified areas 2.Test only the modified areas and the “immediate neighborhood” 3.Test the new/modified areas and also re-run all the previous test scenarios

Some Other Concerns (when evaluating tests) Should code that is hard to test or buggy be re- written? – What is “hard to test” code ? – How much of the code should be re-written if a code piece is judged to be hard-to-test Recreating or reproducing a failed case is not always easy because the state under which the code failed is not always understood. – What do we do with non-reproducible failures? Report them but do nothing? Do not even report those? - We do not have a clear definition of hard to test code – “non-cohesive” code? - Furthermore, are we writing code for users or for testers and other developers?

4. Measuring Test Progress Instead of just reporting “numbers” of test cases designed, or executed; number of failures found; etc., should also ask: – Have we considered “common” programming errors – Have we forced all data to be initialized? – Have we found all seeded errors? – Have we tried the expected user usage modes? Testers and Support would also like to know how many defects are left in the code and the chance of these showing up in the field. We talked about this when you reported bugs for inspection. e.g. problem type and severity; problems/page (rate) How do we deal with this ?

More on Measurement and Models Testability: (not sure how measured) – If a software has “high” testability, then it is easier to test and thus more likely to have the defects found and removed prior to releasing to users. (refers to Voas’ notion of testability --- another paper) If we found small # of bugs in highly testable code --- then we should be happy --- & finding lots of bugs in low testable code is also a good sign? Reliability models – Models the frequency of defects found through testing – Using the model to predict the probability of defects to be found in the user environment

Finally Hire the “smartest” & “qualified” people you can get. Equip them with proper tools and training Give them the authority and time to perform their work Listen to them when they speak about the “quality” of the product. My addition