A Review of Software Testing - P. David Coward

What is Testing? Does software match requirements? Specs might not be correct Testing attempts to assess how well tasks have been performed Expected output vs. output from test

Two types of Testing Testing Non – Functional Implementing customer functions not good enough Need additional requirements Satisfies legal obligations Performs within specified response times Meets documentation standards Functional Addresses whether program produces correct output Regression: name given to testing following modifications

Views to Testing Aims of Testing Constructive Destructive Too gentle of testing so may miss some faults Destructive Finding faults in the software Affects programmer’s ego Solution: QA team Have to find faults and demonstrate their absence by looking for faults Metrics must be used to assess thoroughness of tests and develop techniques that are implemented

Testing Strategies Testing strategies Structural Testing Functions included in software, but not required then use this method Ex: method to retrieve data from database Functional Functions identified and the faults should be documented 2 steps Identify the functions II. Create test data which will check the functions

Functional Testing Don't care how program performs these functions(aka Black Box Testing) Rules constructed for direct identification of functions and data from systematic design documentation Not enough to classify faults (have to distinguish the properties of the faults) Tester submit cases to program based on intended function of the program States output for particular case (Oracle)

Structural Testing White box testing (based upon detailed design rather than functional requirements.) 2 scenarios: Execute program w/ test cases Functions of the program are compared w/ required functions for congruence Structural testing involves execution and may require execution of single path through program Like to test all paths, but we have phenomenal combinatorial explosions

Structural Testing Cont. Problems: Infeasible path: contradiction of some predicates at conditional statements Island code: series of code following transfer of control or program termination, which isn’t the destination

Static vs. Dynamic Analysis Static: testing without executing the software with data (mathematical approach) Dynamic analysis: Probes are utilized during software execution (records frequency of execution of elements of the program) Bridge between functional and structural testing Initially functional testing may dictate set of test cases which can be examined by dynamic analysis. The code can be analyzed structurally to determine test cases which will exercise code left idle by previous test.

Various Testing Techniques Static Structural: no execution, features assessed vary with techniques Symbolic execution: no real execution. Data values are replaced by symbolic values. One expression per output variable. Middle ground between testing data and program proving Done by flow graph: contains decisions points and assignments associated w/ each branch traversing the flow graph from entry point along a particular path. A list of statements and branch predicates are produced.

Various Testing Techniques Cont. Partition Analysis: Uses symbolic execution to identify sub domains of input data domain. Specs are expressed in a manner close to program code. Program Proving: Common method is “inductive assertion verification” Bridge between functional and structural testing Assertions placed at beginning and end of selected procedures. Assertion describes functions mathematically. If input assertion upon entry of code ensures truth of output upon exit then assertions are correct. “Validity” achieved by wide audience who can’t disprove the proof

Various Testing Techniques Cont. Anomaly Analysis: 1st level: compiler checks syntax 2nd level: searches for anomalies not outlawed by programming languages The existence of (island code) unexecutable code Problems concerning array bounds Failure to initialize variables Labels and variables which are not used Jumps into and out of loops This method doesn’t find a fault just the potential one(data flow anomaly)

Dynamic Functional Test Cases Executes test cases. No consideration to detailed designed Domain Testing: Cases are created based on informal classifications of the requirement into domains. Data or requirements provide foundation of cases. Results are compared to expectations. Random Testing: Produces data without reference to code or Spec. Main tool is random number generator where operational reliability can be derived Disadvantage: no guarantee of complete coverage of the code Use this method for a small subset of data

Dynamic Functional Test Cases Cont. Adaptive Perturbation Testing: Assesses quality of a set of test cases effectiveness measure to generate further test cases to increase effectiveness Use of executable assertions which developer inserts into SW Assertion: statement about reasonableness of values of variables. Initial test runs and assertion violations recorded. Optimization routines are used to find best value to replace discarded value to maximize assertion violations. Go until # of assertions can’t be increased.

Dynamic Functional Test Cases Cont. Cause – effect graphing: explores input combinations used for Boolean logic operators. Used for small cases

Dynamic Structural Test Cases Software is executed w/ test cases. Test cases are created upon analysis of software Domain and Computational Testing: are strategies for selecting test cases. Use struc of program and select paths used to identify domains. Path computation: set of algebraic expressions one for each output variable. Path condition is the conjunction of constraints in path. Empty path is infeasible and cannot be executed. Domain error: when cases follows wrong path due to fault in conditional statement

Dynamic Structural Test Cases Cont. Automatic Test Data Generation: Repeated use of path generation and predicate solving paths. This produces potential test cases where high coverage of paths is probable. High coverage may still not meet its specs. Maybe due to omission in program of one of the functions defined in spec. Solution: formal specification methods Mutation Analysis: Concerned w/ the quality of test data. (testing the test data) Mutants are modified programs. The same test data is used on these mutants If the result is different then the mutant is of no use (dead) If the result is the same then the mutant is of interest (live- change hasn’t been detected) Ratio of dead to live mutants is the benchmark

Conclusion Objective of software testing is to gain confidence in the software There are many testing techniques which aim to help achieve thorough testing Debate continues as to whether correctness can be inferred when a set of test cases find no errors For the production of correct software the wider the range of testing techniques used the better the software is likely to be.

Questions?