Software Fault Interactions and Implications for Software Testing Presented By: Sai Praneeth Purighella.

Introduction For example, 20 inputs, each having 10 possible values, yield a total of 1020 combinations of settings. Only a few hundred test cases can be built and executed under most budgets Empirical research into quality and reliability suggests that software failures in a variety of domains were caused by combinations of relatively few parameters. If ‘n’ be the number of parameters involved, then testing all n-tuples of parameters is effectively equivalent to exhaustive testing (pseudo exhaustive). The number of conditions required to trigger a failure is referred as the failure-triggering fault interaction (FTFI) number.

Related Work Nair et al.’s case study of combinatorial testing for a small subsystem of a screen-based administrative database. Wallace and Kuhn’s review of medical device recall data gathered by FDA. Smith et al.’s investigation on the pairwise testing of RAX software on NASA’s Deep Space 1 mission.

Empirical Analysis Analysis of 329 error reports from development and integration of a large distributed system at NASA Goddard Space Flight Center. Results of analysis are shown in last column of Table 1 and in Table 2. Distribution of failure-triggering conditions appears to follow a power law. (See figure 1, last four columns of Table 1).

Figure 1

Implications of Testing Consider the example with 20 inputs, each with 10 possible values. Exhaustive testing requires 1020 test cases. But empirical studies show that most failures are triggered by a single erroneous parameter. However, all could be triggered by fewer than 6 parameters.

Background Calculations Consider the problem of n-tuples of k-parameters, each of which has ‘v’ possible values. The number of n-tuples drawn from k- parameters is

Implications (Continued) Since each parameter has ‘v’ values, total number of test cases would be C(k,n).vn. Let us see at the example previously considered. For a given number N, of test cases, and a specifies level n-tuple, how many values can or must be covered can becalculated using the expression: vn<=N, so n log v <= log N.

Conclusions If all errors in a particular class of software are triggered by finite combinations of ‘n’ values or less, then testing all combinations of n or fewer values would provide a form of “pseudoexhaustive testing”. Appropriate levels of ‘n’ appear to be 4<=n<=6 when considering pseudoexhaustive testing, according to dependability requirements. This approach might not be good effective real-time or other software that depends on testing event sequences, but may be applicable to subsystems within real-time software. Many more empirical studies of other classes of software are needed to evaluate the applicability of combinatorial testing for other classes of systems.

Thank You One and All!