Bayesian Graphical Models for Software Testing David A Wooff, Michael Goldstein, Frank P.A. Coolen Presented By Scott Young

Applying Detailed Metrics to Testing  Can provide insight for performing risk analysis  Can provide concrete values to inform the customer or management, concerning software quality  May result in more efficient testing procedures

Meaningful Metrics  Simple metrics concerning testing can inform concerning the success of the testing process  They also can provide insight for when the test process is reaching completion  More complex testing analysis needs to be done in order to locate points of inefficiency in testing, as well as providing more fine-grained knowledge about test results

What is a Bayesian Graphical Model?  Also commonly called a Bayesian Belief Network  Is a directed graph, with nodes signifying indeterminate factors  Bayesian models are most commonly heard of today in relation to spam filtering  They are used to calculate probability based on pre-defined knowledge and relationships between components

Software Actions (SA’s)  A Software Action is an individual, fine grained component of the software project which accomplishes a single task.  An example of a software action in a system would be the processing of a credit card number.

Specifying Nodes  A node should be a collection of operations with the same prior probability of failure, as well as the same change in probability of failure given a test covering that set of operations.

Factors For Defining Probabilities  Level of code complexity  Reliability comparison with existing code which has been evaluated  Maturity of codebase  Typical reliability of author’s code  Similarities to existing code

Updating The Model  As testing continues, the model must be update in stepwise fashion to follow changes to components as they occur.  The probability of an individual node can be updated according to multiple criteria (which are necessarily assumptions) about remaining defects.

What The Results Provide  Tests should be arranged according to the software action(s) which they provide coverage for. Tests discovered to be redundant may be safely removed  Results demonstrate the perceived probability (or strength of belief/confidence) that there are no more existing faults within each SA

What Does This Mean For V&V?  Software producers can demand a level of confidence for components from their testing according to the role of the software and potential financial impact of defects in specific components.

Drawbacks  Informed knowledge is required in order to build a reliable model  This “informed knowledge” still consists of assumptions of relationships (though an assumption within an order of magnitude can still provide useful results)  The amount of additional work to formally track every SA may be prohibitive

Resources  David A Wooff, Michael Goldstein, Frank P.A. Coolen, “Bayesian Graphical Models for Software Testing”. IEEE Transactions on Software Engineering, May  Murray Cumming, “Bayesian Belief Networks”. Date unknown.  Kevin Murphy, “A brief introduction to Bayes’ Rule”. e.html, Jan 2004.