1. 1 Aditya P. Mathur Head and Professor Department of Computer Science, Purdue University ABB, Sweden Monday April 7, 2008 Towards a Radically New Theory of Software Reliability

2. 2 Reliability Probability of failure free operation in a given environment over a given time. Mean Time To Failure (MTTF) Mean Time To Disruption (MTTD) Mean Time To Restore (MTTR)

3. 3 Claim Existing theories of software reliability simplify the problem to the extent that they (almost) maximize the uncertainty associated with the estimated software reliability.

4. 4 Operational profile Probability distribution of usage of features and/or scenarios. Captures the usage pattern with respect to a class of customers.

5. 5 Reliability estimation Operational profile Random or semi-random Test generation Test execution Failure/Defect data collection Reliability estimation [Uncertainty evaluation?] Decision process

6. 6 Issues: Operational profile Variable. Becomes known only after customers have access to the product. Is a stochastic process…a moving target! Random test generation requires an oracle. Hence is generally limited to specific outcomes, e.g. crash, hang. What about an operational profile with impulse? This creates a non-differentiable probability function of the time-to-failure.

7. 7 Issues: Failure data Should we analyze the failures? If yes then after the cause is removed then the reliability estimate is invalid. If the cause is not removed because the failure is a “minor incident” then the reliability estimate corresponds to irrelevant incidents.

8. 8 Issues: Failure rate “That is, the failure rate, when unambiguously defined, does not have a physical reality; rather, it is a technical device, whose sole purpose is to convey the engineer’s personal opinion about the life characteristic of software.” Nozer Singpurwalla, “The failure rate of software: does it exists?”, IEEE Transactions on Reliability, vol. 44, no. 3,1995.

9. 9 Issues: Model selection Rarely does a model fit the failure data. Model selection becomes a problem. 200 models to choose from? New ones keep arriving! Markov chain models suffer from a lack of estimate of transition probabilities. To compute these probabilities, you need to execute the application. During execution you obtain failure data. Then why proceed further with the model?

10. 10 Issues: Markovian models Markov chain models suffer from a lack of estimate of transition probabilities. To compute these probabilities, you need to execute the application. During execution you obtain failure data. Then why proceed further with the model? C1C3C2 12 13 32 21 12 + 13=1

11. 11 Issues: Assumptions Software does not degrade over time; memory leak is not degradation and is not a random process; a new version is a different piece of software. Reliability estimate varies with operational profile. Different customers see different reliability. Can we not have a reliability estimate that is independent of operational profile? Can we not advertise quality based on metric that are a true representation of reliability..not with respect to a subset of features but over the entire set of features?

12. 12 Estimating Uncertainty Estimates of software reliability must the associated with uncertainty. But how to quantify uncertainty? Entropy based approach [Katerina et al. 2002] Moments based approach [Katerina et al. 2003] Monte Carlo approach [Katerina et al. 2003] Bayesian approach [Dai et al. 2007]

13. 13 Estimating Uncertainty Basic idea: Model the parameters as random variables. Use statistical (e.g. moments) or Simulation approaches to estimate variance. Problem: Does not correlate with likely faulty components in the program under test.

14. 14 Sensitivity of Reliability to test adequacy Coverage low high Desirable Suspect modelUndesirable Risky Reliability Problem with existing approaches to reliability estimation.

15. 15 Basis for an alternate approach Why not develop a theory based on coverage of testable items and test adequacy? Testable items: Variables, statements,conditions, loops, data flows, methods, classes, etc. Pros: Errors hide in testable items. Cons: Coverage of testable items is inadequate. Is it a good predictor of reliability? Yes, but only when used carefully. Let us see what happens when coverage is not used or not used carefully.

16. 16 Saturation Effect FUNCTIONAL, DECISION, DATAFLOW AND MUTATION TESTING PROVIDE TEST ADEQUACY CRITERIA. Reliability Testing Effort True reliability (R) Estimated reliability (R’) Saturation region Mutation Dataflow Decision Functional RmRm R df RdRd RfRf R’ f R’ d R’ df R’ m tfstfs tfetfe tdstds tdetde t df s t df e tmstms tfetfe u:uncertainty u1u1 u2u2 u3u3 u4u4

17. 17 An experiment [TeX] Tests generated randomly exercise less code than those generated using a mix of black box and white box techniques. Application: TeX. Creator: Donald Knuth. [Leath ‘92]

18. 18 An experiment [sort utility] UNIX sort utility [DelFrate et al. 1995]

19. 19 An experiment [coverage-reliability correlations] Unix utilities and space application [Garg 1995. MS Thesis]

20. 20 Modeling an application OS Component Interactions Component Interactions Component Interactions ……….

21. 21 Reliability of a component R(f)=  (covered/total), 0<  <1. Reliability, probability of correct operation, of function f based on a given finite set of testable items. Issue: How to compute  ? Approach: High correlation between coverage metrics and failures has been established via empirical studies. Such studies could provide estimate of  and its variance for different sets of testable items.

22. 22 Reliability of a subsystem R(C)= g(R(f1), R(f2),..R(fn), R(I)) C={f1, f2,..fn} is a collection of components that collaborate with each other to provide services. Issue 1: How to compute R(I), reliability of component interactions? Issue 2: What is g ? Issue 3: Theory of systems reliability creates problems when (a) components are in a loop and (b) are dependent on each other.

23. 23 Scalability Is the component based approach scalable? Powerful coverage measures lead to better reliability estimates whereas measurement of coverage becomes increasingly difficult as more powerful criteria are used. Solution: Use component based, incremental, approach. Estimate reliability bottom-up. No need to measure coverage of components whose reliability is known.

24. 24 Next steps Develop component based theory of reliability. Do experimentation with large systems to investigate the applicability of the their and its effectiveness in predicting and estimating various reliability metrics. Base the new theory on existing work in software testing and reliability.

25. 25 The Future Apple Confidence: 0.999 Level 0: 1.0 Level 1: 0.9999 Level 2: 0.98 Boxed and embedded software with independently variable Levels of Confidence. Mackie Confidence: 0.99 Level 0: 1.0 Level 1: 0.9999

26. 26 Select References F. Del Frate, P. Garg, A. P. Mathur, and A. Pasquini. On the Correlation Between Code Coverage and Software Reliability, Proceedings of the Sixth International Symposium on Software Reliability Engineering, IEEE Press,Toulouse, France, pp 124-132, October 24-27, 1995 S. Krishnamurthy and A. P. Mathur. On the Estimation of Reliability of a Software System Using Reliabilities of its Components, Proceedings of the 8th International Symposium on Software Reliability Estimation, Albuquerque, New Mexico, November 1997. M. H. Chen. A. P. Mathur, and V. J. Rego. A Case Study To Investigate Sensitivity Of Reliability Estimates To Errors In The Operational Profile, Proceedings of the Fifth International Symposium on Software Reliability Engineering, IEEE Computer Society Press, Monterey, California, November 6-9, 1994, pp 276-281. Katerina Goseva–Popstojanova and Sunil Kamavaram. Assessing Uncertainty in Reliability of Component–Based Software. Proceedings of the 14th International Symposium on Software Reliability Engineering (ISSRE’03), 2003. Yuan-Shun Dai and Min Xie and Quan Long and Szu-Hui Ng. Uncertainty Analysis in Software Reliability Modeling by Bayesian Analysis with Maximum-Entropy Principle, IEEE Trans. Softw. Eng.,V 33, No. 11, 2007, pp 781--795. P. Garg. On code coverage and software reliability. MS Thesis. Department of Computer Science, Purdue University. May 1995.