1. Extension of E(Θ) metric for Evaluation of Reliability

2. Presentation ► Background on Reliability and E(Θ) ► Testing Methodologies ► Present formulae for calculating E(Θ) ► Need of a new formula ► Formula Derivation ► Validation of Derivation Technique ► Summary and Future Work

3. E(Θ) ? E(Θ) ? ► E(Θ) - Reliability of a software is defined in terms of its E(Θ) value. ► E(Θ) refers to the probability of the software encountering a failure subsequent to testing

4. Why measure reliability ► Cost of Designing a Faulty S/W in less time >> Cost of Designing a Reliable S/W time >> Cost of Designing a Reliable S/W ► Safety Critical Applications

5. Faults or Bugs ► Failure – Unexpected O/P ► Faults – I/Ps causing Failures

6. How to measure reliability Testing: Testing: ► Means of Probing S/W for bugs ► Means of gaining confidence about its reliability Source : Source : Evaluating Testing Methods by Delivered Reliability -Phyllis G. Frankl, Richard G. Hamlet, Member, IEEE, Bev Littlewood, Member, IEEE,and Lorenzo Strigini, Member, IEEE

7. Testing Methods ► Operational – Test Case = Statistical Distribution or Random (I/P domain) ► Debug – Test Case = Selective Sampling (I/P Domain)

8. Operational Vs. Debug ► Debug – Probes for bugs actively (assumes S/W is faulty) ► Operational – Waits for Failures to occur.

9. Sub-Domain Testing ► A debug testing technique. ► Concentrates test-cases to specific sub-domains with high failure probability ► Based on Debugger’s Intuition ► Eg. Boundary Testing – Bugs lie in the corners

10. Example 0 - ve x + ve x - ve y + ve y Failure Region Input domain Failure Region

11. Sub-Domain Testing So Far… Sub-Domain Testing So Far…

12. Terminologies used ► q = failure probability ► d = detection rate ► T = Number of test cases per sub-domain ► n = n number of sub-domains/failure regions

13. Single Domain - Single Failure Region A, 20 Failure Region B, 30 Domain nature: Single Domain = 1 – 50 (50 points) Failure Region A = 21 – 40 (20 points) E(Θ) = q( 1 - d) T

14. Single Failure Region – Multiple Sub- Domains A i,5 A j, 15 A k,10 B, 20D, 25 Domain nature: Sub-Domain 1= 1 – 25 (25 pt.s) Sub-Domain 2= 25 – 50 (50 pt.s) Sub-Domain 3= 50 – 85 (35 pt.s) Failure Region A = 40-70 (30 pt.s) E(Θ) = q n Π i=1 ( 1 - d i ) T C, 10

15. Multiple Failure Regions – Single Sub-Domain A, 10 B, 20 D, 55 Domain nature: Domain = 1 – 100 (100 pt.s) Failure Region A = 20-30 (20 pt.s) Failure Region B = 50-70 (20 pt.s) Failure Region C = 80- 95(20 pt.s) E(Θ) = n Σ i=1 q i ( 1 - d i ) T C, 15

16. Multiple Failure Regions – Multiple Sub-Domains A, 20 B, 20 C, 30 D, 30 Domain nature: Sub-Domain 1= 1 – 50 (50 pt.s) Sub-Domain 2= 50 – 100 (50 pt.s) Failure Region A = 30-50 (10 pt.s) Failure Region A = 70-90 (20 pt.s) E(Θ) = ? No definitive Formula. Empirical Testing only Solution

17. And Now……

18. New Set of Terminologies ► q = Total failure probability ► d A = Detection rate of a failure region in a sub-domain ► F i = Number of points in a failure region ► M = Total number of failure points in the sub-domain

19. Multiple Failure Regions – Multiple Sub-Domains A, 20 B, 20 C, 30 D, 30 Domain nature: Sub-Domain 1= 1 – 50 (50 pt.s) Sub-Domain 2= 50 – 100 (50 pt.s) Failure Region A = 20-40 (20 pt.s) Failure Region A = 70-90 (20 pt.s) E(Θ) = q – ( d A F 1 /M+ d B F 2 /M)

20. Formula Derivation A, 20 B, 20 D, 30 Fault Case Regions Detected Faults Remaining 00 NoneA, B 01 B A 10AB 11A, B None E (Θ) = (Probability that the particular case will occur) * (probability that the rest of the faults are undetected) C, 30

21. Contd. Assuming 1 test-case per sub-domain Test Case 1 Test Case 2 Fault A Fault BCase Number Region D Region Cundetected Region D Region A Region B Region C Region B undetected detected undetected detected Seq. 1 Seq. 2 Seq. 3 Seq. 4 E (Θ) = E(Θ) (seq1) + E(Θ) (seq2) + E(Θ) seq3) + E(Θ) seq4) contd.

22. E (Θ) = E(Θ) (seq1) + E(Θ) (seq2) + E(Θ) seq3) + E(Θ) seq4) = {(1- dA)* (1- dB)* (q)} + {(1-dA)* (dB)* (q –F1/M)} + {(dA)* (1- dB)* (q –F2/M)} + {(dA)* (dB)* (q –F1/M – F2/M)} = {(30/50 * 30/50) * 40/100} + {(30/50 * 20/50) * 20/100} + {(20/50 * 30/50) * 20/100} + {(20/50 * 20/50) * 0/100} = 0.144 + 0.048 + 0.048 = 0.24

23. Verification of the Technique A, 20 B, 20 C, 60 Domain nature: Domain = 1 – 100 (100 pt.s) Failure Region A = 20-40 (20 pt.s) Failure Region A = 70-90 (20 pt.s) Approach I: E(Θ) = n Σ i=1 q i ( 1 - d i ) T = 20/100(1- 20/100) + 20/100(1- 20/100) = 20/100(1- 20/100) + 20/100(1- 20/100) = 0.32 Approach II: E(Θ) E(Θ) = {(1- dA)* (1- dB)* (q)} + {(1-dA)* (dB)* (q –F1/M)} + {(dA)* (1- dB)* (q –F2/M)} + {(dA)* (dB)* (q –F1/M – F2/M)} contd.

24. = {(80/100 * 80/100)(40/100)} + {(80/100 * 20/100)(20/100)}+ {(20/100 * 80/100)(20/100)} + {(20/100 * 20/100)(0/100)} = 0.32 Both approaches yield the same result E(Θ) Similar tests done for different scenarios validate the 2 nd technique of calculating E(Θ) with respect to Frankl’s Method

25. Getting back to Formula Derivation Upon further solving the method just described we get ….

26. Method Reduction E (Θ) = { (1- dA)* (1- dB)* (q) } + { (1-dA)* (dB)* (q –F1/M) } + { (dA)*(1- dB)* (q –F2/M) } + { (dA)* (dB)* (q –F1/M – F2/M) } = q [ {1–dB – dA + dAdB} + { (dB – dAdB) (1–F2/M*q) } + { (dA – dAdB) (1-F1/M*q) } + { dAdB (1–F1/M*q – F2/M*q) } ] Contd.

27. = q [ {1–dB –dA + dAdB} + { dB – dB*F2/M*q – dAdB + dAdBF2/M*q}+ { dA – dAF1/M*q – dAdB + dAdBF1/M*q} + { dAdB–dAdB*F1/M*q – dAdBF2/M*q} ] = q [ 1– dBF2/M*q – dAF1/M*q] = q – ( dAF1/M + dBF2/M )

28. Three Failure Regions & Three Sub-domains E (Θ) = {(1- dA)* (1- dB)* (1- dC)*(q)} + {(1-dA)* (1-dB)* (dC)*(q –F3/M)} + {(1-dA)* (dB)* (1-dC)*(q –F2/M)} + {(1-dA)* (dB)* (dC)*(q –F2/M–F3/M)} + {(dA)* (1-dB)* (1-dC)*(q –F1/M)} + {(dA)* (1-dB)* (dC)*(q –F1/M–F3/M )} + {(dA)* (dB)* (1-dC)*(q –F1/M–F2/M)} + {(dA)* (dB)* (dC)*(q –F1/M–F2/M–F3/M)} = q[ 1 – dCF3/ M*q – dBF2/M*q – dAF1/M*q] q – { dAF1/M + dBF2/M + dCF3/M} = q – { dAF1/M + dBF2/M + dCF3/M}

29. Simulator ► Based on the above mentioned & verified technique a simulator was designed for empirical calculation of ► Based on the above mentioned & verified technique a simulator was designed for empirical calculation of E (Θ) ► Coding Language : Java ► Graphical User Interface : Java Swings

30. Simulator screen shots Simulator screen shots Step I: Specify the total number of Sub-Domains & Failure Regions

31. Step II: A, 20B, 20 C, 30 D, 30 Specify the number of points per Sub-Domain, Failure Region

32. The result got from running the simulator is similar to the one calculated theoretically (above). This proves validity of the Simulator Design.

33. Summary ► ► Value of E (Θ) for software with multiple failure regions spread across multiple sub- domains can be calculated using: E (Θ) = q – ( dAF1/M + dBF2/M) q – ( dAF1/M + dBF2/M + dCF3/M) E (Θ) = q – ( dAF1/M + dBF2/M + dCF3/M)

34. Future Work ► The number of test-cases per sub-domain was limited to only 1 ► We would like to generalize the present formula further by considering the effect of ‘n’ (multiple) test cases per sub-domain. ► No generalization doesn’t merely involve putting ‘T’ as a exponential term in the equation ( have tried that )