Download presentation

Presentation is loading. Please wait.

Published byAnika Bowman Modified over 2 years ago

1
Extension of E(Θ) metric for Evaluation of Reliability

2
Presentation ► Background on Reliability and E(Θ) ► Testing Methodologies ► Present formulae for calculating E(Θ) ► Need of a new formula ► Formula Derivation ► Validation of Derivation Technique ► Summary and Future Work

3
E(Θ) ? E(Θ) ? ► E(Θ) - Reliability of a software is defined in terms of its E(Θ) value. ► E(Θ) refers to the probability of the software encountering a failure subsequent to testing

4
Why measure reliability ► Cost of Designing a Faulty S/W in less time >> Cost of Designing a Reliable S/W time >> Cost of Designing a Reliable S/W ► Safety Critical Applications

5
Faults or Bugs ► Failure – Unexpected O/P ► Faults – I/Ps causing Failures

6
How to measure reliability Testing: Testing: ► Means of Probing S/W for bugs ► Means of gaining confidence about its reliability Source : Source : Evaluating Testing Methods by Delivered Reliability -Phyllis G. Frankl, Richard G. Hamlet, Member, IEEE, Bev Littlewood, Member, IEEE,and Lorenzo Strigini, Member, IEEE

7
Testing Methods ► Operational – Test Case = Statistical Distribution or Random (I/P domain) ► Debug – Test Case = Selective Sampling (I/P Domain)

8
Operational Vs. Debug ► Debug – Probes for bugs actively (assumes S/W is faulty) ► Operational – Waits for Failures to occur.

9
Sub-Domain Testing ► A debug testing technique. ► Concentrates test-cases to specific sub-domains with high failure probability ► Based on Debugger’s Intuition ► Eg. Boundary Testing – Bugs lie in the corners

10
Example 0 - ve x + ve x - ve y + ve y Failure Region Input domain Failure Region

11
Sub-Domain Testing So Far… Sub-Domain Testing So Far…

12
Terminologies used ► q = failure probability ► d = detection rate ► T = Number of test cases per sub-domain ► n = n number of sub-domains/failure regions

13
Single Domain - Single Failure Region A, 20 Failure Region B, 30 Domain nature: Single Domain = 1 – 50 (50 points) Failure Region A = 21 – 40 (20 points) E(Θ) = q( 1 - d) T

14
Single Failure Region – Multiple Sub- Domains A i,5 A j, 15 A k,10 B, 20D, 25 Domain nature: Sub-Domain 1= 1 – 25 (25 pt.s) Sub-Domain 2= 25 – 50 (50 pt.s) Sub-Domain 3= 50 – 85 (35 pt.s) Failure Region A = 40-70 (30 pt.s) E(Θ) = q n Π i=1 ( 1 - d i ) T C, 10

15
Multiple Failure Regions – Single Sub-Domain A, 10 B, 20 D, 55 Domain nature: Domain = 1 – 100 (100 pt.s) Failure Region A = 20-30 (20 pt.s) Failure Region B = 50-70 (20 pt.s) Failure Region C = 80- 95(20 pt.s) E(Θ) = n Σ i=1 q i ( 1 - d i ) T C, 15

16
Multiple Failure Regions – Multiple Sub-Domains A, 20 B, 20 C, 30 D, 30 Domain nature: Sub-Domain 1= 1 – 50 (50 pt.s) Sub-Domain 2= 50 – 100 (50 pt.s) Failure Region A = 30-50 (10 pt.s) Failure Region A = 70-90 (20 pt.s) E(Θ) = ? No definitive Formula. Empirical Testing only Solution

17
And Now……

18
New Set of Terminologies ► q = Total failure probability ► d A = Detection rate of a failure region in a sub-domain ► F i = Number of points in a failure region ► M = Total number of failure points in the sub-domain

19
Multiple Failure Regions – Multiple Sub-Domains A, 20 B, 20 C, 30 D, 30 Domain nature: Sub-Domain 1= 1 – 50 (50 pt.s) Sub-Domain 2= 50 – 100 (50 pt.s) Failure Region A = 20-40 (20 pt.s) Failure Region A = 70-90 (20 pt.s) E(Θ) = q – ( d A F 1 /M+ d B F 2 /M)

20
Formula Derivation A, 20 B, 20 D, 30 Fault Case Regions Detected Faults Remaining 00 NoneA, B 01 B A 10AB 11A, B None E (Θ) = (Probability that the particular case will occur) * (probability that the rest of the faults are undetected) C, 30

21
Contd. Assuming 1 test-case per sub-domain Test Case 1 Test Case 2 Fault A Fault BCase Number Region D Region Cundetected Region D Region A Region B Region C Region B undetected detected undetected detected Seq. 1 Seq. 2 Seq. 3 Seq. 4 E (Θ) = E(Θ) (seq1) + E(Θ) (seq2) + E(Θ) seq3) + E(Θ) seq4) contd.

22
E (Θ) = E(Θ) (seq1) + E(Θ) (seq2) + E(Θ) seq3) + E(Θ) seq4) = {(1- dA)* (1- dB)* (q)} + {(1-dA)* (dB)* (q –F1/M)} + {(dA)* (1- dB)* (q –F2/M)} + {(dA)* (dB)* (q –F1/M – F2/M)} = {(30/50 * 30/50) * 40/100} + {(30/50 * 20/50) * 20/100} + {(20/50 * 30/50) * 20/100} + {(20/50 * 20/50) * 0/100} = 0.144 + 0.048 + 0.048 = 0.24

23
Verification of the Technique A, 20 B, 20 C, 60 Domain nature: Domain = 1 – 100 (100 pt.s) Failure Region A = 20-40 (20 pt.s) Failure Region A = 70-90 (20 pt.s) Approach I: E(Θ) = n Σ i=1 q i ( 1 - d i ) T = 20/100(1- 20/100) + 20/100(1- 20/100) = 20/100(1- 20/100) + 20/100(1- 20/100) = 0.32 Approach II: E(Θ) E(Θ) = {(1- dA)* (1- dB)* (q)} + {(1-dA)* (dB)* (q –F1/M)} + {(dA)* (1- dB)* (q –F2/M)} + {(dA)* (dB)* (q –F1/M – F2/M)} contd.

24
= {(80/100 * 80/100)(40/100)} + {(80/100 * 20/100)(20/100)}+ {(20/100 * 80/100)(20/100)} + {(20/100 * 20/100)(0/100)} = 0.32 Both approaches yield the same result E(Θ) Similar tests done for different scenarios validate the 2 nd technique of calculating E(Θ) with respect to Frankl’s Method

25
Getting back to Formula Derivation Upon further solving the method just described we get ….

26
Method Reduction E (Θ) = { (1- dA)* (1- dB)* (q) } + { (1-dA)* (dB)* (q –F1/M) } + { (dA)*(1- dB)* (q –F2/M) } + { (dA)* (dB)* (q –F1/M – F2/M) } = q [ {1–dB – dA + dAdB} + { (dB – dAdB) (1–F2/M*q) } + { (dA – dAdB) (1-F1/M*q) } + { dAdB (1–F1/M*q – F2/M*q) } ] Contd.

27
= q [ {1–dB –dA + dAdB} + { dB – dB*F2/M*q – dAdB + dAdBF2/M*q}+ { dA – dAF1/M*q – dAdB + dAdBF1/M*q} + { dAdB–dAdB*F1/M*q – dAdBF2/M*q} ] = q [ 1– dBF2/M*q – dAF1/M*q] = q – ( dAF1/M + dBF2/M )

28
Three Failure Regions & Three Sub-domains E (Θ) = {(1- dA)* (1- dB)* (1- dC)*(q)} + {(1-dA)* (1-dB)* (dC)*(q –F3/M)} + {(1-dA)* (dB)* (1-dC)*(q –F2/M)} + {(1-dA)* (dB)* (dC)*(q –F2/M–F3/M)} + {(dA)* (1-dB)* (1-dC)*(q –F1/M)} + {(dA)* (1-dB)* (dC)*(q –F1/M–F3/M )} + {(dA)* (dB)* (1-dC)*(q –F1/M–F2/M)} + {(dA)* (dB)* (dC)*(q –F1/M–F2/M–F3/M)} = q[ 1 – dCF3/ M*q – dBF2/M*q – dAF1/M*q] q – { dAF1/M + dBF2/M + dCF3/M} = q – { dAF1/M + dBF2/M + dCF3/M}

29
Simulator ► Based on the above mentioned & verified technique a simulator was designed for empirical calculation of ► Based on the above mentioned & verified technique a simulator was designed for empirical calculation of E (Θ) ► Coding Language : Java ► Graphical User Interface : Java Swings

30
Simulator screen shots Simulator screen shots Step I: Specify the total number of Sub-Domains & Failure Regions

31
Step II: A, 20B, 20 C, 30 D, 30 Specify the number of points per Sub-Domain, Failure Region

32
The result got from running the simulator is similar to the one calculated theoretically (above). This proves validity of the Simulator Design.

33
Summary ► ► Value of E (Θ) for software with multiple failure regions spread across multiple sub- domains can be calculated using: E (Θ) = q – ( dAF1/M + dBF2/M) q – ( dAF1/M + dBF2/M + dCF3/M) E (Θ) = q – ( dAF1/M + dBF2/M + dCF3/M)

34
Future Work ► The number of test-cases per sub-domain was limited to only 1 ► We would like to generalize the present formula further by considering the effect of ‘n’ (multiple) test cases per sub-domain. ► No generalization doesn’t merely involve putting ‘T’ as a exponential term in the equation ( have tried that )

Similar presentations

Presentation is loading. Please wait....

OK

1841f06detprob4 Testing Basics Detection probability.

1841f06detprob4 Testing Basics Detection probability.

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google

Ppt on non agricultural activities youtube Ppt on human chromosomes map Ppt on types of forest found in india Download ppt on types of houses Download ppt on turbo generator power Ppt on email etiquettes presentation folder Ppt on state of indian economy Ppt on 3 phase ac circuits Ppt on network layer design issues Ppt on cloud computing architecture