1. Rescaling Reliability Bounds for a New Operational Profile Peter G Bishop pgb@adelard.com pgb@csr.city.ac.uk Adelard, Drysdale Building, Northampton Square, London EC1V 0HB +44 20 7490 9450 www.adelard.com

2. 2 Outline of Talk l Original reliability bound theory (same op. profile) l Extended theory (different operational profile) l Implications of the theory l Experimental evaluation

3. 3 Original Theory Operational profile (I) Defect Observed defect failure frequency    D 1 D 2 D 3 Input Domain

4. 4 Theory assumptions the operational profile is invariant, i.e.  s are constant over time l when a failure occurs the associated defect is immediately and perfectly corrected removal of a defect does not affect the s of the remaining defects

5. 5 Basic idea Given some test interval t : Defects with large s will be removed already Defects with small s will remain - but have little affect on program reliability So there must be an “worst case” for a defect that maximises the program failure rate after t

6. 6 Worst-case bound Original paper showed that, given the assumptions, max failure /unit time for a defect i is:  i | t  1/et (where t is the test time) So if there are N faults in the program the failure rate at time t is bounded by:  | t  N/et

7. 7 Bound is independent of 0.00001 0.0001 0.001 0.01 0.1 1 110100100010000 t Probablity of failure  | t =0.1 =0.01 =0.001 1/et

8. 8 Refinement for discrete tests l For for a discrete sequence of T tests the result is:  |T  N (T/T+1) T /(T+1)  N/(eT) (conservative approx.) l So it is conservative to use original equation.

9. 9 Limitations Assumes operational profile I is constant hence s are constant l But we know that in practice the profile changes. l So the reliability bound does not apply if the operational profile changes l (e.g. from system test to actual use) l but will “settle back” in long term if new profile stable l New theory gives a means for “rescaling” the reliability bound for a different profile

10. 10 Additional assumptions l Each defect is localised to a single code “block” The operational profile I can be characterised by the distribution of code block executions Q in the program { q(1), q(2), … } The failure rate of defect in block,  i  q(i) l There is a constant probability of a fault existing in any line of executable code.

11. 11 Rescaling for known defect l For a defect i in code block j, the re-scaled bound would be: where q’(j) is the new execution rate and q(j) is the old execution rate.

12. 12 Probability of defect in block We do not know which block contains defect i, but we assume that the chance of being in j is: L(j)/L where L(j) is the length of the code block, and L is the total length of the executable code.

13. 13 Re-scaled bound l Taking the average over all blocks: l So the “scale factor” relative to the original bound is: l Also true if there are N faults rather than 1   L jL jq ( jq)( )( )

14. 14 Theory predictions - Fair testing l If q  L of blocks “dominated” by decision branch, scale factor unchanged by any other profile l Applies to any acyclic graph, l And subgraphs with fixed iteration loops Segment j L(j). q(j) q ’ (j) L(j)q ’ (j) q(j) Root 0 1011 Branch 1 100.10.990 Branch 2 900.90.110 Sum 110 S =Sum/L1

15. 15 l Use of “unbalanced” test profile can be very sensitive to changes in profile l Factor can be less than 1 if under-tested blocks avoided, e.g. Q’={1,1,0} gives S = 0.19 Unfair testing Segment j L(j) q(j) Root 0 1011 Branch 1 100.90.1 1.1 Branch 2 900.10.9 810 Sum 110 829 S =Sum/L7.5 q ’ (j) L q ’ /q

16. 16 Limits to fair test approach l Fair test apportionment does not work for variable loops, recursion and subroutines l Even if we identify a fair test profile, it may be infeasible to execute Decisions not independent (shared variable)

17. 17 Maximum scale factor l If we know max. possible execution rates for each block, can estimate a “maximum scale factor”:  ( q(k) max / q(k) ) (L(k) / L) l Where k relates to a worst case “thread” through the graph. Hard to identify this thread, but easier to compute a more pessimistic factor:  ( q(j) max / q(j) ) (L(j) / L) where j includes all blocks. l No knowledge of the new profile is needed

18. 18 Including module tests l Can combine module tests and system tests, composite scale factor is: where x(j) are the total executions under module testing l Module tests can “fill in” uncovered segments that would make the test profile “unbalanced” Tjxjq jq L jL /)()( )()(   

19. 19 Experimental evaluation l Use programs with known set of defects l PODS l simple reactor trip application (<1000 code lines) l simple structure, fixed loops l PREPRO l ~ 10 000 code lines l parses input description file of indefinite length l recursive - max execution unknown l Similar results - will only discuss PODS here

20. 20 PODS evaluation l Measure Q for different test profiles l Uniform, Normal, Inverse normal - “bathtub” Measure defect failure rates (i) under all profiles Predict residual failure rate:  (i) exp(- (i)T) Compute failure rate for new profile:  ’ (i) exp(- (i)T) Compare with scaled bound:  ( L(j)/L)(q ’ (j)/q(j))N/eT

21. 21 Variation in q(j)

22. 22 Predicted scale factors Operational profile Test profile uniforminv-normal normal uniform 1 1.20.9 inv-normal 3.2 16.2 normal115 3461 l Note the predicted reduction in bound

23. 23 Maximum scale factor Test profileMax scale-up factor uniform6.6 inv-normal10.0 normal1059 l 2-5 times worst than bound with a known profile l Can be over-pessimistic l But could indicate relative sensitivity to change

24. 24 “Unfair” Normal test profile N/et bound Scaled bound Max bound

25. 25 “Fairer” Uniform test profile N/et bound Scaled bound Max bound

26. 26 PREPRO l Similar results l changes in failure rates are within the scaled bounds l But could not compute a maximum bound l program is recursive l so no upper bound on the execution of program code blocks

27. 27 Summary Theory suggests: l Can rescale bound (knowing Q and Q’) l Can include module test execution information l Can compute max scale up (knowing Q and Q max ) l For some program structures can identify a totally "fair" test profile - bound insensitive to change The experimental evaluations appear to be consistent with the predictions of the theory

28. 28 Conclusions l Could affect approach to testing: l “fairer” test profiles rather than realistic profiles l integrated module and system test strategy l Could improve reliability bound prediction for new environment l Could assess sensitivity to profile change l e.g. by computing maximum scale factor l But based on quite strong assumptions, need to: l validate assumptions l assess impact of assumption violation l evaluate on more examples

29. 29