1. Software Reliability Engineering: A Roadmap
Future of Software Engineering
ICSE’2007
Minneapolis, Minnesota
May 24, 2007
Software Reliability Engineering: A Roadmap
Michael R. Lyu
Dept. of Computer Science & Engineering
The Chinese University of Hong Kong

2. Introduction
Software reliability is the probability of failure-free operation with respect to execution time and environment.
Software reliability engineering (SRE) is the quantitative study of the operational behavior of software-based systems with respect to user requirements concerning reliability.
SRE has been adopted by more than 50 companies as standards or best current practices.
Creditable software reliability techniques are still in urgent need.

3. Historical SRE Techniques: Fault Lifecycle
Fault prevention: to avoid, by construction, fault occurrences.
Fault removal: to detect, by verification and validation, the existence of faults and eliminate them.
Fault tolerance: to provide, by redundancy and diversity, service complying with the specification in spite of manifested faults.
Fault/failure forecasting: to estimate, by statistical modeling, the presence of faults and occurrence of failures.

4. Fault Lifecycle Technique
Fault Manifestation and Modeling Process
Reliability
Fault Prevention
Fault Removal
Fault Tolerance
Fault/Failure Forecasting

5. Fault Lifecycle Technique
Fault Manifestation and Modeling Process
Reliability
Availability
Safety
Security
Fault Prevention
Fault Removal
Fault Tolerance
Fault/Failure Forecasting

6. Software Reliability Modeling
R = e -t
Testing Time

7. Current SRE Process Overview

8. Current Trends and Problems
The theoretical foundation of software reliability comes from hardware reliability techniques.
Software failures do not happen independently.
Software failures seldom repeat in exactly the same or predictable pattern.
Failure mode and effect analysis (FMEA) for software is still controversial and incomplete.
There is currently a need for a creditable end-to-end software reliability paradigm that can be directly linked to reliability prediction from the very beginning.

9. Future Direction 1: Reliability-Centric Software Architectures
The product view – achieve failure-resilient software architecture
Fault prevention
Fault tolerance
The process view – explore the component-based software engineering
Component identification, construction, protection, integration and interaction
Reliability modeling based on software structure

10. Future Direction 2: Design for Reliability Achievement
Fault confinement
Fault detection
Diagnosis
Reconfiguration
Recovery
Restart
Repair
Reintegration

11. Fault Confinement Fault Detection Failover Diagnosis Online Offline
Reconfiguration
Recovery
Restart
Repair
Reintegration
1. Fault confinement. This stage limits the spread of fault effects to one area of the Web service, thus preventing contamination of other areas. Fault-confinement can be achieved through use of: fault-detection within the Web services, consistency checks and multiple requests/confirmations.
2. Fault detection. This stage recognizes that something unexpected has occurred in the Web services. Fault latency is the period of time between the occurrence of a fault and its detection. Techniques fall in 2 classes: off-line and on-line. With off-line techniques, such as diagnostic programs, the service is not able to perform useful work while under test. On-line techniques, such as duplication, provide a real-time detection capability that is performed concurrently with useful work.
3. Diagnosis. This stage is necessary if the fault detection technique does not provide information about the failure location and/or properties.
4. Reconfiguration. This stage occurs when a fault is detected and a permanent failure is located. The Web services can be composed of different components. When providing the service, there may be failure in individual components. The system may reconfigure its components either to replace the failed component or to isolate it from the rest of the system.
5. Recovery. This stage utilizes techniques to eliminate the effects of faults. Two basic recovery approaches are based on: fault masking, retry and rollback. Fault-masking techniques hide the effects of failures by allowing redundant information to outweigh the incorrect information. Web services can be replicated or implemented with different versions (NVP). Retry attempts a second attempt at an operation and is based on the premise that many faults are transient in nature. Web services provide services through network; retry would be a practical as requests/reply may be affected by the situation of the network. Rollback makes use of the fact that the Web service operation is backed up (checkpointed) to some point in its processing prior to fault detection and operation recommences from this point. Fault latency is important here because the rollback must go back far enough to avoid the effects of undetected errors that occurred before the detected error.
6. Restart. This stage occurs after the recovery of undamaged information.
l           Hot restart: resumption of all operations from the point of fault detection and is possible only if no damage has occurred.
l           Warm restart: only some of the processes can be resumed without loss.
l           Cold restart: complete reload of the system with no processes surviving. The Web services can be restarted by rebooting the server.
7. Repair. At this stage, a failed component is replaced. Repair can be off-line or on-line. Web services can be component-based and consist of other Web services In off-line repair either the Web service will continue if the failed component/sub-Web service is not necessary for operation or the Web services must be brought down to perform the repair. In on-line repair the component/sub-Web service may be replaced immediately with a backup spare or operation may continue without the component. With on-line repair Web service operation is not interrupted.
8. Reintegration. In this stage the repaired module must be reintegrated into the Web service. For on-line repair, reintegration must be performed without interrupting Web service operation.

12. Future Direction 3: Testing for Reliability Assessment
Establish the link between software testing and reliability
Study the effect of code coverage to fault coverage
Evaluate impact of reliability by various testing metrics
Assess competing testing schemes quantitatively

13. Positive vs. negative evidences for coverage-based software testing
Resources
Findings
Positive
Frankl(1988)
Horgan(1994)
Weyuker(1988)
High code coverage brings high software reliability and low failure rate
Chen(1992)
A correlation between code coverage and software reliability
is observed
Wong(1994)
The correlation between test effectiveness and block coverage is higher than that between test effectiveness and the size of test set
Frate(1995)
An increase in reliability comes with an increase in at least one code coverage measures
Cai (2005)
Code coverage contributes to a noticeable amount of fault coverage
Negative
Briand(2000)
The testing result on published data did not support a causal
dependency between code coverage and defect coverage

14. RSDIMU test cases descriptionII
III IV V VI
This is the descriptions of test set, containing the detailed testing purpose of each test case.
Can be classified as functional testing (1-800) and random testing ( ).
Can be classified into six regions according to their different patterns.

15. The correlation: various test regions
Linear regression relationship between block coverage and fault coverage in the whole test set
Linear modeling fitness in various test case regions
Fault Coverage
For overall, moderate; highest, region IV, lowest: region VI

16. The correlation: normal operational testing vs. exceptional testing
Testing profile (size)
R-square
Whole test case (1200)
0.781
Normal testing (827)
0.045
Exceptional testing (373)
0.944
Normal operational testing
very weak correlation
Exceptional testing
strong correlation

17. The correlation: normal operational testing vs. exceptional testing
Normal testing: small coverage range (48%-52%)
Exceptional testing: two main clusters
Fault Coverage
Fault Coverage
Normal testing; code coverage (48%-52%) main control flow/data flow
Exceptional testing: two clusters . The reason is in some cases, part of large-scale computational functions are executed but others will be skipped. But in other cases, all these computational code are skipped.

18. The Spectrum in Software Testing and Reliability
Time
Based
Models
Coverage
Testing
- user oriented tester oriented
- more physical meaning less physical meaning
- abundant models lack of models
- easy data collection hard data collection
- less relevance to testing more relevance to testing
New Model
Software Reliability
Growth Models
Coverage-Based
Analysis
A new model is needed to combine execution time and testing coverage

19. A New Coverage-Based Reliability Model
λ(t,c): joint failure intensity function
λ1(t): failure intensity function with respect to time
λ2(c): failure intensity function with respect to coverage
α1,γ1, α2, γ2: parameters with the constraint of
α 1 + α 2 = 1
joint failure intensity function
failure intensity function with time
failure intensity function with coverage
Dependency factors
In`tegral
De`rivative
Since lambda(t) is the failure intensity function with respect to time, any existing distributions in well-known reliability models can be used, e.g., NHPP,Weibull model,S-shaped model or logarithmic Poisson models.

20. Estimation Accuracy
NHPP model

21. Future Direction 4: Metrics for Reliability Prediction
New models (e.g., BBN) to explore rich software metrics
Data mining approaches
Machine learning techniques
Bridging the gap of the one-way function: feedback to building reliable software
Continuous industrial data collection efforts – demonstration of cost-effectiveness

22. Future Direction 5: Reliability for Emerging Software Applications
“The Internet changes everything”
On-demand customizable software
Service oriented architecture, composition, integration
Customization by middleware – from metadata to metacode
A common infrastructure delivers reliability to all customers

23. A Paradigm for Reliable Web Service
Replication Manager
Web service
selection
algorithm
WatchDog
UDDI
Registry
WSDL
Web Service
IIS
Application
Database
Client
Port
Create Web services
Select primary Web
service (PWS)
3. Register
4. Look up
5. Get WSDL
6. Invoke Web service
Keep check the availability of the PWS
If PWS failed, reselect the PWS.
9. Update the WSDL

24. Conclusions
Software reliability is receiving higher attention as it becomes an important economic consideration for businesses.
New SRE paradigms need to consider software architectures, testing techniques, data analyses, and creditable reliability modeling procedures.
Domain specific approaches on emerging software applications are worthy of investigation.
Still a long way to go, but the directions are clear.