1. Presented by: Hatem Halaoui
Evaluation of Safety Critical Software by D. Parnas, A. J. van Schouwen, and P. Kwan
Presented by:
Hatem Halaoui

2. Presentation Outline Paper Uncovered issues
Introduction
Why is software used?
Software controllers and other controllers
Software concerns
Modular Structure
Reliability Assessment for Safety-critical software
Conclusions
Uncovered issues
Statistical software crisis information due to bad software engineering

3. Introduction
The failure of programmable computer applications could be life threatening. Computers now have safety-critical functions.
To reduce application failures these questions need some answers:
what standards must a software product satisfy?
What documentation is needed?
How much testing is needed?
How should software be structured?

4. Why is software a special concern?
Many tragedies were caused by software failures.
Software systems do not work well until they have been used and have failed repeatedly
Software products may fail in their first use because situations that were anticipated by the programmers were also overlooked by the test planners
The use of randomly generated data reduces the likelihood of shared oversight

5. Why is software used?
Software makes it practical to build more logic into the system. They can distinguish a large number of situations and provide suitable outputs.
Logic implemented in software is easier to change than that implemented in hardware
Software provide more information in more suitable form

6. How are software controllers like other controllers?
Similar to other controllers (hardware), software controllers has the following properties:
Inputs can be described as mathematical functions
Outputs can be described as mathematical functions of the inputs
Values of the variables can also be described as mathematical function of the controller output
Relations between variables can be described

7. How are software controllers different from other controllers?
Complexity:
Programs need more precise documentation which might fill a book case
programs need much more time (years ) before being trusted
Error Sensitivity: software is much more sensitive to small errors.
Hard to test:
Huge number of testing cases in software
what about points between the testing points
Correlated failures:
Assumptions for hardware design are invalid for software: like assuming that a hardware failures are not strongly correlated
In contrast to hardware, duplicating software components does not imply higher reliability
Lack of professional standards
No professional software engineers
No agreement on the skills and knowledge that a software engineer should have

8. Software Testing Concerns
We cannot test software for correctness: large number of states
It is difficult to make accurate predictions of software reliability and availability
It is not practical to measure the trustworthiness of software: a software is trustworthy if the probability of having a potentially catastrophic flaw is acceptably low.
There is a role for testing: some scientist argue that one should test rather than spending this time review and does mathematical verifications
There is a need for an independent validation agency: it is difficult to test one’s own design in an unbiased way

9. Software Reviewability Concerns
Why is the reviewability concern for software?
Before, there were not too much care about software, its documentation, and its trustworthiness
Now, much more software controllers in big equipments and industry
Reviews are very important to increase the correctness of software
Engineers are required to do documentations

10. What Reviews are needed
Review for the correct intended function
Review for maintainable, understandable, well documented structure.
Review each module to verify the algorithm and date structure design are consistent with the specified behavior
Review the code for consistency with the algorithm and data structure design.
Review test adequacy: was the testing sufficient to provide sound confidence in the software functioning.

11. What documentation is required to review functional requirements?
Should be done by engineers and scientists who understand the situation to be monitored and devices to be controlled.
The functional requirement can be stated by giving three mathematical functions:
The required values of the controlled variables in terms of values of the relevant observable environmental parameters
The computer inputs in terms of those observable environmental variables
The values of the controlled environmental variables in terms of the computer output
This can be given as a set of tables and formulae

12. What documentation is required to review software structure?
Documents that describe the breakdown of the program into modules (each module is a unit which could be a set of programs).
The purpose of this review is to make sure that:
The structure allow independent development and change
All programs need are included once and only once
Module interfaces are defined
Modules are compatible to each other and meet the requirements
Three types of documents are needed:
Requirements and specifications
Informal document describing responsibilities of each module
Module specification (black box description)

13. What documentation is required to review module’s internal design?
Design documentation which include description of two types of mathematical functions: program functions and abstraction functions
Programs must be described in a hierarchical way where each small program or subprogram is described
This needs software engineers and specialist in the areas that the software is dealing with

14. What documentation is required to review the code?
Algorithms and data structures
The reviewers should be experienced users of the hardware and compilers

15. What documentation is required to review the test plan review?
Mathematical verification of the code
The test plan should be described in a document not available for designers
Random testing is required
Should be reviewed by specialists in software testing and application area

16. Reviewing relationships between these documents
Relationship between these documents should be verified
Main issues about documents relationships
Module guide should show no ambiguity about module’s responsibilities
Each module design document should show arguments that verify the module specifications
The module design document, which describes the algorithms should be mapped on the code
Test plans must show how the tests are derived and how they cover the requirements

17. Why is configuration management essential for rigorous reviews
All documents must be kept consistent when changes made.
If document is changed, all related documents must be reviewed and possibly change.
Designers and reviewers should be notified of changes and have the latest versions of documents

18. Modular Structure
Software should be organized in accordance with the modern principles of “information hiding”, “Object Orientation”, “Separation of Concerns”, “Encapsulation”, “Data Abstraction”, etc.
Large programs should be organized in smaller and simpler assignments known as modules.
Software engineers should be trained to use abstract mathematical specifications
Details that change a lot should be hidden

19. Reliability Assessments for Safety-Critical Software
Some engineers argue that the correctness of a software is not probabilistic
Not true!! Software appears to exhibit stochastic properties
We can never say that a software is 100% reliable

20. What should be measuring?
Encountering the sequence of input that lead to failures
Different kinds of software needs different measuring
Example
In Safety-critical applications errors are not acceptable
We cannot predict the failure rate from failure rates of individual lines or subprograms (a failure in a subprogram might lead into a failure in others)

21. The finite state machine model of programs
Software is a machine that is described by two functions: Next State and Output
Software can be viewed as a finite state machine described by large tables
Loading a program in the machine selects a terminal submachine consisting of all states can be reached from the initial state
It helps to know the number of tests we need to perform

22. Use of hypothesis testing
We need to confirm that failure probability is below upper bound
Assume the upper bound probability of a failure is 1/h => reliability is 1-1/h
For N random tests the probability there will be no failure during testing is M=(1-1/h)N

23. Hypothesis testing example

24. Reliability estimates of programs
Three classes of programs:
Memoryless batch programs (each run is independent of others)
Batch programs with memory (a run might depend on previous one)
Real time programs (run continuously)
Reliability of memoryless batch programs : tests consist of single run for each set of inputs
Reliability of batch programs: test consist of single run of input data and internal state
Reliability of Real time programs:
when testing the concept of inputs must be replaced with a multidimensional trajectory.
Each trajectory gives the input values as continuous functions of time
Length of trajectories should be estimated

25. Conclusions Information hiding should be used
Documentation should be complete, precise and making use of mathematical notations rather than natural language
Mathematical verification techniques must be used
An independent agency should test the software using valid random testing to get estimates of the reliability of the software at critical situations.
Hypothesis testing should be used to allow us to know the probability that the software meet the requirements

26. Uncovered Issues
One of the most important issues that lead into software crisis is the errors resulting from discreatizing continuous data
Such data will not be accurate
Operations applied on multiple of such data will lead to high loss of precision

27. Software Crisis Examples
Source:
Software crisis:
The cost of owning and maintaining software in the 1980’s was twice as expensive as developing the software.
During the 1990’s, the cost of ownership and maintenance increased by 30% over the 1980’s.
In 1995, statistics showed that half of surveyed development projects were operational, but were not considered successful.
The average software project overshoots its schedule by half.
Three quarters of all large software products delivered to the customer are failures that are either not used at all, or do not meet the customer’s requirements.
Main reasons
Software development is seen as a craft, rather than an engineering discipline.
The approach to education taken by most higher education institutions encourages that "craft" mentality.