1. Testing Overview Software Reliability Techniques Testing Concepts CEN 4010 Class 24 – 11/17

2. CEN 4010 Class 24 - 11/172 Overview of Testing - Terminology Software testing is the process of operating software under specified conditions, observing or recording the results and making an evaluation of some aspect of the software. Software – all artifacts generated during the development process. (IEEE/ANSI std 610.12-1990)

3. CEN 4010 Class 24 - 11/173 Terminology cont Software reliability is the probability that a s/w system will not cause the failure of the system for specified time under specified conditions (IEEE Std 982-1989). Failure – is the manifested inability of a system or component to perform a required function within specified limits, i.e., incorrect output or abnormal termination of a program. Fault (defect or bug) – is the mechanical or algorithmic cause of a failure. Error – a human action that produces a software fault. (Binder 2000)

4. CEN 4010 Class 24 - 11/174 Terminology cont Test case – is a set of inputs and expected results that exercises a component with the purpose of causing failures and detecting faults. Test stub – a partial implementation of a component on which the tested component depends. Test driver – is a partial implementation of a component that depends on the tested component. i.e., a test driver simulates the tested component’s environment.

5. CEN 4010 Class 24 - 11/175 Software Reliability Techniques There are many techniques for increasing the reliability of a software system: –Fault avoidance –Fault detection –Fault tolerance

6. CEN 4010 Class 24 - 11/176 Fault Avoidance Techniques Fault avoidance techniques try to detect faults statically i.e., without relying on the execution of system models, in particular source code. Fault Avoidance Techniques: 1.Development methodologies 2.Configuration management 3.Verification techniques

7. CEN 4010 Class 24 - 11/177 Fault Avoidance Techniques cont 1.Development methodologies –unambiguous representation of requirements, –the use of data abstraction and data encapsulation, –minimizing the couple between subsystems and maximizing cohesion, –the early definition of subsystem interfaces, –capture of rationale for maintenance activities. Traceability of the use case model i.e., the use case driven approach to software development.

8. CEN 4010 Class 24 - 11/178 Fault Avoidance Techniques cont 2.Configuration management –Avoids faults caused by undisciplined change in the system models. 3.Verification –Attempts to find faults before any execution of the system. – Not in a mature enough state that it can be applied to assure the quality of large complex systems. –Assumes requirements are correct - used to create pre- and post-conditions

9. CEN 4010 Class 24 - 11/179 Fault Avoidance Techniques cont –Verifies an operation by showing that if the precondition is true before the operation and postcondition is true after the operation then the operation is correct. –Sometimes it is hard to correctly state the pre- and post-conditions.

10. CEN 4010 Class 24 - 11/1710 Fault Detection Techniques Fault detection techniques e.g., debugging and testing, are used during the development process to identify erroneous states and find the underlying faults before releasing the system. Fault Detection Techniques: 1.Reviews 2.Testing 3.Debugging

11. CEN 4010 Class 24 - 11/1711 Fault Detection Techniques cont 1.Reviews –Manual inspection of parts of all aspects of the system without actually executing the system. –Walkthrough: developer informally presents the API, the code and associated documentation of the component to the review team. –Inspection is a formal presentation of a walkthrough by a review team, not the developer.

12. CEN 4010 Class 24 - 11/1712 Fault Detection Techniques cont 2.Testing –A fault detection technique that tries to create failures or errors in a planned way. Note a successful test is one that identifies errors i.e., testing demonstrates the presence of errors not their absence. –The characteristics of a good test model is that it contains test cases that identify errors.

13. CEN 4010 Class 24 - 11/1713 Fault Detection Techniques cont –Testing Activities: Test Planning allocates resources and schedules the testing activity. Note this activity should occur early in the development phase so that sufficient time and skill is dedicated to the testing activity. Usability testing tries to find faults in the user interface design of the system. Unit testing tries to find faults in participating objects and/or subsystems with respect to (w.r.t.) the use cases from the use case model.

14. CEN 4010 Class 24 - 11/1714 Fault Detection Techniques cont –Testing Activities: Integration testing tries to find faults when testing the individually tested components e.g.,, testing subsystems. This is driven by the subsystem decomposition in the DD and SRD. System testing tests all the components together, seen as a single system to identify errors w.r.t. the scenarios from the problem statement and the goals in the SRD and DD.

15. CEN 4010 Class 24 - 11/1715 Fault Detection Techniques cont –Types of System testing: Functional testing test the requirements from the SRD, and if available from the user manual. Performance testing checks nonfunctional requirements and additional goals form the DD. Acceptance testing and installation testing check the requirements against the project agreement (SRD) and is done by the client.

16. CEN 4010 Class 24 - 11/1716 Fault Tolerance Techniques Fault tolerance techniques assume that a system can be released with faults and that system failures can be dealt with by recovering from them at runtime. Allows a system to recover from failure of a component by passing the erroneous state information back to the calling components, assuming one of them knows how to handle it.

17. CEN 4010 Class 24 - 11/1717 Fault Tolerance Techniques cont Fault Tolerance Techniques: –Modular redundancy in h/w e.g. 5 onboard computers in shuttle. –N-version programming – same spec. implemented by different teams. –Recovery blocks – each block includes a test code and alternative code.

18. CEN 4010 Class 24 - 11/1718 Testing Concepts Test Case components: 1.Name – identifies the test case, it is a good idea to derive the name from the requirement being tested. E.g., SSLS01_Test 1 2.Purpose – states the purpose of the test and relates it to the requirement (or scenario). 3.Test set up – describe the h/w and s/w and environment required for a successful test. 4.Input – description of the input data or commands. 5.Expected output (or Oracle) – expected test results against which the output of the test is compared. 6.Actual output (or log) – output produced by the test.

19. CEN 4010 Class 24 - 11/1719 Testing Concepts cont Note if the expected output and actual output differ then a application failure has been detected. We say that the test has revealed a bug. Debugging is the work required to diagnose and correct a bug. Note debugging is not testing. Debugging typically occurs after a failure has been observed or when a developer identifies a fault by inspection or automatic code analysis.

20. CEN 4010 Class 24 - 11/1720 Testing Concepts cont Note: if you submit the code for your project that is not bug-free it should be highlighted in the documentation for the test cases. You should explain the discrepancies. Test cases are classified depending on which aspect of the system model is tested: –Blackbox (specification-based or functional) tests focus on the input/output behavior (or functionality) of the component. Tests do not deal with the internal aspects of the component nor with the behavior or the structure of the components.

21. CEN 4010 Class 24 - 11/1721 Testing Concepts cont –Whitebox (structural or implementation-based) tests focus on the internal structure of the component. That is, test cases are constructed based on the code that implements the software. A correction is a change to a component whose purpose is to repair a fault. Regression testing includes the re-execution of prior tests after a change, this ensures that the functionality that worked before the change has not been affected.

22. CEN 4010 Class 24 - 11/1722 Homework Write a paragraph describing each of the following unit testing techniques: 1.boundary testing, 2.path testing, 3.state-based testing, 4.data flow testing, 5.branch coverage testing, Provide one example for each using data from your project. For each technique identify if it is a specification- based or implementation-based testing technique.