Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 Software Fault Tolerance (SWFT) Software Testing Dependable Embedded Systems & SW Group Prof. Neeraj Suri Brahim.

Similar presentations

Presentation on theme: "1 Software Fault Tolerance (SWFT) Software Testing Dependable Embedded Systems & SW Group Prof. Neeraj Suri Brahim."— Presentation transcript:

1 1 Software Fault Tolerance (SWFT) Software Testing Dependable Embedded Systems & SW Group Prof. Neeraj Suri Brahim Ayari Dept. of Computer Science TU Darmstadt, Germany

2 2 Fault Removal: Software Testing  So far: checkpointing, recovery blocks, NVP, NCP, microreboots …  But how do we establish that a SW  actually work according to its specification, and that  the desired levels of dependability are achieved?  Verification & Validation  Testing Techniques  Static vs. Dynamic  Black-box vs. White-box  Next lectures: Testing of dependable systems  Modeling, fault-injection, tools, …

3 3 Definition of Software Testing  Software testing is the process of executing a SW with the intent of finding errors  Software is tested to increase its quality or reliability  Raising the reliability of the software means finding and removing errors  So: Don’t test a software to show that it works; rather, you should start with the assumption that the software contains errors (a valid assumption for almost any software) and then test it to find as many of the errors as possible

4 4 Can we find all errors in a software?  A small Software should be tested (represented by a control-flow graph)  The software consists of a loop that iterates up to 20 times  Assuming that all decisions (if- statements) are independent, the number of all possible runs is 5 1 + 5 2 +... + 5 18 + 5 19 + 5 20 = 1,19209E+14  Assuming 100.000 tests per second, we need 38 years to test all possible runs of this trivial program A B

5 5 Software testing principles (1)  Principle 1: A necessary part of a test case is a definition of the expected output or result  Principle 2: A programmer should avoid attempting to test his or her own piece of software or program  Principle 3: A programming organization should not test its own software or programs  Principle 4: Thoroughly inspect the results of each test  Principle 5: Test cases must be written for input conditions that are invalid and unexpected, as well as for those that are valid and expected

6 6 Software testing principles (2)  Principle 6: Examining a software to see if it does not do what it is supposed to do is only half of the battle; the other half is seeing whether the program does what it is not supposed to do  Principle 7: Avoid throwaway test cases unless the software is truly a throwaway software  Principle 8: Do not plan a testing effort under the tacit assumption that no errors will be found

7 7 Software testing principles (3)  Principle 9: The probability of the existence of more errors in a section of a software is proportional to the number of errors already found in this section  Principle 10: Testing is an extremely creative and intellectually challenging task Number of errors found already Probability of the existence of additional errors

8 8 V&V [Correctness Basis]  Verification  the process of ensuring that the software/protocol/system satisfy the design specifications stipulated for them [i.e., the system is designed right]  Validation  the process of establishing that the software/protocol/system operationally satisfies the design requirements [i.e., the right system is designed]

9 9 (Some) Flavors of V&V  Design reviews  Analytical completeness/correctness  Formal Methods: mathematical and logic techniques for inference, deduction, exploration  Testing! [The Art of Software Testing, Glenford Meyers, 2 nd Edition 2004]  Static testing: [at the level of specifications – pre-code – code]  Dynamic testing [code, binary levels] Black-box White-box  Testing Levels: – – Unit Test: small code fragments – – Module Test: API level – – Subsystem → System Integration Test: testing groups of modules – – Acceptance Test: testing complete integration of modules – – Beta Test: Letting the users test it

10 10 Static Testing (1)  Static testing ≈ “testing without executing the program”  Can be performed before the code is implemented  UML, API’s, etc  Can be used for other things than code  Design documents, manuals, etc  Examples:  Inspections  Walkthroughs  Reviews  Compile-time checks

11 11 Static Testing (2)  Why static testing?  It is generally recognized that the earlier errors are found, the lower the costs of correcting the errors and the higher the probability of correcting them correctly  It is also much harder to find the error in the last stages of the software development process than in the earlier stages  Programmers seem to experience a psychological change when computer-based testing starts  A tendency to want to fix the software bugs as soon as possible introduces a pressure on programmers  Because of this pressure programmers tend to make more mistakes when correcting an error found during computer- based testing

12 12 Inspections and Walkthroughs  Code inspections and walkthroughs are performed by humans reading or visually inspecting a part of a software (can be requirements, design, code, …)  Find errors early  Finds the correct location of the error  Does not focus on the solution!  The author of the program is only a part of the testing team  These methods are effective in finding from 30 to 70% of logic-design and coding errors  Inspections/walkthroughs and computer-based testing are complementary with respect to error-detection efficiency

13 13 Code Inspections  Code inspections  ~Four people meet Programmers, Designers, Test specialists One moderator and THE Programmer  The programmer narrates the code  Use of checklists  An error checklist for inspections include  Data reference errors  Data declaration errors  Computation errors  Comparison errors  Control flow errors  Interface errors  Input/output errors  …

14 14 Walkthroughs  Walkthroughs are similar to inspections  But, a different error-detection technique  The team ”acts” as computers instead of using error checklists  Needs test cases (representative sets of inputs and expected outputs of the software part to test)  During the meeting each test case is mentally executed  The test data are walked through the program  Test cases serve as a vehicle for questioning the programmer about his or her logic and assumptions

15 15 Typical errors to test for…  Control flow/ State transitions  Assignments (incorrect, missing)  Race conditions (timing, serialization, initialization of resources, …)  Load conditions (apps, proc/memory usage, error dist.)  Boundary conditions  Functions (incorrect or missing functionality)  Error handling (missed error check, lack of detector sensitivity, …)  Hardware interfacing (wrong/limited memory declarations, initialization…)  User interfaces! (interactions across components, modules, fragments …)  … + testing the run-time assertions!!!

16 16 Static Analysis of SW Code  Static analysis  Checks for code violations  Builds a model of the code and simulates execution  May not find all bugs, but tricky ones …  Things to look for:  malloc/free – new/delete  Buffer overflows  API consistency  Resource leakage  Boundary conditions  Initialization of variables  …  Good news: There are tools for this (including compilers)

17 17 (Dynamic) Functional Testing  Specification Driven Coverage [Black-Box]  Testing every functional specification  Conformance to specifications  Test data are derived solely from the specification  Correctness  Boundary conditions  Special values  Explicit and implicit requirements – error codes & exceptions?  …  Methods used to generate test cases  Equivalence partitioning  Boundary-value analysis  Cause-effect graphing

18 18 Black-Box Testing  Equivalence partitioning  Partition the input domain of a SW or program into a finite number of equivalence classes  Assume that the test of a representative value of each class is equivalent to a test of any other value  Two types of equivalence classes are identified 1. 1.Valid equivalence classes represent valid inputs to the SW or program 2. 2.Invalid equivalence classes represent all other possible erronieous input values  Test-case design by equivalence partitioning proceeds in two steps: 1.Identifying the equivalence classes from the specification 2.Defining the test cases

19 19 Black-Box Testing (2)  Boundary-value analysis  Experience shows that test cases that explore boundary conditions have a higher payoff than others that do not  Select more than one element from each equivalence class such that each edge of the equivalence class is the subject of a test  Test cases are derived by considering not only input space but also result space (output equivalence class)  Examples  If the valid domain of an input value is -1.0 ― +1.0, write test cases for the situations -1.0, 1.0, -1.001, and 1.001  If an input file can contain 1 ― 255 records, write test cases for 0, 1, 255 and 256 records  If the output of a program should be between 0.0 and 112.25, write test cases that cause 0.0 and 112.25 and invent test cases that causes negative output or a value greater than 112.25

20 20 Black-Box Testing (3)  Cause-effect graphing  One weakness of boundary-value analysis and equivalence partitioning is that they do not explore combinations of input circumstances  The specification is divided into workable pieces  The causes and effects in the specification are identified. Causes are distinct input conditions or input equivalence classes. Effects are output conditions or a system transformation (effect on the state of a program or system e.g. database alteration caused by a transaction)  The semantic content of the specification is analyzed and transformed into a Boolean graph linking causes and effects  The graph is annotated with constraints describing combinations of causes and/or effects that are impossible  The graph is converted into a limited-entry decision table by tracing state conditions in the graph  The columns in the decision table are converted into test cases

21 21 Black-Box Testing (4)  Example of cause-effect graphing  Specification: The character in column 1 must be an “A” or a “B”. The character in column 2 must be a digit. In this situation the file update is made. If the first character is incorrect, message X12 is issued. If the second character is not a digit, message X13 is issued  The causes are 1 ― character in column 1 is “A” 2 ― character in column 1 is “B” 3 ― character in column 2 is a digit  And the effects are 70 ― update made 71 ― message X12 is issued 72 ― message X13 is issued  Note that causes 1 and 2 can not be simultaneously present

22 22 Black-Box Testing (5)  Example of cause-effect graphing (cont.) 1 2 3 1 7171 7070 7272 E

23 23 (Dynamic) Structural Testing  Implementation Driven Coverage [White-Box] (Testing every code fragment!)  All paths, branches and states [HW: op. states]  All program statements  All data dependencies  All exceptions  All inputs!  Correctness & Transitions  Stability  Performance

24 24 White-Box Testing  Example public void foo (int a, int b, int x) { if (a>1 && b==0) { x=x/a; } if (a==2 || x>1) { x=x+1; }

25 25 White-Box Testing (2)  Examples of test cases for  Statement Coverage a = 2, b = 0 and x = 3  Decision coverage or branch coverage a = 2, b = 0 and x = 4 a = 2, b = 1 and x = 1 a = 1, b = 0 and x = 2 a = 1, b = 1 and x = 1  Path coverage In addition to the test cases for branch coverage a = 3, b = 0 and x = 1 a > 1 a = 2 x > 1 b = 0 x = x/a x = x + 1 Y Y Y Y N N N N

26 26 White-Box Testing (3)  State Coverage  Example: The state of a program is defined by the values of 2 integer variables A and B (where 0 { "@context": "", "@type": "ImageObject", "contentUrl": "", "name": "26 White-Box Testing (3)  State Coverage  Example: The state of a program is defined by the values of 2 integer variables A and B (where 0

27 27 (Dynamic) Run-Time  Memory leaks  Data flow  Load Testing [events, CPU/memory usage etc]  Stability [apps, data streams, …]

28 28 (Dynamic) Robustness Checks  Code mutation checking (inject code defects, see if it is caught over testing)  Boundary conditions  Error streams  …

29 29 H. Chu: Univ. of Newcastle SW O/P = Generate Test Data Result Generate Expected Results Specs structural functional success? Dynamic SW Testing

30 30 How do we generate the tests?  Test level identification  Test function identification  Dependability considerations  What are the realistic limits of testing?  When do we consider the system to be tested “enough”?

31 31 Testing Why do we test (dependable systems)?  To determine and remove design deficiencies (bugs?)  To establish operational limits (fault-free and stress scenarios) In SW, we test to find bugs, but…  The presence of bugs usually just indicates that more bugs exist   We can never find all the bugs, and  Even if we never find a bug, that’s great – though it does not prove the absence of bugs   Basically via testing we just build a level of assurance for a desired operational requirement!

32 32 Testing To do effective testing we need to know  Where to test?  Which modules/classes/functions etc  What to test?  Integration  Functional  Robustness  How to test?  Static/dynamic, black/white  The expected result?  Specifications, requirements  When to stop?  When have we tested sufficiently?  100% Coverage?

33 33 The Goal of Testing?  To obtain “complete coverage” : 100% coverage! “complete” or “coverage” w.r.t what? at what level?  all program paths?  all viable exceptional conditions? all faults?  all data streams and sequences?  all transistions and states? Coverage is nothing more than a measure of testing done (of a specific type at a specific level of abstraction)  check out:

34 34 Coverage (1) 1.Line coverage. Test every line of code (Or Statement coverage: test every statement). 2.Branch coverage. Test every line, and every branch on multi-branch lines. 3.N-length sub-path coverage. Test every sub-path through the program of length N. For example, in a 10,000 line program, test every possible 10-line sequence of execution. 4.Path coverage. Test every path through the program, from entry to exit. The number of paths is impossibly large to test. 5.Multicondition or predicate coverage. Force every logical operand to take every possible value. Two different conditions within the same test may result in the same branch, and so branch coverage would only require the testing of one of them. 6.Trigger every assertion check in the program. Use impossible data if necessary. 7.Loop coverage. "Detect bugs that exhibit themselves only when a loop is executed more than once." 8.Every module, object, component, tool, subsystem, etc. This seems obvious until you realize that many programs rely on off-the-shelf components. The programming staff doesn't have the source code to these components, so measuring line coverage is impossible. At a minimum (which is what is measured here), you need a list of all these components and test cases that exercise each one at least once.

35 35 Coverage (2) 9.Fuzzy decision coverage. If the program makes heuristically-based or similarity-based decisions, and uses comparison rules or data sets that evolve over time, check every rule several times over the course of training. 10.Relational coverage. "Checks whether the subsystem has been exercised in a way that tends to detect off-by-one errors" such as errors caused by using < instead of <=. This coverage includes: 11.Every boundary on every input variable. 12.Every boundary on every output variable. 13.Every boundary on every variable used in intermediate calculations. © C. Kaner,

36 36 Dependable Systems  Apart from fundamental correctness, how do we establish the level of dependability achieved?  If a ultra-dependable system never fails, then how do we establish its failure rate?

Download ppt "1 Software Fault Tolerance (SWFT) Software Testing Dependable Embedded Systems & SW Group Prof. Neeraj Suri Brahim."

Similar presentations

Ads by Google