Presentation is loading. Please wait.

Presentation is loading. Please wait.

Software Verification Introduction & The Model-Driven Test Design Process.

Similar presentations


Presentation on theme: "Software Verification Introduction & The Model-Driven Test Design Process."— Presentation transcript:

1 Software Verification Introduction & The Model-Driven Test Design Process

2 Testing in the 21st Century n Today’s software market : –is much bigger –is more competitive –has more users n Embedded Control Applications –airplanes, air traffic control –spaceships –watches –ovens –remote controllers n increase pressure on testers –Programmers must unit test – with no training, education or tools ! –Tests are key to functional requirements – but who builds those tests ? 2 – PDAs – memory seats – DVD players – garage door openers – cell phones

3 3 Cost of Testing n In the real-world, testing is the principle post- design activity n Restricting early testing usually increases cost n Extensive hardware-software integration requires more testing You’re going to spend at least half of your development budget on testing, whether you want to or not

4 4 Why Test? n What fact is each test trying to verify? If you don’t start planning for each test when the functional requirements are formed, you’ll never know why you’re conducting the test n Not testing is even more expensive Program Managers often say: “Testing is too expensive.”

5 Testing event generationevent evaluation events instrumentationspecification send(42)

6 Test Design in Context n Test Design is the process of designing input values that will effectively test software n Test design is one of several activities for testing software –Most mathematical –Most technically challenging 6

7 The Sieve program int P[101]; int V=1; int N=1; int I; main() { while (N<101) { for (I=1;I<=N;I++) if (P[I]=0) { P[I]=V++ ; N++ ; } else if (V%P[I]==0) I=N+1; else I++} } Is this program correct?

8 Types of Test Activities n Testing can be broken up into four general types of activities 1.Test Design 2.Test Automation 3.Test Execution 4.Test Evaluation n Each type of activity requires different skills, background knowledge, education and training 8 1.a) Criteria-based 1.b) Human-based

9 1. Test Design – (a) Criteria-Based n This is the most technical job in software testing n Test design is analogous to software architecture on the development side 9 Design test values to satisfy coverage criteria or other engineering goal

10 1. Test Design – (b) Human-Based n Requires knowledge of : –Domain, testing, and user interfaces n Requires almost no traditional CS –A background in the domain of the software is essential –An empirical background is very helpful (biology, psychology, …) –A logic background is very helpful (law, philosophy, math, …) 10 Design test values based on domain knowledge of the program and human knowledge of testing

11 2. Test Automation n This is slightly less technical n Requires knowledge of programming –Fairly straightforward programming – small pieces and simple algorithms n Programming is out of reach for many domain experts 11 Embed test values into executable scripts

12 12 What is JUnit? Open source Java testing framework used to write and run repeatable automated tests ( junit.org ) n JUnit features include: –Assertions for testing expected results –Test features for sharing common test data –Test suites for easily organizing and running tests –Graphical and textual test runners n JUnit is widely used in industry n JUnit can be used as stand alone Java programs (from the command line) or within an IDE such as Eclipse

13 3. Test Execution n This is easy – and trivial if the tests are well automated 13 Run tests on the software and record the results 4. Test Evaluation n This is much harder than it may seem Evaluate results of testing, report to developers

14 Types of Test Activities – Summary n These four general test activities are quite different n It is a poor use of resources to use people inappropriately 14 1a.DesignDesign test values to satisfy engineering goals CriteriaRequires knowledge of discrete math, programming and testing 1b.DesignDesign test values from domain knowledge and intuition HumanRequires knowledge of domain, UI, testing 2.AutomationEmbed test values into executable scripts Requires knowledge of scripting 3.ExecutionRun tests on the software and record the results Requires very little knowledge 4.EvaluationEvaluate results of testing, report to developers Requires domain knowledge

15 Model-Driven Test Design 15 software artifact model / structure test requirements refined requirements / test specs input values test cases test scripts test results pass / fail IMPLEMENTATION ABSTRACTION LEVEL DESIGN ABSTRACTION LEVEL test requirements

16 Model-Driven Test Design – Steps 16 software artifact model / structure test requirements refined requirements / test specs input values test cases test scripts test results pass / fail IMPLEMENTATION ABSTRACTION LEVEL DESIGN ABSTRACTION LEVEL analysis criterionrefine generate prefix postfix expected automate execute evaluate test requirements domain analysis feedback

17 Model-Driven Test Design – Activities 17 software artifact model / structure test requirements refined requirements / test specs input values test cases test scripts test results pass / fail IMPLEMENTATION ABSTRACTION LEVEL DESIGN ABSTRACTION LEVEL Test Design Test Execution Test Evaluation

18 How to cover the executions? IF (A>1)&(B=0) THEN X=X/A; END; IF (A=2)|(X>1) THEN X=X+1; END; n Choose values for A,B,X. n Value of X may change, depending on A,B. n What do we want to cover? Paths? Statements? Conditions?

19 Execute every statement at least once By choosing A=2,B=0,X=3 each statement will be chosen. The case that the tests fail is not checked! IF (A>1)&(B=0) THEN X=X/A; END; IF (A=2)|(X>1) THEN X=X+1; END;

20 IMPORTANT TERMS 20

21 21 Validation & Verification n Validation : The process of evaluating software at the end of software development to ensure compliance with intended usage n Verification : The process of determining whether the products of a given phase of the software development process fulfill the requirements established during the previous phase

22 According to Boehm n Verification means “we are building the product right.” n Validation means “we are building the right product”.

23 Verification or validation Unverifiable (but validatable) spec:... if a user presses a request button at floor i, an available elevator must arrive at floor i soon Example: elevator response Verifiable spec:... if a user presses a request button at floor i, an available elevator must arrive at floor i within 30 seconds...

24 24 Software Faults, Errors & Failures n Software Fault : A static defect in the software n Software Failure : External, incorrect behavior with respect to the requirements or other description of the expected behavior n Software Error : An incorrect internal state that is the manifestation of some fault Faults in software are equivalent to design mistakes in hardware. They were there at the beginning and do not “appear” when a part wears out.

25 25 Testing & Debugging n Testing : Finding inputs that cause the software to fail n Debugging : The process of finding a fault given a failure

26 26 Static and Dynamic Testing n Static Testing : Testing without executing the program –This include software inspections and some forms of analyses –Very effective at finding certain kinds of problems – especially “potential” faults, that is, problems that could lead to faults when the program is modified n Dynamic Testing : Testing by executing the program with real inputs

27 27 White-box and Black-box Testing n Black-box testing : Deriving tests from external descriptions of the software, including specifications, requirements, and design n White-box testing : Deriving tests from the source code internals of the software, specifically including branches, individual conditions, and statements n Model-based testing : Deriving tests from a model of the software (such as a UML diagram

28 Stress Testing n Very large numeric values (or very small) n Very long strings, empty strings n Null references n Very large files n Many users making requests at the same time n Invalid values 28 Tests that are at the limit of the software’s expected input domain

29 29 Top-Down and Bottom-Up Testing n Top-Down Testing : Test the main procedure, then go down through procedures it calls, and so on n Bottom-Up Testing : Test the leaves in the tree (procedures that make no calls), and move up to the root. –Each procedure is not tested until all of its children have been tested

30 30 Test Case n Test Case Values : The values that directly satisfy one test requirement n Expected Results : The result that will be produced when executing the test if the program satisfies it intended behavior

31 Search routine specification procedure Search (Key : INTEGER ; T: array 1..N of INTEGER; Found : BOOLEAN; L: 1..N) ; Pre-condition -- the array has at least one element 1 <= N Post-condition -- the element is found and is referenced by L ( Found and T (L) = Key) or -- the element is not in the array ( not Found and not (exists i, 1 >= i >= N, T (i) = Key ))

32 Search routine test cases

33 TESTING LEVELS 33

34 34 Testing Levels Based on Test Process Maturity  Level 0 : There’s no difference between testing and debugging  Level 1 : The purpose of testing is to show correctness  Level 2 : The purpose of testing is to show that the software doesn’t work  Level 3 : The purpose of testing is not to prove anything specific, but to reduce the risk of using the software  Level 4 : Testing is a mental discipline that helps all IT professionals develop higher quality software

35 35 Level 0 Thinking n Testing is the same as debugging n Does not distinguish between incorrect behavior and mistakes in the program n Does not help develop software that is reliable or safe

36 36 Level 1 Thinking n Purpose is to show correctness n What do we know if no failures? –Good software or bad tests? n Test engineers have no: –Strict goal –Real stopping rule –Formal test technique –Test managers are powerless

37 37 Level 2 Thinking n Purpose is to show failures n Looking for failures is a negative activity This describes most software companies.

38 38 Level 3 Thinking n Testing can only show the presence of failures n Whenever we use software, we incur some risk n Risk may be small and consequences unimportant n Risk may be great and the consequences catastrophic n Testers and developers work together to reduce risk

39 39 Level 4 Thinking n Testing is only one way to increase quality n Test engineers can become technical leaders of the project

40 TESTING MODELS 40

41 41 Testing at Different Levels Class A method mA1() method mA2() Class B method mB1() method mB2() main Class P n Acceptance testing: Is the software acceptable to the user? n Integration testing: Test how modules interact with each other n System testing: Test the overall functionality of the system n Module testing: Test each class, file, module or component n Unit testing: Test each unit (method) individually This view obscures underlying similarities

42 Criteria Based on Structures 42 Structures : Four ways to model software 1.Graphs 2.Logical Expressions 3.Input Domain Characterization 4.Syntactic Structures (not X or not Y) and A and B if (x > y) z = x - y; else z = 2 * x; A: {0, 1, >1} B: {600, 700, 800} C: {swe, cs, isa, infs}

43 43 1. Graph Coverage – Structural Node (Statement) Cover every node This graph may represent statements & branches methods & calls components & signals states and transitions Edge (Branch) Cover every edge Path Cover every path …

44 Example 44 Software Artifact : Java Method /** * Return index of node n at the * first position it appears, * -1 if it is not present */ public int indexOf (Node n) { for (int i=0; i < path.size(); i++) if (path.get(i).equals(n)) return i; return -1; } i = 0 i < path.size() if return i return -1 Control Flow Graph

45 45 1. Graph - FSM Example Memory Seats in a Lexus ES 300 Driver 1 Configuration Driver 2 Configuration [Ignition = off] | Button2 [Ignition = off] | Button1 Modified Configuration sideMirrors ()[Ignition = on] | lumbar ()[Ignition = on] | seatBottom ()[Ignition = on] | seatBack () [Ignition = on] | New Configuration Driver 1 New Configuration Driver 2 [Ignition = on] | Reset AND Button1 [Ignition = on] | Reset AND Button2 Ignition = off (to Modified) Guard (safety constraint) Trigger (input)

46 46 2. Logical Expressions ( (a > b) or G ) and (x < y) Transitions Software Specifications Program Decision Statements Logical Expressions

47 47 2. Logic – Active Clause Coverage ( (a > b) or G ) and (x < y) 1 T F T 2 F F T 3 F T T 4 F F T 5 T T T 6 T T F With these values for G and (x b) determines the value of the predicate

48 48 3. Input Domain Characterization n Describe the input domain of the software –Identify inputs, parameters, or other categorization –Partition each input into finite sets of representative values –Choose combinations of values n System level –Number of students { 0, 1, >1 } –Level of course { 600, 700, 800 } –Major { swe, cs, isa, infs } n Unit level –Parameters F (int X, int Y) –Possible values X: { 2 }, Y : { 10, 20, 30 } –Tests F (-5, 10), F (0, 20), F (1, 30), F (2, 10), F (5, 20)

49 49 4. Syntactic Structures n Based on a grammar, or other syntactic definition n Primary example is mutation testing 1.Induce small changes to the program: mutants 2.Find tests that cause the mutant programs to fail: killing mutants 3.Failure is defined as different output from the original program 4.Check the output of useful tests on the original program n Example program and mutants if (x > y) z = x - y; else z = 2 * x; if (x > y)  if (x >= y) z = x - y;  z = x + y;  z = x – m; else z = 2 * x;

50 50 Coverage Overview Four Test Modeling Software Graphs Logic Input Space Syntax Use cases Specs Design Source Applied to DNF Specs FSMs Source Input Models Integ Source


Download ppt "Software Verification Introduction & The Model-Driven Test Design Process."

Similar presentations


Ads by Google