1. 1 Security Architecture and Analysis Software Inspections and Verification Software Testing and Certification

2. 2 The Software System Development Process Customer requirements Usage specification Architecture Defn (COTS, legacy), Increment Plng Design and Verification Testing and Certification Function specification Deployment and operation Incremental Development Highly Incremental Project team

3. 3 Software Inspections Organization –Team-based activity –Specific roles and steps –Training required Management view –Proven path to quality improvement –A clear “exit criteria” –Inspect all work products: specifications, architecture, designs, implementations,... –Measure and manage results

4. 4 Correctness Verification The next step beyond ordinary inspections –Based on theory, but has practical application –Can do verbal proofs in “verification inspections” –Can do written proofs for more rigor

5. 5 Programming Fundamentals All sequential programs, no matter what language or subject matter, can be represented in just three basic structures: –Sequence: do-end –Alternation:if-then-else-end –Iteration:while-do-end We will explore verification “look and feel” for sequence structures only, using the Trace Table method

6. 6 An Example Sequence Structure In text form: do x := x + 1 y := y - x x := x + y end In flowchart form: x := x - 1y := y - xx := x + y

7. 7 Verifying Long Division Division: 16 232 3712 232 1392 0 Verification: 16 x 232 = 3712 Verification is by a different process than the one that produced the original answer

8. 8 Software Correctness Verification Basics Verification is accomplished by comparing: –What a program is supposed to do to variables -- this is its specification (called its intended function) –What a program actually does to variables -- this is its net effect from beginning to end (called its program function) If the intended function and the program function are identical, the program is correct

9. 9 The Verification Process Write the Intended Function Write the Program Function Step 2: Compare for equivalence DevelopmentVerification (What the program actually does) (What the program is supposed to do; its specification) Step 1: Derive

10. 10 Defining Intended Functions Concurrent assignment statements can be used to define intended functions Values of expressions on the right are simultaneously assigned to corresponding variables on the left (old values on the right, new values on the left) Example: [a, b, c := a + 1, a + b, b - c] means: a := a + 1 b := a + b Assignments are simultaneous c := b - c

11. 11 Concurrent Assignments [x, y, z := x + a, y - x, a + y] means?

12. 12 Correctness Verification Steps Given a program and its intended function: Step 1: Derive the program function –It is the net effect of the program on the variables to which it assigns new values –“Net effect” mean the change from initial values at the start of the program to final values at the end of the program Step 2: Compare the program function to the intended function for equivalence (or not)

13. 13 A Miniature Program -- Is it Correct? Intended function (all variables are integers): [x, y, t := y, x, x] Program: do t := x x := y y := t end

14. 14 Trace Table Derivation of Program Functions Trace Table construction –Define a row for each assignment, and a column for each variable whose value is changed –Mechanically fill in the variable changes in each row Append initial values in the first row with a “0” suffix Increment the suffix with each successive row Trace Table Derivation –Start with last row and mechanically substitute values until you get to the “0” suffixes from first row –Final expression is the program function

15. 15 Verifying the Miniature Program Program t x y 1 t := x t1 := x0 x1 = x0 y1 := y0 2 x := y t2 := t1 x2 = y1 y2 := y1 3 y := t t3 := t2 x3 = x2 y3 := t2 Derivations: t3 = t2 x3 = x2 y3 = t2 = t1 = y1 = t1 = x0 = y0 = x0 Derived program function: [x3, y3, t3 := y0, x0, x0], or simply [x, y, t := y, x, x] Therefore, the program computes the intended function’s exchange of the values of x and y, and the setting of t to the value of x. Trace Table to derive the program function:

16. 16 Is This Program Correct? Intended function: [x, y := y, x] Program: do x := x + y y := x - y x := x - y end

17. 17 The Trace Table Proof Program x y 1 x := x + y 2 y := x - y 3 x := x - y Trace Table: Derivation: x3 = y3 = Derived program function:

18. 18 Is This Program Correct? Intended function: [x, y := y, y + x - 1] Program: do x := x + 1 y := y - x x := x + y end

19. 19 The Trace Table Proof

20. 20 Verifying Branching Logic [(x > y --> x, y := y, x) | (x x, y := x + 1, y - 1)] if x > y then do [x, y := y, x] x := x + y y := x - y x := x - y end else do [x, y := x + 1, y - 1] x := x + 1 y := x - 1 end endif

21. 21 Verification Perspective Other sources of errors –Human fallibility in verification –Compiler errors –Support software errors –Hardware errors Verification, combined with other rigorous software engineering practices, can produce software of extremely high quality

22. 22 Testing and Certification You are the director of quality control for a light bulb factory 100,000 light bulbs come off the assembly line every hour: –50,000 60w bulbs –30,000 75w bulbs –15,000 100w bulbs –5,000 250w bulbs (This is also the distribution of customer purchases)

23. 23 Define a Testing Process for Quality Control How will you predict customer quality experience with the light bulbs?

24. 24 A Light Bulb Testing Process Sample the population of bulbs according to customer usage, then the quality of the sample will predict customer quality experience with the bulbs in the population that were not tested The sample composition should be: –60W: 50% –75w:30% –100w:15% –250w: 5% Test the sample to determine MTTF Compare results to quality objectives

25. 25 Software Testing Realities A software system is capable of a virtually infinite number of possible executions (each is a potential test case) No testing process, no matter how thorough, can exercise more than a minute fraction of these possible executions All testing is sampling; the only question is how to define the sample

26. 26 Software Testing Methods -- 1 Test to debug the code (40 years of widespread use) –Objective of most testing today, but can result in buggy systems Testing can show the presence of bugs, but never their absence –Often called coverage testing –Test cases are often invented based on knowledge of code internals (this is the sample) – Produces only anecdotal evidence of quality (“The code ran these 100 tests ok”) –Not useful for predictions about all the executions not tested

27. 27 Software Testing Methods -- 2 Test to certify the code (10 years of minimal use) –Objective is not to debug, but to certify quality –Called statistical, usage-based testing –Test cases are based on the external usage that actual users engage in –If sample the population of executions according to usage, the results can predict user experience with all the possible executions that were not tested. –Produces scientific measures of code quality

28. 28 Statistical Usage-Based Testing Develop test cases for the sample Define a statistical sample of the population based on how the system will be used Execute the test cases, compute MTTF Population of possible executions Compare to quality goals Improve the production process as necessary

29. 29 Testing According to Usage Example: Usage distribution for a banking system defines the test case distribution –Account updates:60% of usage and tests –Fund transfers:25% –Statement generation:10% –Audit checks: 5%

30. 30 Lessons To obtain valid estimates of software quality –Test the system based on anticipated usage by actual users Usage scenarios play a major role in modern testing methods If a system’s usage environment changes, it will require usage-based testing for the new environment