1. Verification and Validation
Assuring that a software system meets a user's needs

2. Verification vs validation
Verification: "Are we building the product right"
Validation: "Are we building the right product"
… The software should conform to its specification
… The software should do what the user really requires

3. Static and dynamic verification
Software inspections (static verification and validation)
Software testing (dynamic verification)
… Concerned with analysis of the static system representation to discover problems (May be supplement by tool-based document and code analysis)
… Concerned with exercising and observing product behaviour - The system is executed with test data and its operational behaviour is observed

4. Static and dynamic V&V
V & V Is a whole life-cycle process - V & V must be applied at each stage in the software process.
Has three principal objectives
Ensuring that requirements are correct
The discovery of defects in a system
The assessment of whether or not the system is usable in an operational situation.
<POOR DIAGRAM> THE CORRECT POINT – STATIC VALIDATION Should be done during requirements specification, and probably to some extent during high level design. STATIC VERIFICATION should be done during design and programming. DYNAMIC VERIFICATION should be done on prototypes and on programs
Static Inspection includes study of requirements documents, design diagrams, and program code. Program inspections can be formal proofs or less formal walkthroughs (group reads code looking for errors)

5. Program testing Can reveal the presence of errors, NOT their absence
A successful test is a test which discovers one or more errors
The only validation technique for non-functional requirements
Should be used in conjunction with static verification and validation to provide full V&V coverage

6. Types of testing Defect testing Statistical testing Defect testing
Tests designed to discover system defects.
A successful defect test is one which reveals the presence of defects in a system. – Intentionally try to create hard cases – stress the system
Covered in Chapter 20
Statistical testing
tests designed to reflect the frequency of user inputs. Used to get a realistic feel for how things will work in operational environment - for reliability estimation, or other non-functional issues such as performance.
Covered in Chapter 21 (we may skip)

7. V& V goals
Verification and validation should establish confidence that the software is fit for its purpose
This does NOT mean completely free of defects
Must be good enough for its intended use
the type of use will determine the degree of confidence that is needed - Depends on system’s purpose, user expectations and marketing environment
Software function
The level of confidence depends on how critical the software is to an organization – business wise, safety wise etc
User expectations
Users may have low expectations of certain kinds of software <A shot at Microsoft?>
Marketing environment
Getting a product to market early may be more important than finding defects in the program. Users may tolerate more errors in cheap software

8. Testing and debugging
Defect testing and debugging are distinct processes
Verification and validation is concerned with establishing the existence of defects in a program
Debugging is concerned with locating and repairing these errors
Debugging involves formulating a hypothesis about program behaviour then testing these hypotheses to find the system error
… it is generally desirable to have an independent group do testing after the developers have tested and declared the system done
… it is preferable if the developers debug their own problems – they know their code best. But code should be fully documented in case the developer is no longer around or is tied up in other tasks

9. The debugging process
This you guys know from your own debugging for courses. But I want to emphasize – it is common for changes to fix bugs to cause other bugs. Re-testing should not only test for elimination of the bug, but also that other parts of the program have not been broken in the process.
If significant changes have been made, a complete regression test, may be done – repeating all tests that were previously done to make sure only the intended changes have occurred (particularly when a patch release is being prepared)
Also note, sometimes it is a challenge to recreate the problem. The developer probably found the easy bugs before saying the program was done. Testers (and later customers) may do a particular combination of things that just happens to cause a problem. If they don’t remember exactly what they did, then the developer may not be able to recreate the error – and then may be short on clues.
Thus, it is important for testers to create test plans, not just so they make sure they hit everything – but also so they know what it is they did (when I was at IBM, it was common for developers to go over and observe the tester’s screen as the tester recreated the bug)

10. 19.1 V & V planning
Careful planning is required to get the most out of testing and inspection processes
Planning should start early in the development process
The plan should identify the balance between static verification and testing
Planning should define standards for the testing process
½ the budget may be spent on V & V
… I think there are two levels of planning. Here we are talking about high level planning. At a lower level, individual testers can create a detailed plan of what exactly they are going to test – describing product tests

11. The V-model of development
<we saw this same diagram last semester – the take home point is that testing planning starts early, as it is informed by early activities>

12. The structure of a software test plan
The testing process
Requirements traceability
Tested items
Testing schedule
Test recording procedures
Hardware and software requirements
Constraints
… what phases of testing to be done – e.g. unit test, subsystem integration test, system integration test, acceptance test
… plan to test all requirements – must know what parts of system support which requirements in order to do this – requirements traceability …
… including resource allocation
… to be followed – e.g. tester might enter faults into a DB that can then be searched by management and developers, to ensure that all problems are addressed
… needed for testers – including tools to be used
… affecting the testing process
UPDATE as things change

13. 19.2 Software inspections
Involve people examining the source representation with the aim of discovering anomalies and defects
Do not require execution of a system so may be used before implementation
May be applied to any representation of the system (requirements, design, test data, etc.)
Very effective technique for discovering errors
… frequently this means examining code – getting people to read it carefully
(minor?)
… e.g. study that 60% of errors can be detected using informal program inspections. Cheaper to find errors than by extensive program testing

14. Inspection success
Many different defects may be discovered in a single inspection. In testing, one defect ,may mask another so several executions are required
They reuse domain and programming knowledge - reviewers are likely to have seen the types of error that commonly arise and look out for them

15. Inspections and testing
Inspections and testing are complementary and not opposing verification techniques
Inspections can check conformance with a specification but not conformance with the customer’s real requirements
Inspections cannot test integration of subsystems
Inspections cannot check non-functional characteristics such as performance, usability, etc.
… Both should be used during the V & V process
Organizations that have not been doing them may be reluctant to do them – because it moves V & V expenses earlier (with promise of later savings). It takes faith on the part of developers and managers to do them

16. Program inspections
Group meeting to carefully, line-by-line review code
Intended explicitly for defect DETECTION (not correction)
Defects may be logical errors, or non-compliance with standards
Invented by IBM in the 1970’s
… beneficial sometimes if members come from different roles (e.g. QA, not just developers <can’t see users being able to participate – somebody could take on the role of the user>
… correction would make meetings go forever, and waste time
… (or anomalies in the code that might indicate an erroneous condition (e.g. an uninitialized variable) )

17. Inspection teams Made up of at least 4 members
Author of the code being inspected
Inspector who finds errors, omissions and inconsistencies
Reader who reads the code to the team
Moderator who chairs the meeting and notes discovered errors
Other roles are Scribe and Chief moderator
… Developer must try not to be defensive
… some organizations skip the reader
Meeting should be moderated … by somebody other than the developer
Probably better to have a Scribe who records all errors, allowing the moderator to keep meeting on track etc

18. Inspection pre-conditions
A precise specification must be available
Team members must be familiar with the organisation standards
Syntactically correct code must be available
An error checklist should be prepared
Management must accept that inspection will increase costs early in the software process
Management must not use inspections for staff appraisal
… in order that group can check if program meets the spec
… this is not for removing syntax errors – waste of a lot of people’s time when compiler could do this
… otherwise the meeting will not go well – defensive developers, reviewers not raising issues for political reasons (especially if the developer is more senior)

19. The inspection process
Planning done in advance – selecting team, getting a room, ensuring developer is ready, and materials are available
System overview presented to inspection team – an introduction given by developer
Code and associated documents are distributed to inspection team in advance. They must read it and try to identify bugs – otherwise the meeting will not work – will take too long
Inspection takes place and discovered errors are noted – meeting should not last more than 2 hours or people will burn out (lose effectiveness)
Modifications are made to repair discovered errors
Re-inspection may or may not be required

20. Inspection checklists
Checklist of common errors should be used to drive the inspection
Error checklist is programming language dependent
The 'weaker' the type checking, the larger the checklist
Examples: Initialization, Constant naming, loop termination, array bounds, etc.

21. Inspection checks

22. Inspection rate 500 statements/hour during overview
125 source statement/hour during individual preparation
statements/hour can be inspected
Inspection is therefore an expensive process
Inspecting 500 lines costs about 40 person-hours effort

23. 19.3 Automated static analysis
Static analysers are software tools for source text processing
They parse the program text and try to discover potentially erroneous conditions and bring these to the attention of the V & V team
Very effective as an aid to inspections. A supplement to but not a replacement for inspections

24. Static analysis checks
<Some of this the compiler should get – possibly as a warning (but C is very forgiving!) (LINT is a static analyzer for C on UNIX that checks things that a compiler for a stronger typed language would check)>

25. Stages of static analysis
Control flow analysis. Checks for loops with multiple exit or entry points, finds unreachable code, etc.
Data use analysis. Detects uninitialised variables, variables written twice without an intervening assignment, variables which are declared but never used, etc
Interface analysis. Checks the consistency of routine and procedure declarations and their use. Functions never used, results of functions not used

26. Stages of static analysis
Information flow analysis. Identifies the dependencies of output variables. Does not detect anomalies itself but highlights information for code inspection or review
Path analysis. Identifies paths through the program and sets out the statements executed in that path. Again, potentially useful in the review process
Both these stages generate vast amounts of information. Must be used with care.
… basically a documentation showing how variables get their values … kind of a particular kind of abstract of the code
… could also be used to generate test cases.

27. Use of static analysis
Particularly valuable when a language such as C is used which has weak typing and hence many errors are undetected by the compiler
Less cost-effective for languages like Java that have strong type checking and can therefore detect many errors during compilation

28. 19.4 Cleanroom software development
The name is derived from the 'Cleanroom' process in semiconductor fabrication. The philosophy is defect avoidance rather than defect removal
Software development process based on:
Formal specification.
Incremental development
Structured programming
Static verification using correctness arguments
Statistical testing to determine program reliability.
…. Inspections replace unit testing
Software specified using a state-transition model
Software broken into parts (increments) which are separately developed and verified. Critical components developed first
… limited number of control constructs … generate code from spec using correctness-preserving transformations
… mathematical arguments of correctness, not necessarily code formally proved correct

29. The Cleanroom process

30. Incremental development
Change requests result in new release of increment. But increments are small, change requests are isolated (in theory)
As new increments are added, the integration is tested

31. Cleanroom process teams
Specification team. Responsible for developing and maintaining the system specification
Development team. Responsible for developing and verifying the software. The software is NOT executed or even compiled during this process
Certification team. Responsible for developing a set of statistical tests to exercise the software after development. Reliability growth models used to determine when reliability is acceptable
Three teams are used when Cleanroom process is used for large system development
… user level, and formal technical spec
… code inspection plus correctness arguments
… test cases are developed as the software is being developed

32. Cleanroom process evaluation
Results in IBM have been very impressive with few discovered faults in delivered systems
Independent assessment shows that the process is no more expensive than other approaches
Fewer errors than in a 'traditional' development process
Not clear how this approach can be transferred to an environment with less skilled or less highly motivated engineers
Not being quickly adopted by industry