Software Verification, Validation, and Acceptance Testing IMPORTANT NOTICE TO STUDENTS: These slides are NOT to be used as a replacement for student notes. These slides are sometimes vague and incomplete on purpose to spark class discussions. Software Verification, Validation, and Acceptance Testing CS 360 Lecture 20

Notes from experienced software testers You never know what is enough [testing] unless you know what is more than enough. Great engineering is simple engineering. To err is human. To really foul things up requires a computer. Myth #1: The computer only does what you tell it. There’s always one more bug. The definition of an Upgrade: Take old bugs out, put new ones in. Computers make very fast, very accurate mistakes. “It did what? Well, it’s not supposed to do that.” “My software never has bugs. It just develops random features.” If debugging is the process of removing bugs, then programming must be the process of putting them in. The sooner a bug is found and fixed, the cheaper.

Verification and Validation Verification testing: “Are we building the product correctly?” Does the design meet the reqs? Does the implementation meet the reqs and match the models? The software should conform to its specifications Functional/non-functional requirements Validation testing: “Are we building the correct product?” Do our problem requirements accurately capture the real problem? Did we account for the needs of all stakeholders? The software should do what the users really require.

Verification vs. validation

Benefits of Verification and validation Facilitate early detection and correction of software anomalies. Enhance management insight into process and product risk. Support the software life cycle processes to ensure conformance to program performance, schedule, and budget. Provide an early assessment of software and system performance. Improve the software development and maintenance processes. Support the process improvement for software system integration.

Verification and validation criteria Machine domain: The thing (software) to be built. Turning a general-purpose hardware platform into a useful machine for a particular purpose. Application domain: The “world” in which the machine will be introduced, observed, and evaluated. Specifications: Refers to things that cross the application/machine boundaries, such as the inputs and outputs of the software and how the application domain places constraints. The machine domain and application domain must be connected The machine must interact with the world to be useful.

Verification and validation criteria Domain Properties: The things in the application domain that are always true/given. Ex: websites use HTTP, multi tenant, TCP/UDP/IP, etc. Requirements: The things in the application domain that we wish to be made true by the machine. Computer: General-purpose hardware platform. Program: Set of algorithms developed for the computer specified in the machine domain to satisfy the specifications.

Verification and validation criteria Two Verification criteria: The Program running on the Computer satisfies the Specification. The Specification, given the Domain properties, satisfies the Requirements. Two Validation criteria: Did we discover and understand all of the important Requirements? Did we discover and understand all of the relevant Domain properties?

Verification and validation Example 1 Problem Statement: For an aircraft, it is important to prevent accidental engagement of reverse thrust while the aircraft is flying. Requirements (what we want to be true): R1: “Reverse thrust shall only be enabled when the aircraft is moving on the runway, and disabled at all other times.” Domain properties: D1: Wheel sensor pulses are on iff the wheels are turning. D2: Wheels are turning iff the aircraft is on the runway. Specifications: S1: Reverse thrust enabled iff wheel pulses are on.

Verification and validation Example 1 Verification (building the product correctly): Does the flight software, P, running on the aircraft flight computer, C, correctly implement S1? Does S1, based on the assumptions of D1 and D2, satisfy R1? Validation (building the correct product): Are our assumptions, D1 and D2, about the domain correct? Did we miss any relevant application domain assumptions? Is the requirement, R1, what is really needed?

Verification and validation Example 1 Verification (building the product correctly): To verify, write test cases that check whether it meets the specification. Might require a very large set of test cases. May not be possible to test all scenarios. Validation (building the correct product): To validate, check that the requirement and domain properties capture what happens in reality. Landing on a wet runway?

Verification and validation Example 2 Problem Statement: E-commerce website should save the cart state of each user, allowing them to access their cart after logging in using any web-based browser. Requirements (what we want to be true): R1: “User carts should only be viewable after successful login using a compatible web-browser, and inaccessible otherwise.” Domain properties: D1: Website responds to http(s) requests from the user. D2: Cart is accessible iff the user is authenticated. Specifications: S1: Cart is viewable iff a user is authenticated using a compatible web-browser.

Verification and validation Example 2 Team activity Discuss and create: Verification questions/test cases (building the product correctly) Validation questions/test cases (building the correct product)

Verification, Validation and Quality assurance V&V and Software Quality Assurance are not the same, but compliment each other. Software Quality Assurance: Determines if the software development is following a defined process. Closely related to software verification. Verification and Validation: Based in the field of systems engineering and takes a rigorous approach to determine if the product built is the correct product, and that product is built correctly.

Verification, Validation and Quality assurance V&V and Software Quality Assurance Example Code Review Verification and Validation: Analyze the code to determine if there are any static coding errors, correct logic, and design/requirements are captured. May suggest technical solutions for addressing issues. Reviews fixes of previously found software errors. Software Quality Assurance: Complete a checklist that all participants have reviewed the material and are present for the code review. Complete a checklist that all required materials have been provided to the review team: code files, documentation, test results, etc. Ensures all defects from the review are captured with a proposed action. Ensures all defects are reported as closed before allowing the code review to end and allowing the module to be integrated.

Verification and Validation testing Verification test types: Unit tests Integration tests Code reviews (documented) Verification of requirements against the developed product Verification of documented design/models against the developed product Validation test types: Usability/user tests Acceptance tests (with the client) Validation of product goals against the developed product

Acceptance testing Validation by users/client/other stakeholders that the developed product meets the stated goals. Generally tested as pass/fail outcomes. Acceptance test cases should: Cover all given functional and non-functional requirements. Verify the human-machine interactions as specified by the product goals. Ensure that users are able to complete expected tasks using the system.

Acceptance testing Final testing phase before product release

Team Activity Continue developing test plan documentation. Unit test plan Integration test plan System test plan Verification/validation test plan Acceptance test plan