ECE 453 – CS 447 – SE 465 Software Testing & Quality Assurance Instructor Paulo alencar

Overview Software Quality Software Quality and Quality Assurance Non-Execution Based Testing Reviews and inspections

The State of Software Development 2000 Defense Science Board Study: 53% of projects were late and over budget, 16% were on time, 31% were canceled before completion. There is tremendous growth in software content in both manned and unmanned systems. Software requirements now amount to the bulk of the overall specification requirements (65% for the B-2, 80% for the F-22)

Software Quality The American Heritage Dictionary defines quality as “a characteristic or attribute of something.” For software, some kinds of quality may be encountered: Quality of design encompasses requirements, specifications, and the design of the system. Quality of conformance is an issue focused primarily on implementation. user satisfaction: compliant product + delivery within budget and schedule, etc.

Software Quality Quality, simplistically, means that a product should meet its specification. This is problematical for software systems There is a tension between customer quality requirements (efficiency, reliability, etc.) and developer quality requirements (maintainability, reusability, etc.); Some quality requirements are difficult to specify in an unambiguous way; Software specifications are usually incomplete and often inconsistent.

Cost of Quality Prevention costs include quality planning formal technical reviews test equipment Training Internal failure costs include rework repair failure mode analysis External failure costs are complaint resolution product return and replacement help line support warranty work

Software Quality Assurance Process Definition & Standards Formal Technical Reviews Analysis & Reporting Test Planning & Review Measurement

Software Quality Team Size

Role of a Software Quality Team

Tasks of a Software Quality Team

Role of the SQA Group Prepares an SQA plan for a project. The plan identifies evaluations to be performed audits and reviews to be performed standards that are applicable to the project procedures for error reporting and tracking documents to be produced by the SQA group amount of feedback provided to the software project team Participates in the development of the project’s software process description. The SQA group reviews the process description for compliance with organizational policy, internal software standards, externally imposed standards (e.g., ISO-9001), and other parts of the software project plan.

Role of the SQA Group Reviews software engineering activities to verify compliance with the defined software process. identifies, documents, and tracks deviations from the process and verifies that corrections have been made. Audits designated software work products to verify compliance with those defined as part of the software process. reviews selected work products; identifies, documents, and tracks deviations; verifies that corrections have been made periodically reports the results of its work to the project manager. Ensures that deviations in software work and work products are documented and handled according to a documented procedure. Records any noncompliance and reports to senior management. Noncompliance items are tracked until they are resolved.

Analyzing Defects measurement Product & Process ... an understanding of how to improve quality ... Collect information on all defects Find the causes of the defects Move to provide fixes for the process

Analyzing Defects Remember, high quality products Must meet all user requirements Must be free of defects In some processes, the main measure of software product quality is total defect density Total defect density is measured as defects/KLOC (number of defects removed in development per thousand LOC)

Example Quality Measurement Defect Removal Efficiency E is the number of errors found before delivery of the software to the user D is the number of defects found after delivery DRE = E /(E + D)

Why SQA Activities Pay Off? cost to find and fix a defect 100 60.00-100.00 log scale 10.00 10 3.00 1.50 1.00 1 0.75 Design test field system Req. code use test

Software Quality Team Reviewing Applications The major role of the Software Quality Team is to review applications. Reviewing Applications Includes verification and validation It also includes the following: Software reviews and inspections Traceability of software deliverables Testing Auditing of selected key software items Standards and Guidelines

1. Software Reviews and Inspections A software review is an evaluation of a software element to ascertain discrepancies from planned results and to recommend improvements e.g. Design Review, Code Review Different types, e.g.: Walkthrough Software Inspection Technical Review

2. Traceability Forwards Traceability each input to a phase must be matched to an output of the same phase to show completeness Backwards Traceability each output is traceable to an input of a phase

3. Testing Testing is the evaluation of a system or component to: confirm that it satisfies requirements identify differences between actual and expected results Testing may be 50% of project costs; so use cheaper methods before testing starts, i.e., walkthroughs and inspections

4. Auditing Audits are independent reviews that assess compliance with specifications, standards and procedures Perform an audit before software release: Physical audit check that all items identified as part of the configuration are present Functional audit check that unit, integration and acceptance tests have been carried out and that records show their success or failure

5. Software Quality Standards Standards and Guidelines ‘A standard is an approved, documented, and available set of criteria used to determine the adequacy of an action (process standard) or object (product standard).’ ‘A guideline is a well-defined and documented set of criteria that guides an activity or task.’ (Dorfman & Thayer, 1990) differ from standards - allows for judgement and flexibility Categories of Standards and Assessment Product Standards Calibration and Measurement Standards Quality Management Systems Standards

Verification and Validation

Verification and Validation Are we building the right system? Verification: Are we building the system right? Quality Assurance ensures that verification and validation takes place.

Software Reviews and Inspections A way of using the diversity of a group to: uncover errors in function, logic, or implementation verify that software under review meets requirements ensure that the software meets predefined standards achieve uniformity of development make projects more manageable

The Review Players review leader producer reviewer recorder standards bearer (SQA) producer maintenance oracle reviewer recorder user rep

Software Reviews A meeting conducted by technical people for technical people A technical assessment of a work product created during the software engineering process A software quality assurance mechanism A training ground

Types of Review There are a number of types of review ranging in formality and effect. These include: Buddy Checking having a person other than the author informally review a piece of work. generally does not require collection of data difficult to put under managerial control generally does not involve the use of checklists to guide inspection and is therefore not repeatable.

Types of Review Walkthroughs generally involve the author of an artifact presenting that document or program to an audience of their peers The audience asks questions and makes comments on the artifact in an attempt to identify defects often break down into arguments about an issue usually involve no prior preparation on behalf of the audience usually involve minimal documentation of the process and of the issues found process improvement and defect tracking are therefore not easy

Types of Review Review by Circulation similar in concept to a walkthrough artifact to be reviewed is circulated to a group of the author(s) peers for comment avoids potential arguments over issues, however it also avoids the benefits of discussion reviewer may be able to spend longer reviewing the artifact there is documentation of the issues found, enabling defect tracking usually minimal data collection

Types of Review Inspection formally structured and managed peer review processes involve a review team with clearly defined roles specific data is collected during inspections inspections have quantitative goals set reviewers check an artifact against an unambiguous set of inspection criteria for that type of artifact The required data collection promotes process improvement, and subsequent improvements in quality.

Technical Reviews The Review Meeting between 3 to 5 people (reviewers) should be involved advance preparation should occur (< 2 hours work) meeting should be < 2 hours should be for a small part of the overall software is attended by the producer, review leader and reviewers must have a follow-up procedure to ensure any corrections are completed

Technical Reviews The Review Meeting At the end of the review the attendees must decide whether to: accept the work product without modification reject the work product due to severe errors (once corrected another review is performed) accept the work product provisionally (minor errors to be corrected but no further review)

Conducting a Review 1. be prepared—evaluate product before the review 2. review the product, not the producer 3. keep your tone mild, ask questions instead of making accusations 4. stick to the review agenda 5. raise issues, don't resolve them 6. avoid discussions of style—stick to technical correctness 7. schedule reviews as project tasks 8. record and report all review results

Technical Reviews The Review Report should contain: - Type of Review, Team Members, Date, Time, Place - Item being reviewed - Agenda - Checklist - Comments per Checklist item or per Issues list item (Pressman) with problems mentioned Action per Comment - No action, Fix error, Reconsider   Decision for the item being reviewed Accept, Reject (severe errors), Accept (minor errors)

Technical Reviews Review Guidelines Review the product not the producer Set an agenda and maintain it Limit debate and rebuttal Enunciate problem areas, but don’t attempt to solve every problem Take written notes Limit the number of participants and insist on advance preparation Develop a checklist for each work product Allocate resources and time schedule Conduct meaningful training for all reviewers Review earlier reviews

Software Inspection The inspection process comprises three broad stages: preparation collection repair Gilb and Graham [GilbGraham93] expand this three stage process into the inspection steps; Entry, Planning, Kickoff Meeting, Individual Checking, Logging Meeting, Root Cause Analysis Edit, Follow Up, Exit.

Benefits of Software Inspection 30% to 100% net productivity increases; Overall project time saving of 10% to 30%; 5 to 10 times reduction in test execution costs and time; Reduction in maintenance costs of up to one order of magnitude; Improvement in consequent product quality; Minimal defect correction backlash at systems integration time. In addition to these tangible benefits, less tangible benefits such as a training effect for inspectors are also evident.

Disadvantages Up front costs (although far outweighed by benefit): Training Implementation Support Ongoing allocation of staff resources Not strictly repeatable