1. ECE 453 – CS 447 – SE 465 Software Testing & Quality Assurance Instructor Paulo alencar

2. Overview Software Quality Software Quality and Quality Assurance
Non-Execution Based Testing
Reviews and inspections

3. The State of Software Development
2000 Defense Science Board Study:
53% of projects were late and over budget, 16% were on time, 31% were canceled before completion.
There is tremendous growth in software content in both manned and unmanned systems.
Software requirements now amount to the bulk of the overall specification requirements (65% for the B-2, 80% for the F-22)

4. Software Quality The American Heritage Dictionary defines quality as
“a characteristic or attribute of something.”
For software, some kinds of quality may be encountered:
Quality of design encompasses requirements, specifications, and the design of the system.
Quality of conformance is an issue focused primarily on implementation.
user satisfaction: compliant product + delivery within budget and schedule, etc.

5. Software Quality
Quality, simplistically, means that a product should meet its specification.
This is problematical for software systems
There is a tension between customer quality requirements (efficiency, reliability, etc.) and developer quality requirements (maintainability, reusability, etc.);
Some quality requirements are difficult to specify in an unambiguous way;
Software specifications are usually incomplete and often inconsistent.

6. Cost of Quality Prevention costs include quality planning
formal technical reviews
test equipment
Training
Internal failure costs include
rework
repair
failure mode analysis
External failure costs are
complaint resolution
product return and replacement
help line support
warranty work

7. Software Quality Assurance
Process
Definition &
Standards
Formal
Technical
Reviews
Analysis
&
Reporting
Test
Planning
& Review
Measurement

8. Software Quality Team Size

9. Role of a Software Quality Team

10. Tasks of a Software Quality Team

11. Role of the SQA Group Prepares an SQA plan for a project.
The plan identifies
evaluations to be performed
audits and reviews to be performed
standards that are applicable to the project
procedures for error reporting and tracking
documents to be produced by the SQA group
amount of feedback provided to the software project team
Participates in the development of the project’s software process description.
The SQA group reviews the process description for compliance with organizational policy, internal software standards, externally imposed standards (e.g., ISO-9001), and other parts of the software project plan.

12. Role of the SQA Group
Reviews software engineering activities to verify compliance with the defined software process.
identifies, documents, and tracks deviations from the process and verifies that corrections have been made.
Audits designated software work products to verify compliance with those defined as part of the software process.
reviews selected work products; identifies, documents, and tracks deviations; verifies that corrections have been made
periodically reports the results of its work to the project manager.
Ensures that deviations in software work and work products are documented and handled according to a documented procedure.
Records any noncompliance and reports to senior management.
Noncompliance items are tracked until they are resolved.

13. Analyzing Defects measurement Product & Process
... an understanding of how
to improve quality ...
Collect information on all defects
Find the causes of the defects
Move to provide fixes for the process

14. Analyzing Defects Remember, high quality products
Must meet all user requirements
Must be free of defects
In some processes, the main measure of software product quality is total defect density
Total defect density
is measured as defects/KLOC (number of defects removed in development per thousand LOC)

15. Example Quality Measurement
Defect Removal Efficiency
E is the number of errors found before delivery of the software to the user
D is the number of defects found after delivery
DRE = E /(E + D)

16. Why SQA Activities Pay Off?
cost to find
and fix a defect
100
log
scale
10.00 10
3.00
1.50
1.00 1
0.75
Design
test
field
system
Req.
code
use
test

17. Software Quality Team Reviewing Applications
The major role of the Software Quality Team is to review applications.
Reviewing Applications
Includes verification and validation
It also includes the following:
Software reviews and inspections
Traceability of software deliverables
Testing
Auditing of selected key software items
Standards and Guidelines

18. 1. Software Reviews and Inspections
A software review is an evaluation of a software element to ascertain discrepancies from planned results and to recommend improvements
e.g. Design Review, Code Review
Different types, e.g.:
Walkthrough
Software Inspection
Technical Review

19. 2. Traceability Forwards Traceability
each input to a phase must be matched to an output of the same phase to show completeness
Backwards Traceability
each output is traceable to an input of a phase

20. 3. Testing Testing is the evaluation of a system or component to:
confirm that it satisfies requirements
identify differences between actual and expected results
Testing may be 50% of project costs;
so use cheaper methods before testing starts,
i.e., walkthroughs and inspections

21. 4. Auditing
Audits are independent reviews that assess compliance with specifications, standards and procedures
Perform an audit before software release:
Physical audit
check that all items identified as part of the configuration are present
Functional audit
check that unit, integration and acceptance tests have been carried out and that records show their success or failure

22. 5. Software Quality Standards
Standards and Guidelines
‘A standard is an approved, documented, and available set of criteria used to determine the adequacy of an action (process standard) or object (product standard).’
‘A guideline is a well-defined and documented set of criteria that guides an activity or task.’ (Dorfman & Thayer, 1990)
differ from standards - allows for judgement and flexibility
Categories of Standards and Assessment
Product Standards
Calibration and Measurement Standards
Quality Management Systems Standards

23. Verification and Validation

24. Verification and Validation
Are we building the right system?
Verification:
Are we building the system right?
Quality Assurance ensures that verification and validation takes place.

25. Software Reviews and Inspections
A way of using the diversity of a group to:
uncover errors in function, logic, or implementation
verify that software under review meets requirements
ensure that the software meets predefined standards
achieve uniformity of development
make projects more manageable

26. The Review Players review leader producer reviewer recorder
standards bearer (SQA)
producer
maintenance
oracle
reviewer
recorder
user rep

27. Software Reviews
A meeting conducted by technical people for technical people
A technical assessment of a work product created during the software engineering process
A software quality assurance mechanism
A training ground

28. Types of Review
There are a number of types of review ranging in formality and effect. These include:
Buddy Checking
having a person other than the author informally review a piece of work.
generally does not require collection of data
difficult to put under managerial control
generally does not involve the use of checklists to guide inspection and is therefore not repeatable.

29. Types of Review Walkthroughs
generally involve the author of an artifact presenting that document or program to an audience of their peers
The audience asks questions and makes comments on the artifact in an attempt to identify defects
often break down into arguments about an issue
usually involve no prior preparation on behalf of the audience
usually involve minimal documentation of the process and of the issues found
process improvement and defect tracking are therefore not easy

30. Types of Review Review by Circulation
similar in concept to a walkthrough
artifact to be reviewed is circulated to a group of the author(s) peers for comment
avoids potential arguments over issues, however it also avoids the benefits of discussion
reviewer may be able to spend longer reviewing the artifact
there is documentation of the issues found, enabling defect tracking
usually minimal data collection

31. Types of Review Inspection
formally structured and managed peer review processes
involve a review team with clearly defined roles
specific data is collected during inspections
inspections have quantitative goals set
reviewers check an artifact against an unambiguous set of inspection criteria for that type of artifact
The required data collection promotes process improvement, and subsequent improvements in quality.

32. Technical Reviews The Review Meeting
between 3 to 5 people (reviewers) should be involved
advance preparation should occur (< 2 hours work)
meeting should be < 2 hours
should be for a small part of the overall software
is attended by the producer, review leader and reviewers
must have a follow-up procedure to ensure any corrections are completed

33. Technical Reviews The Review Meeting
At the end of the review the attendees must decide whether to:
accept the work product without modification
reject the work product due to severe errors (once corrected another review is performed)
accept the work product provisionally (minor errors to be corrected but no further review)

34. Conducting a Review 1. be prepared—evaluate product before the review2.
review the product, not
the producer 3.
keep your tone mild, ask
questions instead of
making accusations 4.
stick to the review agenda 5.
raise issues, don't resolve them 6.
avoid discussions of style—stick to technical
correctness 7.
schedule reviews as project tasks 8.
record and report all review results

35. Technical Reviews The Review Report should contain:
- Type of Review, Team Members, Date, Time, Place
- Item being reviewed
- Agenda
- Checklist
- Comments per Checklist item
or per Issues list item (Pressman) with problems mentioned
Action per Comment - No action, Fix error, Reconsider
Decision for the item being reviewed
Accept, Reject (severe errors), Accept (minor errors)

36. Technical Reviews Review Guidelines
Review the product not the producer
Set an agenda and maintain it
Limit debate and rebuttal
Enunciate problem areas, but don’t attempt to solve every problem
Take written notes
Limit the number of participants and insist on advance preparation
Develop a checklist for each work product
Allocate resources and time schedule
Conduct meaningful training for all reviewers
Review earlier reviews

37. Software Inspection
The inspection process comprises three broad stages:
preparation
collection
repair
Gilb and Graham [GilbGraham93] expand this three stage process into the inspection steps; Entry, Planning, Kickoff Meeting, Individual Checking, Logging Meeting, Root Cause Analysis Edit, Follow Up, Exit.

38. Benefits of Software Inspection
30% to 100% net productivity increases;
Overall project time saving of 10% to 30%;
5 to 10 times reduction in test execution costs and time;
Reduction in maintenance costs of up to one order of magnitude;
Improvement in consequent product quality;
Minimal defect correction backlash at systems integration time.
In addition to these tangible benefits, less tangible benefits such as a training effect for inspectors are also evident.

39. Disadvantages Up front costs (although far outweighed by benefit):
Training
Implementation Support
Ongoing allocation of staff resources
Not strictly repeatable