1. CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE7315M34 Slide 1 January 10, 2001 SMU CSE 7315 / NTU SE 584-N Planning and Managing a Software Project Module 34 Software Quality Assurance

2. CSE7315M34 Slide # 2 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Objectives of This Module To examine the techniques of software quality assurance To look at some techniques of software quality engineering – Notably, cost of quality analysis

3. CSE7315M34 Slide # 3 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Software Quality Engineering and Assurance References to text: Managing Quality16 Engineering Quality9,10 Software Testing11 Other Relevant Material8

4. CSE7315M34 Slide # 4 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Evolution of Software Quality Quality Control Quality Assurance Quality Engineering This Module Introduced in This Module

5. CSE7315M34 Slide # 5 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Software Quality Assurance Purposes: – To provide management and developers with visibility into the process being followed and the products being built So they can manage the outcome – To provide a more effective quality control mechanism

6. CSE7315M34 Slide # 6 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Software Quality Assurance Typical Practices: – Inspections – Reviews – Audits – Communication – Measurements – Independent confirmation of compliance Standards, requirements, procedures

7. CSE7315M34 Slide # 7 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Quality Assurance Goal: To assure that quality is achieved -- Add quality during development -- Improve processes, standards, and up- front evaluations of the product as it is being developed Quality Control Quality Assurance Quality Engineering

8. CSE7315M34 Slide # 8 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Quality Assurance Looks at the Entire Process Requirements DevelopmentQC Inspection Pass Fail Standards of Quality Process and Design Standards

9. CSE7315M34 Slide # 9 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Each Step of the Process Has an Effect on The Final Defect Rate Process Step I = Defects Input O = Defects Output F = Defects Found and Fixed C = Defects Created O = I + C - F

10. CSE7315M34 Slide # 10 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Typical Responsibilities of QA Review all development plans for completeness Participate as inspection moderators in design and code inspections Review a significant sample of all test results to determine adherence to standards Periodically audit SCM performance to determine adherence to standards

11. CSE7315M34 Slide # 11 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved At Each Step, QA must... Inspect the product for – Conformance with standards – Satisfaction of requirements Evaluate the process for – Conformance with standards – Opportunities to improve – Process defects that may cause problems later on Evaluate the support processes as well

12. CSE7315M34 Slide # 12 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved At Each Step, QA must... Monitor reviews, inspections, walkthroughs, etc. to see that they are accomplishing their objectives Trace back to prior steps when defects are found In addition to evaluating the development process, the QA procedures must also be tailored and revised for the needs of each new project

13. CSE7315M34 Slide # 13 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved At the Requirements Analysis Step... Trace software requirements or specifications to original system or customer requirements Inspect or evaluate software requirements against standards to see that they are -- complete, -- correct, -- consistent, -- testable, and -- understandable “Clean Room” technique takes this farther PROGRAM SAMPLE (IN1,IN2) INTEGAR IN1,IN2,A,B,C; FOR I = 1 STEP 3 UNTIL 99 DO IF IN1 < A*B THEN IN2 := C + A*B ELSE IN2 := C + IN1 ENDIF ENDDO ** THE NEXT PART WILL SORT DO FOREVER Ii := I + 1 TEMP := ARRAY[I] - ARRAY[I+1] IF TEMP < 0, SWAP (ARRAY, i) ENDLOOP CALL CHECKORDER(ARRAY) ** NEXT WE DO THE i/o CALL PRINT(ARRAY) CALL DEBUG (ARRAY, I) CASE I OF CALL SUB1

14. CSE7315M34 Slide # 14 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved “Clean Room” Software Development Technique Motive: don’t let the bugs get in Method: filter them out from the beginning -- Define the requirements using a formal notation – Reduces ambiguity – Some consistency/correctness issues can be checked automatically -- Carry out rigid inspections Benefits: fewer problems later in the development process (e.g., reduced testing).

15. CSE7315M34 Slide # 15 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved At the Design Step... Trace the design to the requirements specification (plus “derived requirements”) Evaluate the design against standards -- complexity -- cohesion -- understandability -- etc. Evaluation of design effectiveness, correctness, consistency, completeness Evaluate plans for testing and test cases

16. CSE7315M34 Slide # 16 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved At the Coding Step... Trace code to design specifications Evaluate code against coding standards (example: give code to a new maintainer and see if she or he can understand it) Code walkthroughs or inspections to check on coding mistakes Test coverage analysis -- Do the tests address all requirements? (black box tests) -- Do the tests adequately evaluate the design and its implementation? (glass box tests)

17. CSE7315M34 Slide # 17 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved At the Module Testing Step... Test procedures and code analysis -- Evaluate test code like any other code -- Are the procedures documented? -- Are the expected results documented? Test results analysis -- Were the tests run? -- Were the results correct?

18. CSE7315M34 Slide # 18 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Two Philosophies of Testing Code Test a Little Review Thorough Test CodeReview Thorough Test

19. CSE7315M34 Slide # 19 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved During Integration... Make sure that regression tests are performed -- Retest all modules potentially affected by any change Do independent tests -- Someone other than the person who developed the software Formal qualification tests (acceptance tests)

20. CSE7315M34 Slide # 20 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved When Do You Perform the Formal Qualification Test? Hardware Development Operating System Development Software Development System Integration System FQT Software FQT: Here or Here

21. CSE7315M34 Slide # 21 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Planning for QA Each development project should have a documented Software Quality Assurance Plan (SQAP) – May be part of SW Development Plan The SQAP documents tasks to be performed, standards of verification, procedures and organizational structure There is an IEEE standard for SQAPs

22. CSE7315M34 Slide # 22 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Pitfalls in Implementing QA Lack of sufficiently skilled staff in QA – Software professionals are typically more interested in development jobs than QA QA management cannot effectively negotiate with the development team Senior management commitment may fade under schedule pressures – The developers stop caring about QA & action revolves around insignificant issues

23. CSE7315M34 Slide # 23 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Pitfalls in Implementing QA (Continued) QA organization operates without suitably documented and approved development standards and procedures Development team does not produce verifiable quality plans – Eventually results in a debate over which bugs are more important to fix rather than whether the product has sufficient quality or not

24. CSE7315M34 Slide # 24 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Pitfalls in Implementing QA (continued) Disputes over who has responsibility for product quality and for performing QA functions – The software developers and managers are responsible for product quality – Quality assurance is a method to help them achieve quality through independent evaluation of their work (like a coach)

25. CSE7315M34 Slide # 25 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Reporting Chain for QA Staff Except in the highest maturity organizations, it is strongly recommended that the QA staff have a separate reporting chain from the project team – This improves their objectivity – And assures that disputes are resolved at a level in the organization that has responsibility for the possible consequences

26. CSE7315M34 Slide # 26 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Benefits Achieved by Quality Assurance over Quality Control Significant insight is gained into the process that can be used to gain long term benefits – Development can focus on specific weak points in the quality – Process improvements can be identified – Steps that are difficult to perform can be automated

27. CSE7315M34 Slide # 27 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Problems with Quality Assurance Can still be adversarial (although perhaps less than with quality control) Motivation to fix the process is still weak Can be more costly than quality control It can be hard to show benefits until long after the product has been shipped

28. CSE7315M34 Slide # 28 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Quality Engineering Goal: Build quality in as part of the software engineering process A philosophical change that relies on the professional pride of the software engineer Engineering staff define and execute the quality assurance tasks Team approach to quality, with rewards based on quality Quality professionals are more like coaches and educators than evaluators Quality Control Quality Assurance Quality Engineering

29. CSE7315M34 Slide # 29 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Quality Engineering Focuses on People As Well As Process Finding errors is good -- it keeps them from leaking through to the customer Problems result in process changes, not punishment of people Everyone appreciates that a competitive process is the way to remain a competitor Measurements are used so that decisions are based on fact (in addition to intuition) Independent tests are welcome -- they tell you if you are as good as you think you are

30. CSE7315M34 Slide # 30 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Peer Reviews Motivation: how to find problems in your work in a non-threatening way Method: evaluation of one individual’s work by others at the same organizational level (i.e., team members, not supervisors) Software DevelopersSoftware Managers

31. CSE7315M34 Slide # 31 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Peer Reviews - Cost Cost: may require 10-20% of the total development time -- this must be planned into the schedule -- it must also be planned into progress measures and reward systems

32. CSE7315M34 Slide # 32 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Peer Reviews - Benefits Benefits: reduced rework and faster development due to a more effective evaluation -- the peers understand the software best -- and they will not be as much of a threat when it comes to promotions and raises -- reciprocity -- you help them and they help you

33. CSE7315M34 Slide # 33 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Big Problem with Peer Reviews People don’t want to attend others’ peer reviews because they are behind in their own schedules Solutions? Class discussion

34. CSE7315M34 Slide # 34 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Benefits of the Quality Engineering Approach Less adversarial Motivation and information to improve Flexibility to change the process in response to a problem -- you understand the problem and its cause -- you understand the consequences of a change in the process Knowledge is the foundation of successful quality engineering

35. CSE7315M34 Slide # 35 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved The Cost of Quality Engineering Your process may be more complex, costly and time consuming - at least when you start But you gain knowledge -- what and where to automate -- how and where to improve the process And you gain higher quality And you can begin to optimize the process, just as you optimize software

36. CSE7315M34 Slide # 36 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Cost of Quality Analysis

37. CSE7315M34 Slide # 37 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Quality (Fewer Defects; Customer Satisfaction) Productivity Cycle Time Customer Value Quality Improvements Cost Money How can we justify them?

38. CSE7315M34 Slide # 38 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Managers and Technical Staff must be convinced that Quality problems are serious Poor quality costs them money Quality is worth fixing Quality can be fixed by proper techniques In Order to Justify Quality Improvements...

39. CSE7315M34 Slide # 39 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved The Fundamental Prejudice Quality improvement is a non-value- added activity – It costs overhead resources – The benefits are not necessarily real So we seek to avoid it Cost of Quality (COQ) Analysis is used to address this prejudice

40. CSE7315M34 Slide # 40 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved The First Question This is what management and technical staff will typically ask when approached with a proposal to improve quality “What does it cost to improve quality?”

41. CSE7315M34 Slide # 41 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved The Right Questions “What’s the payoff?” “What is the return on investment for quality improvements?

42. CSE7315M34 Slide # 42 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Categorizing Quality-Related Costs Quality Related Costs Cost of Conformance Cost of Non- Conformance Prevention Costs Appraisal Costs Failure Costs

43. CSE7315M34 Slide # 43 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Cost of Conformance Things that improve quality – Prevention costs - prevent poor quality Predictive metrics, training, root cause analysis, process improvement – Appraisal costs - detect poor quality Tests, inspections, audits

44. CSE7315M34 Slide # 44 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Cost of Non-Conformance The price you pay when you fail to achieve quality – Internal failures Costs incurred before product shipment – External failures Costs incurred after product shipment

45. CSE7315M34 Slide # 45 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Effects of Maturity on Costs SEI Maturity Level Cost as a Percent of Development Cost 10 20 30 40 50 60 1 2 34 5 Prevention Appraisal Internal Failures External Failures Total COQ As reported by Knox (see references)

46. CSE7315M34 Slide # 46 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved Module Summary Quality Control focuses on the end product Quality Assurance focuses on the process Quality Engineering focuses on process and people, integrating QA with the software engineering process

47. CSE7315M34 Slide # 47 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved References Knox, 1993, Raytheon studies reported by Houston and Keats, Software Quality Matters, vol 5, no 1 (Spring, 1997), U. of Texas SW Quality Institute Musa, John D, 1992, “The Operational Profile,” in Software Reliability Engineering: An overview. Weinberg, Gerald M., 1992, Quality Software Management, Volume 1, Systems Thinking, Dorset House, New York, ISBN 0-932633-22-6.

48. CSE7315M34 Slide # 48 January 10, 2001 CSE 7315 - SW Project Management / Module 34 - Software Quality Assurance Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved END OF MODULE 34