1. VanQ - January 25, 2007© Josip Pajk 2007Page 1 SW Quality Assurance in CMMI ® VanQ meeting January 25, 2007 Josip Pajk

2. VanQ - January 25, 2007© Josip Pajk 2007Page 2Agenda ● Brief overview of the CMMI ® ● Testing & Quality Assurance Process Areas ● Some Considerations and Examples ● Q&A ● Discussion CMMI, CMM, and Capability Maturity Model are registered in the U.S. Patent and Trademark Office. CMM Integration, SCAMPI, and IDEAL are service marks of Carnegie Mellon University. http://www.sei.cmu.edu/cmmi/cmmi.html

3. VanQ - January 25, 2007© Josip Pajk 2007Page 3 What is CMMI ® ?

4. VanQ - January 25, 2007© Josip Pajk 2007Page 4 1 Process Framework

5. VanQ - January 25, 2007© Josip Pajk 2007Page 5 22 Process Areas in 4 Categories

6. VanQ - January 25, 2007© Josip Pajk 2007Page 6 6 Process Capability Levels 0. Incomplete 0. Incomplete - a process that is either not performed or is partially performed. 1. Performed 1. Performed - a process that supports and enables the work needed to produce work products. 2. Managed documented 2. Managed - a performed (capability level 1) process that is planned, executed and tracked in accordance with a (documented) plan, policy or procedure. SG 1 Objectively Evaluate Processes and Work Products SP 1.1Objectively Evaluate Processes SP 1.2Objectively Evaluate Work Products and Services SG 2 Provide Objective Insight SP 2.1Communicate and Ensure Resolution of Noncompliance Issues SP 2.2Establish Records

7. VanQ - January 25, 2007© Josip Pajk 2007Page 7 Capability Levels (cont) 3. Defined tailoredstandard processes organizational process assets 3. Defined - is a managed (capability level 2) process that is tailored from the organization’s set of standard processes and contributes work products, measures, and other process improvement information to the organizational process assets. 4. Quantitatively Managed special causes of variation 4. Quantitatively Managed - a defined (capability level 3) process that is controlled by establishing quantitative objectives for quality and process performance which is understood in statistical terms and managed by eliminating special causes of variation. 5. Optimizing common causes of variation 5. Optimizing - a quantitatively managed (capability level 4) process that is improved based on an understanding of the common causes of variation inherent in the process.

8. VanQ - January 25, 2007© Josip Pajk 2007Page 8 5 Organizational Maturity Levels Evolutionary Plateau organization A Maturity Level is an “Evolutionary Plateau” for the organization on its way of process improvement. =O= 1. Initial competence and heroics of the people inability to repeat their successes 1. Initial - Success in these organizations depends on the competence and heroics of the people in the organization. Organizations are characterized by a tendency to over commit, abandonment of processes in a time of crisis, and an inability to repeat their successes. 2. Managed documented plans process descriptions, standards, and procedures 2. Managed - Processes are planned and executed monitored, controlled, and reviewed in accordance with documented plans; involve relevant stakeholders; and are evaluated for adherence to their process descriptions. Work products and services are appropriately controlled and satisfy their specified process descriptions, standards, and procedures.

9. VanQ - January 25, 2007© Josip Pajk 2007Page 9 Maturity Levels (Cont.) 3. Definedorganization’s set of standard processes defined processestailoring 3. Defined - The organization’s set of standard processes are described in standards, procedures, tools, and methods. Standard processes are used to establish consistency across the organization and different projects. Projects establish their defined processes by tailoring the organization’s set of standard processes. 4. Quantitatively Managed quantitatively predictable 4. Quantitatively Managed - At maturity level 3, processes are typically only qualitatively predictable while at maturity level 4, the performance of processes is controlled using statistical and other quantitative techniques, and are quantitatively predictable. 5. Optimizing understanding of the common causes of variation 5. Optimizing – The organization continually improves its processes based on a quantitative understanding of the common causes of variation inherent in processes.

10. VanQ - January 25, 2007© Josip Pajk 2007Page 10 Generic Goals & Practices 3 Institutionalize a Defined Process 3.1 Establish a Defined Process 3.2 Collect Improvement Information 4 Institutionalize a Quantitatively Managed Process 4.1 Establish Quantitative Objectives for the Process 4.2 Stabilize Subprocess Performance 5 Institutionalize an Optimizing Process 5.1 Ensure Continuous Process Improvement 5.2 Correct Root Causes of Problems 1 Achieve Specific Goals 1.1 Perform Specific Practices 2 Institutionalize a Managed Process 2.1 Establish an Organizational Policy 2.2 Plan the Process 2.3 Provide Resources 2.4 Assign Responsibility 2.5 Train People 2.6 Manage Configurations 2.7 Identify and Involve Relevant Stakeholders 2.8 Monitor and Control the Process 2.9 Objectively Evaluate Adherence 2.10 Review Status with Higher Level Management Are inclusive and define both the Capability level of a PA as well as the Maturity Level of the organization

11. VanQ - January 25, 2007© Josip Pajk 2007Page 11 Organizational Assessments MLs The MLs are defined by a # of Generic Goals Practices Generic Goals and Generic Practices Process Area Each Process Area is described by a # of Specific GoalsSpecific Practices Specific Goals and Specific Practices 2 3

12. VanQ - January 25, 2007© Josip Pajk 2007Page 12 QA & Testing the CMMI ® Way

13. VanQ - January 25, 2007© Josip Pajk 2007Page 13 Verification and Validation (ENG) iterative (1) (2) (3) requirement (1) A condition or capability needed by a user to solve a problem or achieve an objective. (2) A condition or capability that must be met or possessed by a product or product component to satisfy a contract, standard, specification, or other formally imposed documents. (3) A documented representation of a condition or capability as in (1) or (2). any product or service The Engineering process areas apply to the development of any product or service in the development domain (e.g., software products, hardware products, services, or processes). ML3 ML2

14. VanQ - January 25, 2007© Josip Pajk 2007Page 14VERIFICATION Verification (VER) work products The purpose of Verification (VER) is to ensure that selected work products meet their specified requirements. Ensures “you built it right.” work productany useful result of a process is not necessarily part of the deliverable product work product - any useful result of a process. Can include files, documents, products, parts of a product, services, process descriptions, specifications, and invoices. A key distinction between a work product and a product component is that a work product is not necessarily part of the deliverable product.

15. VanQ - January 25, 2007© Josip Pajk 2007Page 15VALIDATION Validation (VAL) product or product component in its intended environment. The purpose of Validation (VAL) is to demonstrate that a product or product component fulfills its intended use when placed in its intended environment. Ensures “you built the right thing.” productintended for delivery product- a work product that is intended for delivery to a customer or end user. product component integrated to produce the product product component- a work product that is a lower level component of the product. Product components are integrated to produce the product. There may be multiple levels of product components.

16. VanQ - January 25, 2007© Josip Pajk 2007Page 16 Process & Product QA (SUP) quality inherent characteristics of a product qualityThe ability of a set of inherent characteristics of a product, product component, or process to fulfill requirements (and expectations) of customers. quality assurance quality assuranceA planned and systematic means for assuring management (and the customer) that the defined (product requirements and) standards, practices, procedures, and methods of the (development) process are applied. (additions are mine) Note: quality standards and procedures are NOT defined by QA Advanced PAs ML3&5 Basic PAs ML2

17. VanQ - January 25, 2007© Josip Pajk 2007Page 17 Process & Product QA (2) objective insight processes and associated work products The purpose of Process and Product Quality Assurance (PPQA) is to provide staff and management with objective insight into processes and associated work products. PPQA involves the following: – Objectively evaluating – Objectively evaluating performed processes, work products, and services against the applicable process descriptions, standards, and procedures noncompliance – Identifying and documenting noncompliance issues feedback – Providing feedback to project staff and managers on the results of quality assurance activities – Ensuring that noncompliance issues are addressed

18. VanQ - January 25, 2007© Josip Pajk 2007Page 18 Relevant Specific Practices PPQA VER VER and PPQA are applied on work products while VAL is performed on products. VAL

19. VanQ - January 25, 2007© Josip Pajk 2007Page 19 Objective Evaluation objectively evaluate minimize subjectivity and bias independent quality assurance function objectively evaluate - To review activities and work products against criteria which minimize subjectivity and bias by the reviewer. An example of an objective evaluation is an audit against requirements, standards, or procedures by an independent quality assurance function. in an organization with an open, quality-oriented culture, the process and product quality assurance role may be performed, partially or completely, by peers It may be appropriate in some organizations, however, to implement the process and product quality assurance role without that kind of independence. For example, in an organization with an open, quality-oriented culture, the process and product quality assurance role may be performed, partially or completely, by peers; and the quality assurance function may be embedded in the (development? management?) process. Peer review ?

20. VanQ - January 25, 2007© Josip Pajk 2007Page 20 Objective Evaluation (2) if quality assurance is embedded in the process Issues that must be addressed to ensure objectivity if quality assurance is embedded in the process: trained in quality assurance ● Everyone performing quality assurance activities should be trained in quality assurance. separate ● Those performing quality assurance activities for a work product should be separate from those directly involved in developing or maintaining the work product. independent reporting channel ● An independent reporting channel to the appropriate level of organizational management must be available so that noncompliance issues can be escalated as necessary. Peer Reviews ?

21. VanQ - January 25, 2007© Josip Pajk 2007Page 21 Examples and Some Final Considerations

22. VanQ - January 25, 2007© Josip Pajk 2007Page 22 Example 1- All in One product-testing dutiesassure specified guidelines are being followed procedural or processing deficiencies programs conform to documentation specifications develop, apply and maintain quality requirements The quality assurance engineer performs various product-testing duties to assure specified guidelines are being followed. The engineer begins corrective action for procedural or processing deficiencies by making sure programs conform to documentation specifications; certifies that products are bug free and stable and works to develop, apply and maintain quality requirements that include the creation and execution of methods and procedures for testing and debugging programs. potential problems SWAT Understand and follow agile methodologies and software engineering practices / Develop and execute test plans and test cases according to requirements documents / Perform black box, white box, regression, and load testing on test units / Work with developers for clarification in resolving issues / Participate in design and code reviews / Write user procedures and participate in system testing and training / Notify group leader of any problems or potential problems as they may arise / Participate in SWAT Team activities as assigned.

23. VanQ - January 25, 2007© Josip Pajk 2007Page 23 Example 2 – Mainly Testing contributing to a test plancreating and maintaining test scenariosfor a given test type A SIT Tester is the individual responsible for contributing to a test plan, and creating and maintaining test scenarios and test cases for a given test type. As required, a SIT Tester is also expected to contribute to the ongoing improvement of the testing process and to maintain current knowledge of testing tools and techniques. providing feedback to developers relating to the testability of requirements project team approval on all assigned testing deliverables Record defects when actual SIT results do not match SIT expected resultstesting-related defects, issues and risks Analyze requirements, providing feedback to developers relating to the testability of requirements / Prepare SIT test estimates for assigned change requests as deemed appropriate / Determine test environment requirements including test data, physical location, user access, change control and any training needs / Obtain project team approval on all assigned testing deliverables (e.g., scenarios, cases, data, and where applicable, scripts) / Execute test cases for the assigned test type according to the test plan (if a test plan is available and signed off) / Turn over testing deliverables to specified testers / Record defects when actual SIT results do not match SIT expected results / Identify and report all SIT testing-related defects, issues and risks / Record required process and product measurements

24. VanQ - January 25, 2007© Josip Pajk 2007Page 24 Example 3 – Mainly QA? objectively implement the elements of the Quality Program training of plant personnel in the application of the quality program ensure internal and external customer requirements are met develop and initiate preventative corrective actions Quality Engineer / Quality Assurance Specialist / Define and establish the necessary systems, practices, methods and tooling to objectively implement the elements of the Quality Program / Prepare product quality and control plans / Assist in training of plant personnel in the application of the quality program/new procedures/instructions / Assist the Quality Assurance Manager in the implementation of continuous process improvement methodology / Analyze statistical data used to monitor quality performance and improvement, to ensure internal and external customer requirements are met / Investigate product/process quality issues and develop and initiate preventative corrective actions / Conduct external supplier audits when required / Maintain good relationships with all functional groups and locations within the company by participating in functional and multi-functional teams / Attend various multi-disciplinary meetings as member and/or QA Manager back up / Maintain good relationships and liaison with external customers and suppliers on quality related matters / Adhere to all health and safety rules and procedures / Any direction from Manager to be followed / Maintenance and housekeeping of work area.

25. VanQ - January 25, 2007© Josip Pajk 2007Page 25 Testing ≡ QA ? objectively evaluating test ● Who is then responsible for objectively evaluating test work products? ● Customers? ● Peers? – Other testers? – Developers, SE, PM? subjective stakeholders ● They are subjective stakeholders (evaluation criteria?) multi-subjectivity = objectivity ● Is multi-subjectivity = objectivity? processes ● Also, who is responsible for the evaluation (audit) of development (including testing) processes?

26. VanQ - January 25, 2007© Josip Pajk 2007Page 26 QA ≡ Testing ? work products processes ● Evaluation (testing?) of work products and processes (peer review enough?) – Pair programming (and testing) ? – Test Driven Development ? ● What about creativity? is must ● Development (including testing) is creative while QA must be bureaucratic! – Do we really want moving quality criteria (not only requirements)? ● Again, who would you choose to evaluate your work?

27. VanQ - January 25, 2007© Josip Pajk 2007Page 27 The Cost View (P. Crosby) ● Cost of Performance (CoP) ● Cost of Performance (CoP) – when everything is done properly the first time (no failures whatsoever) ● Cost of Quality (CoQ) ● Cost of Quality (CoQ) - extra cost because we are not sure it was done right the first time – Cost of Conformance (CoC) ● Cost of Apraisal (CoA) ● Cost of Apraisal (CoA) – Product Reviews, Testing (only the first time, not re-testing) ● Cost of Prevention (CoP) ● Cost of Prevention (CoP) – Process reviews, Training, Process Improvement. – Cost of Failure or Non-conformance (CoN) – Cost of Failure or Non-conformance (CoN) – Any rework (whenever we find a bug) CoQ=CoC+CoN

28. VanQ - January 25, 2007© Josip Pajk 2007Page 28 Cost of Quality From: Schiffauerova, A. and Thomson, V., “A review of research on cost of quality models and best practices”, International Journal of Quality and Reliability Management, Vol.23, No.4, 2006 http://www.mcgill.ca/files/mmm/CoQModels-BestPractices.pdf Traditional view Modern view

29. VanQ - January 25, 2007© Josip Pajk 2007Page 29 Questions ?

30. VanQ - January 25, 2007© Josip Pajk 2007Page 30Discussion ● Should QA be independent from Engineering and PM? ● Is quality just in the “eye of the beholder” (the user)? ● Can we assure quality by following some particular process? ● Is QA a management tool? ● How much QA is enough?

31. VanQ - January 25, 2007© Josip Pajk 2007Page 31 Thank You I would love to hear from you jpajk@telus.net