Capability Maturity Model Integration: Implications for Quality Assurance and Testing Presentation to SCQAA 20 June 2002 Rick Hefner, TRW rick.hefner@trw.com (310) 812-7290 Capability Maturity Model Integrated (CMMI): Implications for QA and Testing   For over a decade, the Capability Maturity Model for Software has been a business imperative for improving and assessing software processes. The Department of Defense and Software Engineering Institute recently released a replacement to this model, the Capability Maturity Model Integrated. The new CMMI provides guidelines for organizational, management, engineering and support practices associated with software engineering, systems engineering, integrated product & process development, and source selection. Leading edge companies, serving both commercial and defense customers, have already started transitioning from the SW-CMM to CMMI. This presentation will provide an overview of the CMMI, including the model’s content and structure, assessment and certification methods, available training, and strategies for transition. Special emphasis will be paid to the role of the quality assurance and testing specialist in the new model. The presentation will help in answering the following questions. Should our company consider using the CMMI? What advantages/disadvantages does it provide? We only develop software – does the CMMI apply to us? We’re using the SW-CMM now – should we transition to CMMI? How long will it take? Where can I find additional resources and information? What does the CMMI mean to me – the QA and/or testing specialist? Rick Hefner, Ph.D. [rick.hefner@trw.com] Dr. Hefner has been a member of the CMMI development project since its inception, and currently serves on the Change Control Board. He has over 25 years of experience in software development, research, and management, and has authored over 50 papers and presentations. Dr. Hefner is an SEI-authorized Lead Assessor, and has performed over 40 assessments. He is the Manager of Process Initiatives at TRW, where his responsibilities include CMM and ISO-based improvement programs, training, metrics, and assessment.

Agenda Project History CMMI Structure Comparisons with SW-CMM v1.1, SE-CMM, and EIA/IS 731 Assessment Methodology Transition Timelines and Strategies Implications for Quality Assurance Professionals SCQAA - Rick Hefner

What is a CMM? Capability Maturity Model: A reference model of mature practices in a specified discipline -- industry best practices CMMs differ by Discipline (software, systems, acquisition, etc.) Structure (staged versus continuous) How Maturity is Defined (process improvement path) How Capability is Defined (institutionalization) “Capability Maturity Model®” and CMM® are used by the Software Engineering Institute (SEI) to denote a particular class of maturity models Capability Maturity Model®, CMM®, CMM Integration, and CMMI are service marks and registered trademarks of Carnegie Mellon University SCQAA - Rick Hefner

Commonly Used CMMs Software CMM staged software development System Engineering CMM continuous system engineering System Engineering Capability Model continuous system engineering Software Acquisition CMM staged software acquisition System Security Engineering CMM continuous security engineering Personal Software Process staged individual software development FAA-CMM continuous software engineering, systems engineering, and acquisition IPD-CMM continuous integrated product development People CMM staged workforce SPICE Model continuous software development SCQAA - Rick Hefner

What is the Problem? Different structures, formats, terms, ways of measuring maturity Causes confusion, especially when using more than one CMM Hard to integrate them in a combined improvement program Hard to use multiple models in supplier selection Systems Engr CMM IPD CMM FAA iCMM People CMM People CMM CMMI Software CMM Software Acq CMM ZZZ CMM Security Systems Engr CMM Systems Security Engr CMM SCQAA - Rick Hefner

<We could add organizational slides here for backup> The CMMI Project DoD sponsored collaboration between industry, Government, academia Over 100 people involved U.S. Army, Navy, Air Force Federal Aviation Administration National Security Agency Software Engineering Institute ADP, Inc. AT&T Labs BAE Boeing Computer Sciences Corporation EER Systems Ericsson Canada Ernst and Young General Dynamics Harris Corporation Honeywell KPMG Litton Lockheed Martin Motorola Northrop Grumman Pacific Bell Q-Labs Raytheon Rockwell Collins Sverdrup Corporation Thomson CSF TRW So, based on the problem and an understanding of the fundamental similarities of CMMs and CMM-based improvement methods, the CMMI project was born! <We could add organizational slides here for backup> SCQAA - Rick Hefner

CMMI Models Source Models Disciplines systems engineering software engineering integrated process and product development (IPPD) supplier sourcing combinations of the above Representations staged continuous Source Models Capability Maturity Model for Software V2, draft C (SW-CMM V2C) EIA Interim Standard 731, System Engineering Capability Model (SECM) Integrated Product Development Capability Maturity Model, draft V0.98 (IPD-CMM) SCQAA - Rick Hefner

Maturity Levels Each level is a layer in the foundation for continuous process improvement Level 5 Optimizing Focus on process improvement Level 4 Quantitatively Managed Process measured and controlled Level 3 Defined Process characterized for the organization and is proactive Level 2 Managed Process characterized for projects and is often reactive Level 1 Performed Process unpredictable, poorly controlled and reactive SCQAA - Rick Hefner

CMMI Process Areas Level Focus Process Areas Continuous process improvement Causal Analysis and Resolution Organizational Innovation and Deployment 5 Optimizing 4 Quantitatively Managed Quantitative management Quantitative Project Management Organizational Process Performance Organizational Process Focus Organizational Process Definition Organizational Training Integrated Project Management Risk Management Decision Analysis and Resolution Requirements Development Technical Solution Product Integration Verification Validation Process standardization 3 Defined Requirements Management Project Planning Project Monitoring and Control Supplier Agreement Management Measurement and Analysis Process and Product Quality Assurance Configuration Management 2 Managed Basic project management 1 Performed SCQAA - Rick Hefner

Process Area Group Relationships Process Management Organizational Process Focus Organizational Process Definition Organizational Training Organizational Process Performance Organizational Innovation and Deployment empower analyze standardize processes Project Management Project Planning Project Monitoring and Control Supplier Agreement Management Integrated Project Management Risk Management Quantitative Project Management Support Configuration Management Process and Product Quality Assurance Measurement and Analysis Decision Analysis and Resolution Causal Analysis and Resolution analyze Engineering Requirements Development Requirements Management Technical Solution Product Integration Verification Validation employ measure & assist SCQAA - Rick Hefner

Common Features For Each Process Area Commitment to Perform Establish an Organizational Policy Ability to Perform Plan the Process Provide Resources Assign Responsibility Train People Establish a Defined Process Directing Implementation Manage Configurations Identify and Involve Relevant Stakeholders Monitor and Control the Process Collect Improvement Information Verification Objectively Evaluate Adherence Review Status with Higher-Level Management SCQAA - Rick Hefner

CMMI Staged Model Structure Maturity Levels ... Process Area 1 Process Area 2 Process Area n Generic Goals Specific Goals Common Features Activities Performed Commitment to Perform Ability to Perform Directing Implementation Verification Specific Practices Generic Practices Specific Practices Specific Practices Specific Practices Specific Practices Subpractices Amplifications Elaborations Subpractices Amplifications Elaborations Subpractices Amplifications Elaborations Subpractices Amplifications Elaborations Subpractices Amplifications Elaborations Subpractices Amplifications Elaborations SCQAA - Rick Hefner

Required, Expected, Informative Maturity Levels ... Process Area 1 Process Area 2 Process Area n Generic Goals Specific Goals Required Needed to satisfy CMMI Expected Typical practices to meet goals; Used as a starting point in assessments Informative Ideas to consider Common Features Activities Performed Commitment to Perform Ability to Perform Directing Implementation Verification Specific Practices Generic Practices Specific Practices Specific Practices Specific Practices Specific Practices Subpractices Amplifications Elaborations Subpractices Amplifications Elaborations Subpractices Amplifications Elaborations Subpractices Amplifications Elaborations Subpractices Amplifications Elaborations Subpractices Amplifications Elaborations SCQAA - Rick Hefner

SW-CMM v1.1 vs. CMMI Process Areas Defect Prevention Causal Analysis and Resolution Technology Change Mgmt Organizational Innovation & Deployment Process Change Management Quantitative Process Mgmt Organizational Process Performance Software Quality Mgmt Quantitative Project Management Organization Process Focus Organization Process Focus Organization Process Definition Organization Process Definition Training Program Organizational Training Integrated Software Mgmt Integrated Project Management Risk Management Software Product Engr Requirements Development Technical Solution Product Integration Intergroup Coordination Verification Peer Reviews Validation Decision Analysis and Resolution Requirements Management Requirements Management Software Project Planning Project Planning Software Project Tracking & Oversight Project Monitoring and Control Software Subcontract Mgmt Supplier Agreement Management Software Quality Assurance Product & Process Quality Assurance Software Configuration Mgmt Configuration Management Measurement and Analysis LEVEL 5 OPTIMIZING LEVEL 4 MANAGED LEVEL 3 DEFINED LEVEL 2 REPEATABLE 14

SW-CMM v1.1 vs. CMMI Common Features 15

SE-CMM and SECM (EIA/IS 731) vs. CMMI Common Features - Engineering Technical Engineering PA06 Understand Customer Needs and Expectations PA02 Derive and Allocate Requirements PA03 Evolve System Architecture PA01 Analyze Candidate Solutions PA05 Integrate System PA07 Verify and Validate System PA04 Integrate Disciplines 1.1 Define Stakeholder and System Level Requirements 1.2 Define Technical Requirements 1.3 Define Solution 1.4 Assess and Select 1.5 Integrate System 1.6 Verify System 1.7 Validate System Requirement Mgmt Requirement Development Technical Solution Decision Analysis and Resolution Product Integration Verification Validation 2.3 SCQAA - Rick Hefner

SE-CMM and SECM (EIA/IS 731) vs SE-CMM and SECM (EIA/IS 731) vs. CMMI Common Features - Project Management SE-CMM SECM CMMI Project Management Project Management PA12 Plan Technical Effort PA11 Monitor and Control Technical Effort PA10 Manage Risk PA09 Manage Configurations PA08 Ensure Quality Project Planning Project Mon and Con Supplier Agree Mgt Integrated Proj Mgt Risk Management 2.1 Plan and Organize 2.2 Monitor and Control 2.3 Integrate Disciplines 2.4 Coordinate with Suppliers 2.5 Manage Risk 2.6 Manage Data 2.7 Manage Configurations 2.8 Ensure Quality PA04 PA18 Support Configuration Mgt Proc and Prod QA Meas and Analysis * L2 GP * Not in SE-CMM SCQAA - Rick Hefner

SE-CMM and SECM (EIA/IS 731) vs SE-CMM and SECM (EIA/IS 731) vs. CMMI Common Features - Process Management SE-CMM SECM CMMI Organization Environment Process Management PA13 Define Organization’s SE Process PA14 Improve Organization’s SE Process PA17 Provide Ongoing Knowledge and Skills PA15 Manage Product Line Evolution PA16 Manage SE Support Environment PA18 Coordinate With Suppliers 3.1 Define and Improve the SE Process 3.2 Manage Competency 3.3 Manage Technology 3.4 Manage SE Support Environment Org Process Focus Org Process Def Organizational Tng Quant Project Mgt Org Process Perf Causal An & Res Org Innovation and Deployment (IPPD Extension) Organizational Environment Integration L4 GP L4 GP L5 GP 2.4 SCQAA - Rick Hefner

Summary of Changes Organizations that are currently using SW-CMM v1.1, SE-CMM, or EIA/IS 731 should be able to smoothly transition Will require additional infrastructure Identification and involvement of relevant stakeholders Configuration management of intermediate work products Will require additional processes Measurement and Analysis Decision Analysis and Resolution Requirements Development Risk Management Additional focus on engineering processes Software organizations adopting CMM-based improvements for the first time should use the CMMI SCQAA - Rick Hefner

Appraisal Requirements for CMMI (ARC) Similar to the current CMM Appraisal Framework (CAF) V1.0 A guide to assessment method developers Specifies the requirements for classes of assessment methods Class A: Full, comprehensive assessment methods Class B: Initial, incremental, self-assessments Class C: Quick-look Method developers can declare which class their method fits Method buyers will understand the implications of the method proposed SCQAA - Rick Hefner

Standard CMMI Assessment Method for Process Improvement (SCAMPI) v1.1 Similar to CBA IPI method Led by authorized Lead Assessor Team of 4-9 trained assessors Tailorable to organization and model scope Products: SCAMPI Method Description Document Maturity questionnaire(s), electronic work aids, templates The method was updated to version 1.1 to improve efficiency SCQAA - Rick Hefner

CMMI Transition Plan Development Phase Transition Phase Development of CMMI products Verification and validation of CMMI products Transition Phase Approval of initial CMMI products for public release Evidence of sufficient use Transition planning to help organizations use CMMI products Sustainment Phase Upkeep and continuous improvement of the product suite Additional evidence of adoption and use Pilots V0.2 Aug 1999 V1.0 Aug 2000 V1.1 Dec 2001 SW-CMM, EIA 731 phased out Aug 2003 SCQAA - Rick Hefner

Role of the Quality Assurance Professional QA has a defined role in every process area, and a process area of their own SCQAA - Rick Hefner

Generic Practices GP 2.9 (VE1) Objectively Evaluate Adherence Objectively evaluate adherence of the process and the work products and services of the process to the applicable requirements, objectives, and standards, and address noncompliance. Quality assurance must be performed in all processes, work products, and services Engineering Support Project Management Organizational process management SCQAA - Rick Hefner

Process and Product Quality Assurance - 1 Purpose Objectively review activities and work products for their adherence to applicable requirements, process descriptions, standards, and procedures, and communicate the results to staff and management. P&PQA ensures planned processes are implemented; Verification ensures the specified requirements are satisfied. QA can be embedded in the process, but: Objectivity must be maintained All QA performers must be trained An independent reporting channel must be maintained QA should participate in establishing the plans, processes, standards and procedures to ensure they: Fit the project’s needs Will be useable for performing quality assurance evaluations. SCQAA - Rick Hefner

Process and Product Quality Assurance - 2 SG 1 Objectively Evaluate Processes and Work Products SP 1.1 Objectively Evaluate Processes Objectively evaluate the designated performed processes against the applicable process descriptions, standards and procedures. SP 1.2 Objectively Evaluate Work Products and Services Objectively evaluate the designated work products and services against the applicable process descriptions, standards, and procedures. Projects must designate which processes and work products they intend to audit What processes and work products drive overall quality? Where is insight into quality the most valuable? Assessor typically expect all processes to be audited SCQAA - Rick Hefner

Process and Product Quality Assurance - 3 SG 2 Provide Objective Insight SP 2.1 Communicate and Ensure Resolution of Noncompliance Issues Communicate quality issues and ensure resolution of noncompliance issues with the staff and managers. SP 2.2 Establish Records Establish and maintain records of the quality assurance activities. Requires a independent reporting channel for noncompliance issues Makes senior management responsible for adjudicating quality versus schedule conflicts SCQAA - Rick Hefner

Process and Product Quality Assurance - 4 Quality assurance activities must be fully supported by the project and organizational infrastructure GG 2 Institutionalize a Managed Process GP 2.1 (CO 1) Establish an Organizational Policy GP 2.2 (AB 1) Plan the Process GP 2.3 (AB 2) Provide Resources GP 2.4 (AB 3) Assign Responsibility GP 2.5 (AB 4) Train People GP 2.6 (DI 1) Manage Configurations GP 2.7 (DI 2) Identify and Involve Relevant Stakeholders GP 2.8 (DI 3) Monitor and Control the Process GP 2.9 (VE 1) Objectively Evaluate Adherence GP 2.10 (VE 2) Review Status with Higher-Level Management SCQAA - Rick Hefner

Software requirements Verification - 1 Purpose Assure that selected work products meet the specified requirements (e.g., peer reviews, testing) Key Issues Recognizing verification is an incremental process that occurs throughout the development of the product and work products Verification of the Requirements Verification of the evolving work products Verification of the completed product Identifying where verifications will occur (as part of program strategy) User needs System requirements Software requirements Software design Software code Test cases Test results Code testing Code peer review SCQAA - Rick Hefner

Verification - 2 SG 1 Prepare for Verification Preparation for verification is conducted. SP 1.1 Select Work Products for Verification Select the work products to be verified and the verification methods that will be used for each. SP 1.2 Establish the Verification Environment Establish and maintain the environment needed to support verification. SP 1.3 Establish Verification Procedures and Criteria Establish and maintain verification procedures and criteria for the selected work products. Projects identify (and document) the work products they intend to verify Verification environment typically refers to a test bed Could also imply the facilities and tools needed for peer reviews “Procedures” are not typically called out in the CMMI – implies the need for more detailed planning SCQAA - Rick Hefner

Verification - 3 Different verification methods: peer review, inspection, demonstration, test, analysis Preparing for peer review includes identification of: Staff involved, roles Review criteria Schedule Peer review data are analyzed to improve review effectiveness Size of the product Composition of review team Type of peer review Preparation time per reviewer Length of the review meeting Number of defects found, type and origin of defect, etc. SG 2 Perform Peer Reviews Peer reviews are performed on selected work products. SP 2.1 Prepare for Peer Reviews Prepare for peer reviews of selected work products. SP 2.2 Conduct Peer Reviews Conduct peer reviews on selected work products and identify issues resulting from the peer review. SP 2.3 Analyze Peer Review Data Analyze data about preparation, conduct, and results of the peer reviews. SG 3 Verify Selected Work Products Selected work products are verified against their specified requirements. SP 3.1 Perform Verification Perform verification on the selected work products. SP 3.2 Analyze Verification Results and Identify Corrective Action Analyze the results of all verification activities and identify corrective action. SCQAA - Rick Hefner

Verification - 4 Verification activities must be fully supported by the project and organizational infrastructure GG 2 Institutionalize a Managed Process GP 2.1 (CO 1) Establish an Organizational Policy GP 2.2 (AB 1) Plan the Process GP 2.3 (AB 2) Provide Resources GP 2.4 (AB 3) Assign Responsibility GP 2.5 (AB 4) Train People GP 2.6 (DI 1) Manage Configurations GP 2.7 (DI 2) Identify and Involve Relevant Stakeholders GP 2.8 (DI 3) Monitor and Control the Process GP 2.9 (VE 1) Objectively Evaluate Adherence GP 2.10 (VE 2) Review Status with Higher-Level Management SCQAA - Rick Hefner

Implications for the Quality Assurance Professional More companies are adopting the CMMI (and CMM) Required by customers Improves predictability and efficiency; reduces rework and cost A competitive discriminator The CMMI/CMM requires a significant QA presence Involvement in all projects, all processes Significant support required to satisfy PPQA process area CMMI/CMM provides an opportunity to educate the community on the proper role of QA SCQAA - Rick Hefner

Summary CMMI is a natural evolution in industry maturity Organizations that are currently using SW-CMM v1.1, SE-CMM, or EIA/IS 731 should be able to smoothly transition Quality assurance professionals have an important role in the CMMI See http://www.processimprovement.com for more information and resources SCQAA - Rick Hefner