1. Capability Maturity Model Integration: Implications for Quality Assurance and Testing Presentation to SCQAA 20 June Rick Hefner, TRW (310)
Capability Maturity Model Integrated (CMMI): Implications for QA and Testing
For over a decade, the Capability Maturity Model for Software has been a business imperative for improving and assessing software processes. The Department of Defense and Software Engineering Institute recently released a replacement to this model, the Capability Maturity Model Integrated. The new CMMI provides guidelines for organizational, management, engineering and support practices associated with software engineering, systems engineering, integrated product & process development, and source selection. Leading edge companies, serving both commercial and defense customers, have already started transitioning from the SW-CMM to CMMI.
This presentation will provide an overview of the CMMI, including the model’s content and structure, assessment and certification methods, available training, and strategies for transition. Special emphasis will be paid to the role of the quality assurance and testing specialist in the new model.
The presentation will help in answering the following questions.
Should our company consider using the CMMI?
What advantages/disadvantages does it provide?
We only develop software – does the CMMI apply to us?
We’re using the SW-CMM now – should we transition to CMMI? How long will it take?
Where can I find additional resources and information?
What does the CMMI mean to me – the QA and/or testing specialist?
Rick Hefner, Ph.D.
Dr. Hefner has been a member of the CMMI development project since its inception, and currently serves on the Change Control Board. He has over 25 years of experience in software development, research, and management, and has authored over 50 papers and presentations. Dr. Hefner is an SEI-authorized Lead Assessor, and has performed over 40 assessments. He is the Manager of Process Initiatives at TRW, where his responsibilities include CMM and ISO-based improvement programs, training, metrics, and assessment.

2. Agenda Project History CMMI Structure
Comparisons with SW-CMM v1.1, SE-CMM, and EIA/IS 731
Assessment Methodology
Transition Timelines and Strategies
Implications for Quality Assurance Professionals
SCQAA - Rick Hefner

3. What is a CMM?
Capability Maturity Model: A reference model of mature practices in a specified discipline -- industry best practices
CMMs differ by
Discipline (software, systems, acquisition, etc.)
Structure (staged versus continuous)
How Maturity is Defined (process improvement path)
How Capability is Defined (institutionalization)
“Capability Maturity Model®” and CMM® are used by the Software Engineering Institute (SEI) to denote a particular class of maturity models
Capability Maturity Model®, CMM®, CMM Integration, and CMMI are service marks and registered trademarks of Carnegie Mellon University
SCQAA - Rick Hefner

4. Commonly Used CMMs Software CMM staged software development
System Engineering CMM continuous system engineering
System Engineering Capability Model continuous system engineering
Software Acquisition CMM staged software acquisition
System Security Engineering CMM continuous security engineering
Personal Software Process staged individual software development
FAA-CMM continuous software engineering, systems engineering, and acquisition
IPD-CMM continuous integrated product development
People CMM staged workforce
SPICE Model continuous software development
SCQAA - Rick Hefner

5. What is the Problem?
Different structures, formats, terms, ways of measuring maturity
Causes confusion, especially when using more than one CMM
Hard to integrate them in a combined improvement program
Hard to use multiple models in supplier selection
Systems Engr
CMM
IPD
CMM
FAA
iCMM
People
CMM
People
CMM
CMMI
Software
CMM
Software Acq
CMM
ZZZ
CMM
Security Systems
Engr
CMM
Systems
Security Engr CMM
SCQAA - Rick Hefner

6. <We could add organizational slides here for backup>
The CMMI Project
DoD sponsored collaboration between industry, Government, academia
Over 100 people involved
U.S. Army, Navy, Air Force
Federal Aviation Administration
National Security Agency
Software Engineering Institute
ADP, Inc.
AT&T Labs
BAE
Boeing
Computer Sciences Corporation
EER Systems
Ericsson Canada
Ernst and Young
General Dynamics
Harris Corporation
Honeywell
KPMG
Litton
Lockheed Martin
Motorola
Northrop Grumman
Pacific Bell
Q-Labs
Raytheon
Rockwell Collins
Sverdrup Corporation
Thomson CSF
TRW
So, based on the problem and an understanding of the fundamental similarities of CMMs and CMM-based improvement methods, the CMMI project was born!
<We could add organizational slides here for backup>
SCQAA - Rick Hefner

7. CMMI Models Source Models
Disciplines
systems engineering
software engineering
integrated process and product development (IPPD)
supplier sourcing
combinations of the above
Representations
staged
continuous
Source Models
Capability Maturity Model for Software V2, draft C (SW-CMM V2C)
EIA Interim Standard 731, System Engineering Capability Model (SECM)
Integrated Product Development Capability Maturity Model, draft V0.98 (IPD-CMM)
SCQAA - Rick Hefner

8. Maturity Levels
Each level is a layer in the foundation for continuous process improvement
Level 5
Optimizing
Focus on process improvement
Level 4
Quantitatively
Managed
Process measured and controlled
Level 3
Defined
Process characterized for the organization and is proactive
Level 2
Managed
Process characterized for projects and is often reactive
Level 1
Performed
Process unpredictable, poorly controlled and
reactive
SCQAA - Rick Hefner

9. CMMI Process Areas Level Focus Process Areas Continuous process
improvement
Causal Analysis and Resolution
Organizational Innovation and Deployment
5 Optimizing
4 Quantitatively
Managed
Quantitative
management
Quantitative Project Management
Organizational Process Performance
Organizational Process Focus
Organizational Process Definition
Organizational Training Integrated Project Management
Risk Management
Decision Analysis and Resolution
Requirements Development
Technical Solution
Product Integration
Verification
Validation
Process
standardization
3 Defined
Requirements Management
Project Planning
Project Monitoring and Control
Supplier Agreement Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
2 Managed
Basic
project
management
1 Performed
SCQAA - Rick Hefner

10. Process Area Group Relationships
Process Management
Organizational Process Focus
Organizational Process Definition
Organizational Training
Organizational Process Performance
Organizational Innovation and Deployment
empower
analyze
standardize processes
Project Management
Project Planning
Project Monitoring and Control
Supplier Agreement Management
Integrated Project Management
Risk Management
Quantitative Project Management
Support
Configuration Management
Process and Product Quality Assurance
Measurement and Analysis
Decision Analysis and Resolution
Causal Analysis and Resolution
analyze
Engineering
Requirements Development
Requirements Management
Technical Solution
Product Integration
Verification
Validation
employ
measure & assist
SCQAA - Rick Hefner

11. Common Features For Each Process Area
Commitment to Perform
Establish an Organizational Policy
Ability to Perform
Plan the Process
Provide Resources
Assign Responsibility
Train People
Establish a Defined Process
Directing Implementation
Manage Configurations
Identify and Involve Relevant Stakeholders
Monitor and Control the Process
Collect Improvement Information
Verification
Objectively Evaluate Adherence
Review Status with Higher-Level Management
SCQAA - Rick Hefner

12. CMMI Staged Model Structure
Maturity Levels
...
Process Area 1
Process Area 2
Process Area n
Generic Goals
Specific Goals
Common Features
Activities Performed
Commitment
to Perform
Ability
to Perform
Directing Implementation
Verification
Specific Practices
Generic Practices
Specific Practices
Specific Practices
Specific Practices
Specific Practices
Subpractices Amplifications
Elaborations
Subpractices Amplifications
Elaborations
Subpractices Amplifications
Elaborations
Subpractices Amplifications
Elaborations
Subpractices Amplifications
Elaborations
Subpractices Amplifications
Elaborations
SCQAA - Rick Hefner

13. Required, Expected, Informative
Maturity Levels
...
Process Area 1
Process Area 2
Process Area n
Generic Goals
Specific Goals
Required
Needed to satisfy CMMI
Expected
Typical practices to meet goals;
Used as a starting point in assessments
Informative
Ideas to consider
Common Features
Activities Performed
Commitment
to Perform
Ability
to Perform
Directing Implementation
Verification
Specific Practices
Generic Practices
Specific Practices
Specific Practices
Specific Practices
Specific Practices
Subpractices Amplifications
Elaborations
Subpractices Amplifications
Elaborations
Subpractices Amplifications
Elaborations
Subpractices Amplifications
Elaborations
Subpractices Amplifications
Elaborations
Subpractices Amplifications
Elaborations
SCQAA - Rick Hefner

14. SW-CMM v1.1 vs. CMMI Process Areas
Defect Prevention Causal Analysis and Resolution
Technology Change Mgmt Organizational Innovation & Deployment
Process Change Management
Quantitative Process Mgmt Organizational Process Performance
Software Quality Mgmt Quantitative Project Management
Organization Process Focus Organization Process Focus
Organization Process Definition Organization Process Definition
Training Program Organizational Training
Integrated Software Mgmt Integrated Project Management
Risk Management
Software Product Engr Requirements Development
Technical Solution
Product Integration
Intergroup Coordination Verification
Peer Reviews Validation
Decision Analysis and Resolution
Requirements Management Requirements Management
Software Project Planning Project Planning
Software Project Tracking & Oversight Project Monitoring and Control
Software Subcontract Mgmt Supplier Agreement Management
Software Quality Assurance Product & Process Quality Assurance
Software Configuration Mgmt Configuration Management
Measurement and Analysis
LEVEL 5
OPTIMIZING
LEVEL 4
MANAGED
LEVEL 3
DEFINED
LEVEL 2
REPEATABLE 14

15. SW-CMM v1.1 vs. CMMI Common Features15

16. SE-CMM and SECM (EIA/IS 731) vs. CMMI Common Features - Engineering
Technical
Engineering
PA06 Understand Customer Needs and Expectations
PA02 Derive and Allocate Requirements
PA03 Evolve System Architecture
PA01 Analyze Candidate Solutions
PA05 Integrate System
PA07 Verify and Validate System
PA04 Integrate Disciplines
1.1 Define Stakeholder and System Level Requirements
1.2 Define Technical Requirements
1.3 Define Solution
1.4 Assess and Select
1.5 Integrate System
1.6 Verify System
1.7 Validate System
Requirement Mgmt
Requirement Development
Technical Solution
Decision Analysis and Resolution
Product Integration
Verification
Validation
2.3
SCQAA - Rick Hefner

17. SE-CMM and SECM (EIA/IS 731) vs
SE-CMM and SECM (EIA/IS 731) vs. CMMI Common Features - Project Management
SE-CMM
SECM
CMMI
Project
Management
Project Management
PA12 Plan Technical Effort
PA11 Monitor and Control Technical Effort
PA10 Manage Risk
PA09 Manage Configurations
PA08 Ensure Quality
Project Planning
Project Mon and Con
Supplier Agree Mgt
Integrated Proj Mgt
Risk Management
2.1 Plan and Organize
2.2 Monitor and Control
2.3 Integrate Disciplines
2.4 Coordinate with Suppliers
2.5 Manage Risk
2.6 Manage Data
2.7 Manage Configurations
2.8 Ensure Quality
PA04
PA18
Support
Configuration Mgt
Proc and Prod QA
Meas and Analysis *
L2 GP *
Not in SE-CMM
SCQAA - Rick Hefner

18. SE-CMM and SECM (EIA/IS 731) vs
SE-CMM and SECM (EIA/IS 731) vs. CMMI Common Features - Process Management
SE-CMM
SECM
CMMI
Organization
Environment
Process Management
PA13 Define Organization’s SE Process
PA14 Improve Organization’s SE Process
PA17 Provide Ongoing Knowledge and Skills
PA15 Manage Product Line Evolution
PA16 Manage SE Support Environment
PA18 Coordinate With Suppliers
3.1 Define and Improve the SE Process
3.2 Manage Competency
3.3 Manage Technology
3.4 Manage SE Support Environment
Org Process Focus
Org Process Def
Organizational Tng
Quant Project Mgt
Org Process Perf
Causal An & Res
Org Innovation and Deployment
(IPPD Extension) Organizational Environment Integration
L4 GP
L4 GP
L5 GP
2.4
SCQAA - Rick Hefner

19. Summary of Changes
Organizations that are currently using SW-CMM v1.1, SE-CMM, or EIA/IS 731 should be able to smoothly transition
Will require additional infrastructure
Identification and involvement of relevant stakeholders
Configuration management of intermediate work products
Will require additional processes
Measurement and Analysis
Decision Analysis and Resolution
Requirements Development
Risk Management
Additional focus on engineering processes
Software organizations adopting CMM-based improvements for the first time should use the CMMI
SCQAA - Rick Hefner

20. Appraisal Requirements for CMMI (ARC)
Similar to the current CMM Appraisal Framework (CAF) V1.0
A guide to assessment method developers
Specifies the requirements for classes of assessment methods
Class A: Full, comprehensive assessment methods
Class B: Initial, incremental, self-assessments
Class C: Quick-look
Method developers can declare which class their method fits
Method buyers will understand the implications of the method proposed
SCQAA - Rick Hefner

21. Standard CMMI Assessment Method for Process Improvement (SCAMPI) v1.1
Similar to CBA IPI method
Led by authorized Lead Assessor
Team of 4-9 trained assessors
Tailorable to organization and model scope
Products:
SCAMPI Method Description Document
Maturity questionnaire(s), electronic work aids, templates
The method was updated to version 1.1 to improve efficiency
SCQAA - Rick Hefner

22. CMMI Transition Plan Development Phase Transition Phase
Development of CMMI products
Verification and validation of CMMI products
Transition Phase
Approval of initial CMMI products for public release
Evidence of sufficient use
Transition planning to help organizations use CMMI products
Sustainment Phase
Upkeep and continuous improvement of the product suite
Additional evidence of adoption and use
Pilots
V0.2
Aug 1999
V1.0
Aug 2000
V1.1
Dec 2001
SW-CMM, EIA 731 phased out
Aug 2003
SCQAA - Rick Hefner

23. Role of the Quality Assurance Professional
QA has a defined role in every process area, and a process area of their own
SCQAA - Rick Hefner

24. Generic Practices GP 2.9 (VE1) Objectively Evaluate Adherence
Objectively evaluate adherence of the process and the work products and services of the process to the applicable requirements, objectives, and standards, and address noncompliance.
Quality assurance must be performed in all processes, work products, and services
Engineering
Support
Project Management
Organizational process management
SCQAA - Rick Hefner

25. Process and Product Quality Assurance - 1
Purpose
Objectively review activities and work products for their adherence to applicable requirements, process descriptions, standards, and procedures, and communicate the results to staff and management.
P&PQA ensures planned processes are implemented; Verification ensures the specified requirements are satisfied.
QA can be embedded in the process, but:
Objectivity must be maintained
All QA performers must be trained
An independent reporting channel must be maintained
QA should participate in establishing the plans, processes, standards and procedures to ensure they:
Fit the project’s needs
Will be useable for performing quality assurance evaluations.
SCQAA - Rick Hefner

26. Process and Product Quality Assurance - 2
SG 1 Objectively Evaluate Processes and Work Products
SP 1.1 Objectively Evaluate Processes Objectively evaluate the designated performed processes against the applicable process descriptions, standards and procedures.
SP 1.2 Objectively Evaluate Work Products and Services Objectively evaluate the designated work products and services against the applicable process descriptions, standards, and procedures.
Projects must designate which processes and work products they intend to audit
What processes and work products drive overall quality?
Where is insight into quality the most valuable?
Assessor typically expect all processes to be audited
SCQAA - Rick Hefner

27. Process and Product Quality Assurance - 3
SG 2 Provide Objective Insight
SP 2.1 Communicate and Ensure Resolution of Noncompliance Issues Communicate quality issues and ensure resolution of noncompliance issues with the staff and managers.
SP 2.2 Establish Records Establish and maintain records of the quality assurance activities.
Requires a independent reporting channel for noncompliance issues
Makes senior management responsible for adjudicating quality versus schedule conflicts
SCQAA - Rick Hefner

28. Process and Product Quality Assurance - 4
Quality assurance activities must be fully supported by the project and organizational infrastructure
GG 2 Institutionalize a Managed Process
GP 2.1 (CO 1) Establish an Organizational Policy
GP 2.2 (AB 1) Plan the Process
GP 2.3 (AB 2) Provide Resources
GP 2.4 (AB 3) Assign Responsibility
GP 2.5 (AB 4) Train People
GP 2.6 (DI 1) Manage Configurations
GP 2.7 (DI 2) Identify and Involve Relevant Stakeholders
GP 2.8 (DI 3) Monitor and Control the Process
GP 2.9 (VE 1) Objectively Evaluate Adherence
GP 2.10 (VE 2) Review Status with Higher-Level Management
SCQAA - Rick Hefner

29. Software requirements
Verification - 1
Purpose
Assure that selected work products meet the specified requirements (e.g., peer reviews, testing)
Key Issues
Recognizing verification is an incremental process that occurs throughout the development of the product and work products
Verification of the Requirements
Verification of the evolving work products
Verification of the completed product
Identifying where verifications will occur (as part of program strategy)
User needs
System requirements
Software requirements
Software design
Software code
Test cases
Test results
Code testing
Code peer review
SCQAA - Rick Hefner

30. Verification - 2
SG 1 Prepare for Verification Preparation for verification is conducted.
SP 1.1 Select Work Products for Verification Select the work products to be verified and the verification methods that will be used for each.
SP 1.2 Establish the Verification Environment Establish and maintain the environment needed to support verification.
SP 1.3 Establish Verification Procedures and Criteria Establish and maintain verification procedures and criteria for the selected work products.
Projects identify (and document) the work products they intend to verify
Verification environment typically refers to a test bed
Could also imply the facilities and tools needed for peer reviews
“Procedures” are not typically called out in the CMMI – implies the need for more detailed planning
SCQAA - Rick Hefner

31. Verification - 3
Different verification methods: peer review, inspection, demonstration, test, analysis
Preparing for peer review includes identification of:
Staff involved, roles
Review criteria
Schedule
Peer review data are analyzed to improve review effectiveness
Size of the product
Composition of review team
Type of peer review
Preparation time per reviewer
Length of the review meeting
Number of defects found, type and origin of defect, etc.
SG 2 Perform Peer Reviews Peer reviews are performed on selected work products.
SP 2.1 Prepare for Peer Reviews Prepare for peer reviews of selected work products.
SP 2.2 Conduct Peer Reviews Conduct peer reviews on selected work products and identify issues resulting from the peer review.
SP 2.3 Analyze Peer Review Data Analyze data about preparation, conduct, and results of the peer reviews.
SG 3 Verify Selected Work Products Selected work products are verified against their specified requirements.
SP 3.1 Perform Verification Perform verification on the selected work products.
SP 3.2 Analyze Verification Results and Identify Corrective Action Analyze the results of all verification activities and identify corrective action.
SCQAA - Rick Hefner

32. Verification - 4
Verification activities must be fully supported by the project and organizational infrastructure
GG 2 Institutionalize a Managed Process
GP 2.1 (CO 1) Establish an Organizational Policy
GP 2.2 (AB 1) Plan the Process
GP 2.3 (AB 2) Provide Resources
GP 2.4 (AB 3) Assign Responsibility
GP 2.5 (AB 4) Train People
GP 2.6 (DI 1) Manage Configurations
GP 2.7 (DI 2) Identify and Involve Relevant Stakeholders
GP 2.8 (DI 3) Monitor and Control the Process
GP 2.9 (VE 1) Objectively Evaluate Adherence
GP 2.10 (VE 2) Review Status with Higher-Level Management
SCQAA - Rick Hefner

33. Implications for the Quality Assurance Professional
More companies are adopting the CMMI (and CMM)
Required by customers
Improves predictability and efficiency; reduces rework and cost
A competitive discriminator
The CMMI/CMM requires a significant QA presence
Involvement in all projects, all processes
Significant support required to satisfy PPQA process area
CMMI/CMM provides an opportunity to educate the community on the proper role of QA
SCQAA - Rick Hefner

34. Summary CMMI is a natural evolution in industry maturity
Organizations that are currently using SW-CMM v1.1, SE-CMM, or EIA/IS 731 should be able to smoothly transition
Quality assurance professionals have an important role in the CMMI
See for more information and resources
SCQAA - Rick Hefner