1. Presented To: 3rd Annual CMMI Technology Conference and User Group
Impacts of the CMMI
November 19, 2003

2. Objective
Contribute to the body of knowledge on the impacts of the CMMI
Focus on ways to quantify
Process baseline
Progress
Explore traditional and non-traditional measures
How to know what state you were in when you started the PI effort?

3. Agenda Breadth of projects covered
A baseline for pre-CMMI process behavior
Training feedback
Current project monitoring and control measures data
Qualitative Impacts

4. Breadth of Projects Covered
Less Systems Engineering
Fewer technical skills
Less complex
More Systems Engineering
More technical skills required
More complex
Government Budget Software
Document Management
Projects (some were a combination of these types):
Proposal project
Document scanning and document query and retrieval interface development
Financial management software development
Network support/OS and network software upgrades/Help Desk
Requirements elicitation and analysis consulting/training
Project size from 3 staff to 35
Proposal Development
Technology Infrastructure
System Management

5. Establishing a Baseline for Process Implementation
Audits
Lessons Learned
Ineffective processes impact
Preliminary results, no statistical significance
Audit of projects whose period of performance began prior to implementation of the CMMI standard processes and were not involved in the appraisal
All major process areas addressed (exceptions were SAM and DAR)
In most cases too few data points for statistical significance.

6. Pre-CMMI Process Baseline
Projects already functioning at 33% average compliance
Projects 67% non-compliant
Projects already functioning at 33% average compliance with the standard processes before they were fully defined (source for best practices for process action teams)
On average, projects were not in compliance with 67% of the (to-be) defined processes

7. Pre-CMMI Audit Results
Ordered by least to most compliant
Requirements Management
Configuration Management
Quality Assurance
Design, Development and Test
Project Management

8. Lessons Learned Objectives Questionnaire and meeting
Process effectiveness
Impact of ineffective processes
Questionnaire and meeting
Cross walk to CMS Defined Process ETVXs
Objective: Process effectiveness (not addressed by SCAMPI appraisal)
Format: Questionnaire and meeting
Questions on:
Awareness of standard processes
What went right
What went wrong
What to do differently
Separate PM question on effectiveness impact

9. Lessons Learned Results
Shows proportion of comments that addressed each category.
X axis reflects cross-walk to CMS defined processes

10. LL Process Ineffectiveness Impact
50% of any cost overrun
25% - 50% of any schedule overrun
Estimate is that up to 50% of any cost overrun was due to ineffective process
Estimate is that 25%-50% of any schedule overrun was due to ineffective process

11. During CMMI Effort Compliance tracked every two weeks
Sample set = each project type
Projects:
Proposal project
Network support/OS and network software upgrades/Help Desk
Financial management software development
Requirements elicitation and analysis consulting/training
Document scanning and document query and retrieval interface development
Place in life cycle impacts engineering products and PMC for those phases more than any other set of work products
Work products reflect CMS standard process

12. Progress during PI Effort
Many work products assessed for each process area; not the same number in each process area.
M = Month
Some early results reflect delay in getting work products to the EPG.
Engineering and monitoring tied to life cycle phase and some projects were not far enough along their life cycle to have work products.
Support in this definition was confined to DA and a specific DA task or N/A was defined for each project.

13. Training Questionnaire
13 questions
Example of results for:
Perceived value of training to Managers
Time spent on training
Value of company-sponsored training
Questions for managers on:
Value of training
Customer support for training
Management self-ratings on process strengths and weaknesses

15. Post-CMMI Results CMMI Class A SCAMPI Appraisal results
Audits and lessons learned
Audits and lessons learned will provide more data on status over time

16. Current PMC Measures EPG Project Process Change Requests Risks
Resource allocations by phase for a month
Resource allocations over time
Overtime or ‘grief’ index
These are examples of current PMC measures

17. PCRs and Risks No distinction made between major and minor PCRs
This is one of the measures reported monthly to the ESC.

18. Project Measures

19. Selected Phase Measures Roll-Up

20. Grief Index
This one for fun and to show how creative one can be with data.

21. Results Ineffective processes contribute to cost and schedule overruns
Lack of planning affects other process areas
Process training is highly valued
Need to plan for QA, CM, Risk, Measures, etc. If not, result is lower ability to execute in those areas throughout the life cycle.

22. Results of Data Analysis
Require additional 25% of staff training by EOY
Planning sets the stage
Initiate tasks and resource tracking on support organizations
Focus training on project planning
Although other processes were deficient in the audit results, the foundation for them is laid in planning which analysis confirmed.
New hires and new projects can be expected to show similar soft spots

23. Qualitative Impacts Potential increase in revenues and profits: 50%
External perception of CMS project management capabilities increased by a factor of 3
Risk Management- insight from audits improves ability to track risk status
3rd bullet: Can tell from audits if the right staff are going to the courses, if there is use of process work products, if there is support from senior management for status reports, risk tracking, etc.

24. Qualitative Impacts Less sense of the CMMI as ‘bolted on’
True institutionalization of CMMI compliant processes as ‘the way we do business’
Increase in use of best practices across the organization

25. Questions?

26. Contact Information
Office:
Fax:

27. List of Acronyms AI – Action Item CM – Configuration Management
CMMI – Capability Maturity Model Integration
DAR – Decision Analysis and Resolution
DD&T – Design, Development and Test
EPG – Engineering Process Group
PCR – Process Change Request

28. List of Acronyms PMC – Project Monitoring and Control
PP – Project Planning
QA – Quality Assurance
ROI – Return on Investment
RM – Requirements Management
TS – Technical Solution