1. An Effective and Efficient Approach to ARC C Appraisals
SCAMPI ‘C’: A Foundation for Successful SCAMPI ‘A’
Northrop Grumman Mission Systems has conducted a quarter of the total SCAMPI ‘A’ appraisals across the globe in year This paper discusses how SCAMPI ‘C’ appraisals have helped NGMS divisions for conducting successful SCAMPI ‘A’ appraisals and there by achieving their organizational goals for process improvement.
The presentation will focus on
How SCAMPI C appraisals helped the organization to conduct SCAMPI ‘A’ in 30 to 40% less time (60 hours against the industry average of 90 hours) than the industry average and at lower cost.
How this approach has made projects focus on small but steady improvements while still meeting their customer schedule & commitments
How the SCAMPI ‘C’ appraisals have increased the confidence of the organization to go for SCAMPI ‘A’ appraisal
How properly scheduled SCAMPI ‘C’ provided low cost and efficient assessment exercise for process improvement
How the use of tools and organization wide guidance helped in institutionalizing the process improvements (using the bottom up approach – inputs from projects to organization)
How the CMMI implementation was quantitatively measured and monitored across the entire organization
How SCAMPI ‘C’ was used in providing the organization wide strategic direction on process improvement
How the organization wide resource pool was established (with right knowledge and skills) to support various appraisals
Experiences of NGMS in conducting SCAMPI ‘B’ and ‘C’ appraisals
How Six Sigma was used to improve the NGMS appraisal process
The other lessons learned
The intended audiences for this presentation are
·Organizations implementing CMMI and are aiming at a maturity level
·Organizations planning to implement CMMI
·Organizations with different types of projects, multiple locations looking for efficient ways of enterprise wide process assessments
CMMI Technology Conference & User Group
17-20 November 2003
Madhu Parella, Satyam
Janice Tauser, Northrop Grumman
Rick Hefner, Northrop Grumman

2. Agenda Appraisal Classes Why ARC C? Our Approach ARC C Tools
Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved.
Agenda
Appraisal Classes
Why ARC C?
Our Approach
ARC C Tools
Measured Benefits
In this presentation SCAMPI A is referred as SCAMPI. SCAMPI B and SCAMPI C are referred as ARC B and ARC C or Class B and Class C appraisal.

3. Appraisal Method Classes
Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved.
Appraisal Method Classes
Appraisal Requirements for CMMI (ARC) introduction
Various requirements for ARC Classes
This slide is to bring the audience to a common platform (considering that there may be some who are not familiar with appraisal classes).
This presentation tries providing various measures taken to increase the confidence in ARC C appraisal results while maintaining the low cost.
Source: CMU/SEI-2001-TR-034

4. Copyright 2003 Northrop Grumman Space and Mission Systems Corp
Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved.
Why ARC C?
Baseline CMMI implementation status for less resources and time than SCAMPI
Establish a common internal benchmarking method
Provide a Go/No Go decision for SCAMPI
Formalized method minimizes the subjectivity associated with different appraisers, different projects
These two slides provide information on why such a formal appraisal technique is required when some of the objectives like independent verification can be met by internal audit function.
ARC C is a formal verification method with the flexibility of less formal verification methods like internal audit etc.
Baseline CMMI implementation status: When the project first fills responses to SAT questions, the responses are based on project understanding which may or may not match organization/CMMI expectations. When an ARC C is performed all these issues are resolved and that becomes the true process improvement status on the project against which the monthly progress is measured.
Uniform implementation/verification approach: For Large organizations like NGMS with different kinds of projects a verification method like ARC C provides common way of measurement across the organization.
ARC C Provides formal, out of routine verification (audits) feeling to the projects.

5. Copyright 2003 Northrop Grumman Space and Mission Systems Corp
Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved.
Six Sigma
Six Sigma methodology was used to analyze the entire CMMI appraisal process – preparation, onsite
Based on 20+ SCAMPIs conducted to date
Goal was to reduce the overall time spent
Projects preparing for the appraisal
Appraisers conducting the appraisal
Also used to increase the stability (predictability) and effectiveness of the appraisal process
Appraisal is viewed as a process that must be measured and continuously improved
Six sigma methodology is used to identify improvements in the appraisal process. Various six sigma projects like improvement of the overall appraisal process, improvement of evidence collection and presentation process have direct impact on conducting of ARC C appraisals on a project.

6. Our Approach Documented process – an ARC C requirement!
Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved.
Our Approach
Documented process – an ARC C requirement!
Used to verify gaps at the start of the improvement, and to verify readiness for SCAMPI A
Performed by 1 or 2 qualified appraisers, collaborating with key project personnel
Uses the same evidence notebooks used for SCAMPI A, although may include helping to identify the evidence
Uses our SCAMPI A consolidation and rating tools
This slide explains NGMS approach in conducting an ARC C and how the confidence level of the ARC C appraisal findings is improved.

7. Using the Results for Appraisal Readiness
Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved.
Using the Results for Appraisal Readiness
Typically conducted for appraisal readiness 2 months prior to the SCAMPI A
Have found that if more than 40 practices have insufficient evidence, should delay the appraisal
Should conduct another ARC C appraisal prior to new appraisal date
The number of practices that have insufficient evidence greatly impacts SCAMPI A duration
Schedules are built expected 20 or fewer evidence requests
This slide and next slide explains how the results from an ARC C are used to decide on when a project is likely to get ready for a SCAMPI A

8. Infrastructure to Support ARC C Appraisals
Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved.
Infrastructure to Support ARC C Appraisals
Qualified People – SCAMPI Lead Appraisers
SCAMPI ATM trained
Integrated set of tools for evidence gathering, reviews, and reporting
Common documented procedures, standards and templates
How NGMS institutionalizes and supports ARC C?
Primarily through having a pool of qualified people who can support ARC C,
Providing various tools that are required for implementation as well as verification
Developing processes to support conducting of an ARC C
Providing standards and templates for various common activities and use.
The set of integrated tools support both projects and organizations in various phases of the process improvement life cycle. These integrated set of tools compliment each other in implementation and verification process (for example, QA can generate the process verification questionnaire based on the tailored process). More details are provided in the next slide.
A documented procedure for ARC C helped in implementing the process uniformly across the organization.
NGMS

9. Integrated Set of Tools
Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved.
Integrated Set of Tools
Self Assessment Tool (SAT)
Projects record gaps against CMMI practices
Policy Compliance and Tailoring Report Tool (PCTR)
New projects document intended compliance
Audit Checklists
QA verifies compliance with policy & process
Appraisal Findings Tool
Appraisers record compliance with CMMI practices
This chart shows various tools associated with process improvement life cycle phases.
These tools ensures consistent implementation of various practices across the organization.

10. Benefits of Using Tools
Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved.
Benefits of Using Tools
Project road map (prepared and owned by project)
Roll up to multiple levels in the organization (aligned with organizational goals)
Common measurement system across the projects
Quantitative – no intuition or judgments
Provides indirect evidence
Clearly defined responsibility for each activity
The set of integrated tools support the overall process improvement at NGMS.
These tools help projects plan and monitor their own process improvement. Also the tools are helpful in knowing the status of the organization process improvement status and rolling up of projects data to organization level.
The status is measured uniformly across the organization with clearly defined responsibility for each activity. Since these tools are updated and the data is owned by projects, they also provide indirect evidence for an appraisal.
Some of the tools facilitate quantitative measurement of process improvement activities.

11. Quantitative Measurement for CMMI Compliance
Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved.
Quantitative Measurement for CMMI Compliance
Self Assessment Tool (SAT)
This shows an example of the how the various process improvement activities are measured and monitored using the quantitative data, This data is used both at project and organization level.
Identify areas requiring
special attention
Planned Vs Actual Progress

12. Implementation Deployment Strategy
Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved.
Measurement Data Flow
Projects
Implementation
Measurement Data roll up
Deployment
OU Level
The process improvement status is baselined and progress data is collected at project level (shows the planned and actual status every month).
Project measurements are rolled up to organizational unit level and provides input to the organizational S/SEPG group for deployment of process improvements.
Organization level measurements are collected and presented to Senior Management indicating areas of performance and areas of concern. At the Organization level these measurements provide inputs for process improvement strategy.
Measurement Data roll up
Strategy
Org. Level

13. Copyright 2003 Northrop Grumman Space and Mission Systems Corp
Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved.
Organization Support
Defined & documented method for ARC C appraisal at organizational level
Management tracked quantitative measures of CMMI compliance across all levels of the organization
Made ARC C a prerequisite for SCAMPI A
Provided an CMMI interpretations document to ensure common understanding
Provided expert advise, where required
Shared lessons learned through SSEPG meetings, etc.
This slide gives more details on the NGMS support to ARC C appraisals.
The organization support helps institutionalizing the process quicker.
Experts like SCAMPI Lead appraisers and Six sigma black belts and subject matter experts provide advise where required.
These lessons learned are used as inputs to update the CMMI guidance documents in the organization.

14. Feedback Input to Organizational Strategy
Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved.
Feedback Input to Organizational Strategy
Provide additional guidance
Conduct ARC C appraisals
Collect appraisal results at org. level
Projects use the additional guidance
Appraisals typically provide information about the process improvement bottlenecks. By ensuring this feedback is used as one of the inputs to the process improvement strategy, timely guidance can be provided to the projects.
If an organization depends on the SCAMPI feedback, it may loose some important updates to organization strategy to be communicated on time. The number of SCAMPI A appraisals are limited in an organization compared to to ARC C. Also the ARC C is conducted in the earlier phases of the process improvement life cycle and the findings help an organization fix the problems before going to SCAMPI A appraisal.
Identify common problem areas
What happens if organizations depend only on SCAMPI to provide this feedback to strategy?

15. How ARC C Supports our SCAMPIs
Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved.
How ARC C Supports our SCAMPIs
Project Road Map
First ARC C
Baseline
Second ARC C
Monthly Progress
Pre Assessment
This flow chart shows the major steps involved in the process improvement life cycle of a project going for a SCAMPI A appraisal
Initially the project fills the SAT tool which acts as a project road map for achieving CMMI. This is based on the project’s understanding of process improvement activities. The first ARC C is conducted to make sure that project’s understanding is inline with organization/CMMI expectations. The output of this ARC C may be a typical gap analysis report.
This forms a baseline for measurement of process improvement progress.
When the project reaches a certain milestone an intermediate ARC C is conducted. Based on the results the projects are advised further on their approach to achieve CMMI maturity level goal. In some cases the same ARC C may be taken as pre assessment for a final SCAMPI appraisal.
If required an additional ARC C is scheduled for further verification.
SCAMPI Readiness
SCAMPI A
Additional ARC C

16. ARC C: Effect on SCAMPI A
Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved.
ARC C: Effect on SCAMPI A
Activities with minor or no Impact
Activities significantly Impacted
Team Training
Opening Briefing
Evidence Review
This chart shows a typical flowchart for a SCAMPI A appraisal indicating the activities that are impacted by an ARC C appraisal.
The time taken for activities like evidence review and number of evidence requests will be substantially less when one or more ARC C appraisals are conducted prior to a SCAMPI A. These activities take major time share in a SCAMPI appraisal onsite period.
The evidence review is much faster in verification mode and a prior ARC C helps an appraisal to continue in verification mode. In discovery mode the number of evidence requests tend to be more than verification.
Mini team consolidation and full team consolidation take less time due to development of common interpretations of various CMMI practices.
Interviews
Evidence Requests
Mini Team Consolidation
Draft Briefing
Full Team Consolidation
Final Briefing

17. ARC C: Effect on SCAMPI A (continued)
Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved.
ARC C: Effect on SCAMPI A (continued)
Keeps SCAMPI A in ‘verification’ mode
Where was maximum time spent in a CBA IPI assessment?
Control gate for SCAMPI A – Saves resources (and agony?)
Familiarized project/organization with appraisal process
Majority of model interpretation issues dealt already
Obvious weaknesses are minimized
ARC C keeps any of the obvious weaknesses dealt before the SCAMPI appraisal and brings out many interpretation issues.
The appraisal team can communicate their requirements much easily when a project/organization becomes familiar with appraisal process and requirements.

18. SCAMPI Effort With and Without ARC C
Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved.
SCAMPI Effort With and Without ARC C
For a Level 3 organization with 3 medium to large projects, and a 10 person SCAMPI A team
SCAMPI with ARC C
4 member ARC C team typically spent 150 hours
SCAMPI A appraisal team spent 450 hours
SCAMPI without ARC C
SCAMPI A appraisal team spent 600 hours
Based on small number of appraisals, our experience shows it does not consume significantly more resources or more time to conduct an ARC C Appraisal prior to a SCAMPI A appraisal.
The resources spent in both the cases shown above are almost equal. In the second case (with ARC C) the appraisal process remains in verification mode and has more probability to achieve the maturity level.
In the first case, typically the teams spend more onsite time and this calculation takes an average of 10 hours per day for a 6 to 7 day appraisal
In the second case an ARC C with a four member qualified team takes approximately 4 days for an ARC C and this might go up based on the organization history in process improvement. In this case the on site appraisal time is limited to 5 days and the experience shows that they tend to spend less onsite time. This onsite appraisal effort is dependent on various other factors (explained in the six sigma slide) along with a prior ARC C.
Total Effort 600 Hrs

19. Copyright 2003 Northrop Grumman Space and Mission Systems Corp
Copyright 2003 Northrop Grumman Space and Mission Systems Corp. All Rights Reserved.
Conclusions
ARC C:
Provides an efficient and affordable verification approach
Helps identify areas of improvement early
Supports entire CMMI implementation life cycle
For NGMS ARC C provided an efficient verification approach and its suitability and flexibility to use at various phases of life cycle has provided many benefits