1. Paul J. Frenz General Dynamics Advanced Information Systems
The Nuts, Bolts and Duct Tape of Establishing a System Engineering Measurement Program
Paul J. Frenz
General Dynamics Advanced Information Systems

2. Team Assignment Establish a Systems Engineering Measurement Program
Support CMMI Level 5
Support Organizational Initiatives
Provide Quantifiable evidence of improvements achieved

3. Measures Research Investigation of industry practices
Software Productivity Consortium (SPC)
Practical Software and Systems Measurement (PSM)
INCOSE
Generic web and literature searches
Summer Intern did industry survey

4. Research Findings Measures such as: requirements per hour
defects per requirement
can be useful to an organization:
current capabilities and setting goals
don’t provide insight into the daily management of a program

5. Measures Guidelines
Measurements selected must be tied to business goals!
This lead to measurements that supported the goals of faultless execution, customer satisfaction, and quality

6. Measurement Challenge
The objective was to give the program leads the measures they needed to support these goals
The measures needed to guide the leads with knowing what to pay attention to and when to pay attention!

7. Initial Measures Established
Building off of our lessons learned within Software Engineering, we established the initial measures using these principles:
Measure collection should be uncomplicated - preferably automated
Measures are justified

8. Initial Measures Established
We use a standard time period - calendar month
Although we permit tailoring it is discouraged
Avoid frequent replans and focus on using measures to return to the planned baseline
Consistent criteria and an approval when a measurement baseline is replanned

9. Initial Measures Selected
Productivity Measure – Requirements per Hour
Inch-stone Schedule Performance
Requirements Volatility
Requirement Volatility Impact
Problem Report Status Aging
Effort Expended to Planned Effort
Peer Review Scheduled to Plan
Peer Review Execution and Results
Requirements Defect Density

10. Productivity Measure – Requirements Per Hour

11. Inch-stone Schedule Performance to Plan

12. Inch-stone Schedule Performance to Plan

13. Requirements Volatility

14. Requirement Volatility Impact Analysis

15. Problem Report (PR) Status Aging

16. Effort Expended to Effort Planned

17. Peer Review Schedule to Plan

18. Peer Review Execution and Results

19. Peer Review Execution and Results

20. Requirements Defect Density

21. Guidance for Implementing Your Own Measurement Program
Critical to Success:
Strong measurement champion
Strong management support
Start small – limit your base measures to less than 7

22. Guidance for Implementing Your Own Measurement Program
Critical to Success:
Provide clear definitions for base measures
Measure at a high enough level to identify issues without burdening
Minimize collection effort through the use of tools and automation

23. Guidance for Implementing Your Own Measurement Program
Pitfalls:
Measures that have little value to the program
Not being able to explain how to use the data
Trying to collect too many base measures
Poorly defined, ambiguous base measures
Not creating a change in culture where managing with measures is the norm

24. Summary
The development and sustainment of a measurement program requires top down support
There must be a champion to continually sell the benefits of a measurement program

25. Summary
Champion needs depth and breadth of understanding of measures and the ability to enthusiastically communicate this understanding
The measures need to be overwhelmingly geared to assisting the program in its day-to-day execution

26. Summary
Our measurement program has assisted our organization in dramatically improving the execution of our development programs by quantitatively managing key processes and focusing attention on critical tasks to guide our leads through the development process.