Paul J. Frenz General Dynamics Advanced Information Systems The Nuts, Bolts and Duct Tape of Establishing a System Engineering Measurement Program Paul J. Frenz General Dynamics Advanced Information Systems

Team Assignment Establish a Systems Engineering Measurement Program Support CMMI Level 5 Support Organizational Initiatives Provide Quantifiable evidence of improvements achieved

Measures Research Investigation of industry practices Software Productivity Consortium (SPC) Practical Software and Systems Measurement (PSM) INCOSE Generic web and literature searches Summer Intern did industry survey

Research Findings Measures such as: requirements per hour defects per requirement can be useful to an organization: current capabilities and setting goals don’t provide insight into the daily management of a program

Measures Guidelines Measurements selected must be tied to business goals! This lead to measurements that supported the goals of faultless execution, customer satisfaction, and quality

Measurement Challenge The objective was to give the program leads the measures they needed to support these goals The measures needed to guide the leads with knowing what to pay attention to and when to pay attention!

Initial Measures Established Building off of our lessons learned within Software Engineering, we established the initial measures using these principles: Measure collection should be uncomplicated - preferably automated Measures are justified

Initial Measures Established We use a standard time period - calendar month Although we permit tailoring it is discouraged Avoid frequent replans and focus on using measures to return to the planned baseline Consistent criteria and an approval when a measurement baseline is replanned

Initial Measures Selected Productivity Measure – Requirements per Hour Inch-stone Schedule Performance Requirements Volatility Requirement Volatility Impact Problem Report Status Aging Effort Expended to Planned Effort Peer Review Scheduled to Plan Peer Review Execution and Results Requirements Defect Density

Productivity Measure – Requirements Per Hour

Inch-stone Schedule Performance to Plan

Inch-stone Schedule Performance to Plan

Requirements Volatility

Requirement Volatility Impact Analysis

Problem Report (PR) Status Aging

Effort Expended to Effort Planned

Peer Review Schedule to Plan

Peer Review Execution and Results

Peer Review Execution and Results

Requirements Defect Density

Guidance for Implementing Your Own Measurement Program Critical to Success: Strong measurement champion Strong management support Start small – limit your base measures to less than 7

Guidance for Implementing Your Own Measurement Program Critical to Success: Provide clear definitions for base measures Measure at a high enough level to identify issues without burdening Minimize collection effort through the use of tools and automation

Guidance for Implementing Your Own Measurement Program Pitfalls: Measures that have little value to the program Not being able to explain how to use the data Trying to collect too many base measures Poorly defined, ambiguous base measures Not creating a change in culture where managing with measures is the norm

Summary The development and sustainment of a measurement program requires top down support There must be a champion to continually sell the benefits of a measurement program

Summary Champion needs depth and breadth of understanding of measures and the ability to enthusiastically communicate this understanding The measures need to be overwhelmingly geared to assisting the program in its day-to-day execution

Summary Our measurement program has assisted our organization in dramatically improving the execution of our development programs by quantitatively managing key processes and focusing attention on critical tasks to guide our leads through the development process.