1. Value of Systems Engineering
Summary report
SECOE and related projects
Eric Honour
INCOSE Director, Sponsored Research
Value of Systems Engineering; Summary Report 1/04

2. Agenda Background Gathered results on Value of SE
Heuristic Claims of SE Value
Gathered results on Value of SE
NASA Tracking 1980s
“Boundary Management” study
“Large Engineering Projects” MIT study
“Impact of SE at NASA” (SECOE 02-02)
“Impact of SE on Quality & Schedule” Boeing
“SE Effectiveness” IBM study
“Value of SE” research (SECOE 01-03)
Value of Systems Engineering; Summary Report 1/04

3. “System Thinking” Design
Heuristic Claim of SE
Better systems engineering leads to
Better system quality/value
Lower cost
Shorter schedule
SYSTEM
DESIGN
DETAIL
PRODUCTION
INTEGRATION
TEST
Traditional Design
Time
Risk
Well-known claims, but never supported to date with any real numbers. Very similar to the concepts of the MIT Lean Aircraft Initiative.
Traditional design allows risk at “acceptable” levels; the risk is reduced during integration and testing; risk goes to zero at end of project.
“System thinking” design spends more time at the systems engineering level, thereby reducing risk earlier and at less cost/time.
No proof exists for these claims.
Saved
Time/
Cost
“System Thinking” Design
Time
Risk
Value of Systems Engineering; Summary Report 1/04

4. NASA Tracking 1980s Source Werner Gruhl NASA Comptroller’s Office
GRO76
OMV
TDRSS
GALL
IRAS
HST
TETH
GOES I-M
MARS
EDO
CEN
LAND76
ACT
MAG
ERB77
COBE
CHA.REC.
STS
Chart is transcribed from a manual chart created by Werner Gruhl in 1992 prior to retirement. Individual data points are labeled with program names such as “STS” (Shuttle), Pioneer/Venture, Venus probes, etc. Each point is a major NASA project. “Definition” percent is based on phases A & B of the five-phase NASA project model. This chart shows how important it is to define the project before starting the detail work. Trend line shows that about 15% project definition appears to be optimum.
Note distinction from SE. This is “project definition,” not SE. Project definition includes some tasks that are not SE, such as model prototyping, program management, etc. SE includes tasks in phases C, D, & E that are not included in this chart.
LAND78
GRO82
SEASAT
ERB88
UARS
VOY
HEAO
EUVE/EP DE
ULYS
SMM
PIONVEN
IUE
ISEE
Source Werner Gruhl NASA Comptroller’s Office
Value of Systems Engineering; Summary Report 1/04

5. “Boundary Management” Study
Study of 45 high-tech new product development teams
...Ancona and Caldwell, Research Technology Management, 1990
Significant portion of time is spent at team boundaries
Individual Time Spent
Outside Team 14%*
* Typically limited to few individuals
Within Team 38%
Alone 48%
Study performed in late 1980s. The 45 teams were small teams, on the order of 5-15 people, developing specific commercial technical products. Purpose of the study was to determine how people spent their time, and how much of that time was spent working at the boundaries of the project.
Value of Systems Engineering; Summary Report 1/04

6. Boundary Management in Technical Teams
Boundary management occurs in four different roles.
Level of effort in each role changes with time.
Ambassador
Buffering, building support, reporting, strategy
Task Coordinator
Lateral group coordination, info transfer, planning, negotiating
Scout
Obtain possibilities from outside - interface with marketing
Guard
Withhold information, prevent disclosure
CREATION
DEVELOPMENT
DIFFUSION
The types of effort included in “boundary management” cover four different roles. All of these roles are also coincident with many systems engineering management tasks.
Value of Systems Engineering; Summary Report 1/04

7. “Boundary Management” Study
Significant Findings:
High-performing teams did more external interaction than low-performing teams
System technical leadership
creates greater success
The findings are shown in black. The words in red are an interpretation of the findings by SECOE, made in light of SE.
No surprise in the first finding, although it was generally a surprise to the rank-and-file team members when it was presented back to them. Typical response was to decry that “politics” seems to be more important than technical merit. The project leaders, however, who did most of the boundary management, were not at all surprised.
The second finding is rather alarming to today’s process-oriented world, for it seems to say that the processes were not as important as the right leadership. This echoes the primary criticism made of CMMs, that their focus is on the processes rather than on the results of the processes (i.e. the quality of the products themselves). Note however that this study looked only at small teams, where process can be assumed to be less important than on larger teams.
Internal team dynamics (goals, processes, individual satisfaction) did not correlate with performance
Process definition is important but not sufficient.
Value of Systems Engineering; Summary Report 1/04

8. “Large Engineering Projects”
Study of 60 LEPs (power generation, transportation, oil production, technology)
The Strategic Management of Large Engineering Projects, MIT Press 2000
Evaluation by interviews and by objective and subjective quality measures.
Cost Targets
Schedule Targets
Objective Targets
82%
Percent of Projects Meeting:
Study was primarily in the engineering management realm. Note that most project met cost goals, a lesser “most” met schedule goals. Note how few of the projects actually met their objectives. This shows the extreme influence of program management in large projects, the ultimate use of CAIV, an unbalanced set of priorities in which cost and schedule are far more important than achieving objectives.
72%
45%
18%
37%
Failed!
Value of Systems Engineering; Summary Report 1/04

9. “Large Engineering Projects”
Significant Findings:
Most important determinant was a coherent, well-developed organizational/team structure
A structure of leadership creates greater success
Technical difficulties, social disturbance, size were not statistically linked to performance
All projects had turbulent events
Technical excellence could not save a socially unacceptable project
Process definition is important but not sufficient.
The findings are shown in black. The words in red are an interpretation of the findings by SECOE, made in light of SE.
Findings were mostly on the management realm rather than on technical processes. The “organization” referred to in the first finding is the macro-organization for the LEP, not the project technical organization. For LEPs, politics was *the* most important factor – get the right buy-in from the right people to make the LEP successful.
Success of the projects was independent of the technical difficulties. Success was far more dependent on getting the politics right.
Contrast the last finding with the last finding of the “Boundary Management” study. For both very small projects (5-15 people) and very large projects (LEPs), it appears that technical process definition is less important. Speculate: Perhaps it is only in the middle that it matters? Perhaps the individual phase work on an LEP *is* subject to process definition? Unknown, but there is certainly a large current momentum toward process solutions to quality/product problems that is so far contradictory to these findings.
Value of Systems Engineering; Summary Report 1/04

10. Impact of SE at NASA (SECOE 02-02)
Survey research within NASA
Form with 38 questions, answers on graded scale
Typical questions:
On your most recent project, what percent of your total project cost was spent on Systems Engineering tasks?
On your most recent project, did systems engineering shorten or lengthen the planned schedule?
Aggressive survey pursuit with management push
NASA: 250 sent, 54% valid response
INCOSE: 650 sent, 38% valid response
Engineering of Complex Systems – The Impact of Systems Engineering at NASA,
A.K.P.Kludze, Jr. doctoral dissertation George Washington Univ. 2003
Value of Systems Engineering; Summary Report 1/04

11. Response Demographics
Participating Organization
NASA 136
INCOSE 243
Total 379
Education Level
Bachelor 34%
Master 55%
Doctor 11%
Work Experience % %
21+ 22%
Job Titles
SE 56%
PM 17%
Other 27%
...significant differences by organization
The project had a good representation of demographics. Responses were tracked separately for NASA and INCOSE sources, seeking differences between the two populations. There were some significant differences in the demographics for work experience, age, education level, and job titles between NASA and INCOSE sources. These differences may affect the perceptive results as reported.
Age Groups % %
60+ 10%
Value of Systems Engineering; Summary Report 1/04

12. Key Survey Results - Cost
Percent Spent on SE
0-5%
6-10%
11-15%
16% +
Respondents marked bracket to show percent of total cost spent on SE on last project.
Mode at 6-10% of project
Few projects spent 11-15%
Unexplained bimodal response >16% (perhaps interpretation of “project”)
Cost Benefit of SE
Very Poor
Poor
Fair
Good
Excellent
Respondents believe strongly in cost benefit of SE
In secondary question, few respondents could quantify
All questions were multiple choice, most with suggested brackets. Respondents believed that their last projects typically spent 6-10% of total project on SE, although there is an unexplained second mode at “16% or more.” (It is possible that this second mode resulted from an interpretation of “last project” to include many system definition projects. It is unlikely that a large number of end-to-end development projects spent over 16% on SE.)
Respondents believed strongly that SE provided a good or excellent cost benefit to their projects, but few could quantify that benefit sufficiently to respond to a secondary question.
Value of Systems Engineering; Summary Report 1/04

13. Key Survey Results - Schedule
At What Stage is SE Most Effective?
Vast majority of respondents believe that SE is most effective very early in a project.
Very Early
Midway
End
No Matter
No Need
Impact of SE on Schedule
Shorten
Lengthen
Don’t Know
INCOSE respondents believe SE shortened schedule on most recent project
NASA respondents uncertain
Secondary questions uncertain in the quantification.
...other results available in dissertation
Respondents believed overwhelmingly that systems engineering is most effective when applied very early in a project. Other effects on schedule were uncertain.
The survey explored many other facets of systems engineering, including the documentation produced, overall perceptive impact, impact on meeting objectives, satisfaction, CASE tools, risk impact, and technical impact. The survey also includes a summary of several NASA case studies.
Value of Systems Engineering; Summary Report 1/04

14. Impact of Systems Engineering on Quality and Schedule
Empirical evidence obtained from three parallel (same time) projects
Each developed a complex, robotic Universal Holding Fixture (UHF)
Each used a different level of SE
Results are compared
Trait
UHF1
UHF2
UHF3
Size
10’ x 40’
8’ x 50’
6’ x 14’
Accuracy
±0.005”
±0.003”
Contact Sensors
None 57
108
Vacuum Sensors 1 70
Real-time checks No
Yes
Probe contours
NC interface
Impact of Systems Engineering on Quality and Schedule – Empirical Evidence,
W. Forrest Frantz, Boeing Corp. 1995
Value of Systems Engineering; Summary Report 1/04

15. Project Differences Sys mgmt experience Low Low-Medium
Project Trait
UHF1
UHF2
UHF3
Sys mgmt experience
Low
Low-Medium
Subcontract approach
Design Reviews
Full-time SE on site
Access to SE support
High, but not used
High, used
Requirements approach
Token req’s
Complete, detailed, integrated req’s
Design approach
HW/SW specs, multi-org approach
Functional specs fully address HW/SW processes and interfaces
Functional adherence
Design docs took precedence; specs updated per design
Specs followed, CCB control
Design reviews
Weekly team reviews
Formal internal; little external
Formal internal and external
Integration approach
Patterned after design
Drive by functional specs; defined early in life cycle
Acceptance testing
Defined in high-level plan
Formal tests based on Req’s and Functional specs
Value of Systems Engineering; Summary Report 1/04

16. Impacts Use of better SE reduced
Requirements to RFP (weeks)
Design to Production (weeks)
Overall Development Time (weeks)
Use of better SE reduced
Overall cycle time
Time to create req’s
Time to design/produce
Time to test
...even in the face of more complex, higher quality systems!
Value of Systems Engineering; Summary Report 1/04

17. Systems Engineering Effectiveness
Study of 8 software product development projects during upgrade of SE processes
Determining Systems Engineering Effectiveness, Bruce Barker, IBM Commercial Products, Conference on Systems Integration, Stevens Institute 2003
Evaluation by cost and schedule against a standard estimating method.
Identify
affected
components
Evaluate
Impact,
Complexity
Convert to
“points”
Estimate
Cost,
Schedule
Historical Database,
Cost per “Point”
New
Product
Concept
Line
Architecture
Costing method applies only to project management, business management, systems engineering, system integration, and delivery into production. Application development costs are not included.
IBM Commercial Products division reported this study in early Division implemented new SE processes in a largely software-oriented group.
Prior costing method for non-SW tasks is based on a “point” system, where points are determined by the attributes of the product to be created/modified. This division maintains a proprietary historical database on cost per “point.”
Value of Systems Engineering; Summary Report 1/04
© Copyright IBM Corp 2003
Used With Permission

18. Project Data $K/Point Averages 2000 2001 2002 $1,454/pt $1,142/pt
12,934
1,223
10,209
8,707
4,678
5,743
14,417
929
18,191
2,400
11,596
10,266
5,099
5,626
10,026
1,600
9.2
10.7
14.4
10.2
16.0
1,406
1,962
1,136
1,179
1,090
980
695
1,739
Year
Project
“Points”
Cost
($K)
SE Costs
(%)
$K/
Point
Project data is on eight projects that spanned the introduction of the new SE procedures. Data shows a significant improvement in cost per “point” with the inclusion of SE procedures.
$K/Point Averages
2000
2001
2002
$1,454/pt
$1,142/pt
$818/pt
Without SE
With SE
$1,350/pt
$944/pt
Value of Systems Engineering; Summary Report 1/04
© Copyright IBM Corp 2003
Used With Permission

19. SE organization created SE Formal Training Started
Timeline of Projects
Project w/o SE
Cost per “Point”
Project With SE
2500
SE organization created
As the Systems Engineering process has been enabled and integrated through the organization, productivity has increased
Yearly Avg
SE Process documented
2000
SE Formal Training Started
1500
1000
Chart shows graphically the improvement in cost per “point” over time as the SE procedures were implemented.
500
1999
2000
2001
2002
Value of Systems Engineering; Summary Report 1/04
© Copyright IBM Corp 2003
Used With Permission

20. Systems Engineering Effectiveness
Significant Findings:
Impact and complexity provide an effective method to perform parametric costing.
Early parametric costing works.
Preliminary data indicates that the use of Systems Engineering will improve project productivity when effectively combined with the Project Management and Test Processes.
Systems engineering improves productivity.
$K/Point Averages
Without SE
With SE
$1,350/pt
$944/pt
2000
2001
2002
$1,454/pt
$1,142/pt
$818/pt
Study evaluated the “point” costing method and found it to be reasonably reliable within risk bounds.
Of more interest to SE, the costing method showed a significant reduction in cost per “point” when the SE processes were implemented. Note that this is the cost of the non-SW tasks, indicating that a good SE process significantly reduces the costs incurred in program management, business management, systems engineering, system integration, and delivery into production. On further study the authors expect that SW costs will also be found to have been reduced, adding to the benefit.
Study only includes eight projects – eight data points – and so is possibly subject to variability issues. Nonetheless, the improvement numbers are significant enough to outweigh any expected variability.
Value of Systems Engineering; Summary Report 1/04
© Copyright IBM Corp 2003
Used With Permission

21. “Value of SE” (SECOE 01-03)
Multi-year effort to obtain statistical data
Correlate amount/quality of SE with project quality/success
Parameterized by
Technical “Size”
Technical Complexity
Risk Level
Development Quality
(function of
Technical Value,
Cost,
Schedule,
Risk)
Effort started in 2001 with small seed money funded by INCOSE. Gathering project data to quantify the value of SE in terms of quality, cost, schedule, and risk parameterized by the amount and types of SE efforts.
Original goal was to obtain sufficient data to parameterize the systems engineering efforts by phase and by type of task. The reality has been that gathering even the basic top-level data has been so difficult as to preclude any effective striation by phase or task.
6-10% ?
SE Effort
Value of Systems Engineering; Summary Report 1/04

22. Respondent Data 43 respondents 1 project not completed
Values $1.1M - $5.6B
SE Cost 0.3% - 26%
Cost, schedule, quality correlate better with “Systems Engineering Effort”:
SEE = SE Qual * (SE Cost %)
Respondent data so far has all been subjective and anonymous, submitted in response to widely-distributed forms. Data includes basic project data on project size, planned cost & schedule, actual cost & schedule, percent SE effort, quality of SE effort. Median expenditure is 5% SE. Data correlation is better, however, when SE effort is qualified by the perceived subjective quality. When so qualified, median expenditure is 2%.
Value of Systems Engineering; Summary Report 1/04

23. Cost Overrun vs. SE Effort
90% Assurance (1.6)
Average Cost Overrun
This is the companion data to the NASA study, but based on SE Effort rather than on Project Definition. The NASA chart shows that 15% of the project should be spent in definition; this chart shows that about 15% of the project should be spent on SE tasks. Despite the similarity of the numbers, these are two completely different findings.
Note the opportunity for Skinnerian (a la B.F. Skinner) behavior at low levels of SE Effort. Some projects with little SE do finish on or near cost goals, leading program managers to conclude that it *is* possible and that <2% *is* an appropriate level for SE Effort. Ask any golfer. Intermittent reinforcement is powerful training.
Source: SECOE INCOSE 2003
Value of Systems Engineering; Summary Report 1/04

24. Schedule Overrun vs. SE Effort
90% Assurance (1.6)
Average Schedule Overrun
Companion data chart for schedule overruns.
Source: SECOE INCOSE 2003
Value of Systems Engineering; Summary Report 1/04

25. Test Hypothesis: Quality
SE Effort
Quality
Hypothesis
The upper left chart shows that the data supports the original hypothesis, that cost and schedule performance are improved (up to a point) by the use of more SE effort. (The vertical axis on this chart is the inverse average of cost and schedule overruns.)
The lower right chart is an independent, subjective check on the hypothesis, based on the subjective inputs of respondents. Their evaluation of “comparative success” on a scale of 0-10 also shows higher project success with greater SE effort.
Source: SECOE INCOSE 2003
Value of Systems Engineering; Summary Report 1/04

26. Conclusions “Value of SE”
SE effort improves development quality
Cost, schedule, subjective
Hypothesis is supported by the data
Optimum SE effort is 10-15% or more
Cost, schedule overruns are minimized
However, note wide dispersion of data
Also note few data points at this level; most projects spent far less
Quality of the SE effort matters
Lower quality SE reduces effectiveness
Findings are reasonably self-evident, but.
See the 2002 INCOSE symposium paper for a discussion of limitations of the data.
Value of Systems Engineering; Summary Report 1/04

27. Company Participation
Project benchmarking service funded by participants
Aggregated data shared among participants
Raw data protected by data blinding
Every two years
Public
Quarterly
Participating
Companies
(all data)
1 month
Reported
results
Participating
Companies
This slide defines a mutually beneficial company participation in SECOE Companies choose to participate in a project benchmarking service. For their participation, they receive comparative benchmarking reports on the selected projects as well as quarterly reports of the aggregated data from all participants. All raw data is heavily protected by a blinding system that prevents identification of the gathered data with any specific project or company. Aggregated data will not be provided until sufficient data is obtained to mask the effects of individual projects.
SECOE
Blinded
Raw Data
Statistical data
Value of SE
Project
Benchmarking
data
Value of Systems Engineering; Summary Report 1/04

28. Company Participation
Data gathering
Select 4-6 programs
One day session per participating company every 4 months
1½-hour sessions with PM+SE of each program
Data gathered by two SECOE researchers
Forms & notes do not identify programs
Reports
Benchmark report within 30 days of each quarterly session, compares to all prior data
4-month reports to all participants with aggregated results from all data, all sources
Participation price $??K for each year
Participation involves quarterly data gathering by interviews with the Program Manager and Systems Engineer of 4-6 projects. Interviews take one day for each participant company, with comparative results provided within 30 days. Participation price is not yet established, but would be on the order of $15-25K per year.
Value of Systems Engineering; Summary Report 1/04

29. Questions? Eric Honour INCOSE Director for Sponsored Research
Pensacola, FL, USA
+1 (850)
Value of Systems Engineering; Summary Report 1/04