1. Advancing the knowledge of systems engineering
Toward an Ontology for Measuring Systems Engineering Return on Investment
Advancing the knowledge of systems engineering
Eric Honour
+1 (850)
Dr. Ricardo Valerdi
+1 (617)
Toward Ontology for SE-ROI
Version

2. Topics SE-ROI Project “Ontology” concept COSYSMO work toward ontology
Categorization from current standards
Future directions
Toward Ontology for SE-ROI

3. Systems Engineering Return on Investment
Summary of the SE-ROI Project
Toward Ontology for SE-ROI

4. “System Thinking” Design
Heuristic Claim of SE
Better systems engineering leads to
Better system quality/value
Lower cost
Shorter schedule
SYSTEM
DESIGN
DETAIL
PRODUCTION
INTEGRATION
TEST
Traditional Design
Time
Risk
Saved
Time/
Cost
“System Thinking” Design
Time
Risk
Toward Ontology for SE-ROI

5. Value of SE – 2004 Results Problems/challenges:
Value = 1.0 if program met cost/schedule goals
Each dot is one program, with sizes between $1M and $6.5B
The upper left chart shows that the data supports the original hypothesis, that cost and schedule performance are improved (up to a point) by the use of more SE effort. (The vertical axis on this chart is the inverse average of cost and schedule overruns.)
The lower right chart is an independent, subjective check on the hypothesis, based on the subjective inputs of respondents. Their evaluation of “comparative success” on a scale of 0-10 also shows higher project success with greater SE effort.
Problems/challenges:
Quantitative data on SE not available in program databases
All data points were subjective
Detailed structure not available
Greater SE Effort led to better cost/schedule compliance and better predictability
Source: SECOE INCOSE 2003
Toward Ontology for SE-ROI
Measurable Systems Engineering

6. SE-ROI Project Interviews Just-completed programs Desired Results
Key PM/SE/Admin
Translate program data into project structure
Desired Results
Statistical correlation of SE methods with program success.
Leading SE indicators that can be used during a program.
Identification of good SE practices under different conditions.
Program characterization
Program success data
SE data (hours, quality, methods)
Statistical correlation
Toward Ontology for SE-ROI

7. What is this word and how does it relate to systems engineering?
“Ontology” Concept
What is this word and how does it relate to systems engineering?
Toward Ontology for SE-ROI

8. Current State of SE Definition
Fragmented by domain opinions
Military – DOD/MOD
Space - NASA/ESA
Commercial products
Aircraft
Automobiles
Nuclear waste
Process engineering
Tool vendors
Etc. Etc. Etc.
Fragmented by discipline opinions
Technical leaders
System architects
System analysts
Requirements engineers
Operations analysts
Design engineers
Fragmented by standards
ANSI/EIA-632
IEEE-1220
ISO-15288
CMMI
MIL-STD-499C
Toward Ontology for SE-ROI

9. Ontology
“…a branch of metaphysics concerned with the nature and relations of being”
aesthetics
interfaces
functions
structure
inputs
components
methods
outputs
categories
understanding
POSIWID – The Purpose of a Systems Is What It Does
Jack Ring
The purpose of systems engineering is different in the eyes of different people, because they perceive different actions/results from SE
Toward Ontology for SE-ROI

10. Purpose of this Paper Explore the variety of what people see in SE
Formulate some general categories
Interpret historical SE effort data
Provide a structure for the data-gathering in the SE-ROI project.
Toward Ontology for SE-ROI

11. COSYSMO work toward ontology
An exploration of ontology as performed in the COSYSMO project
Toward Ontology for SE-ROI

12. COSYSMO Effort Profile
How is Systems Engineering effort distributed over time?
Phase
Conceptualize
Develop
Operational
Test
& Eval
Transition to
Operation
%Effort
(STDEV)
23 (12)
36 (16)
27 (13)
14 (9)
Toward Ontology for SE-ROI

13. Effort Distribution Across ANSI/EIA 632 Fundamental Processes
Average
Standard Deviation
Acquisition & Supply 7%
3.5
Technical Management
17%
4.5
System Design
30%
6.1
Product Realization
15%
8.7
Technical Evaluation
31%
Toward Ontology for SE-ROI

14. Systems Engineering Effort Profile
Toward Ontology for SE-ROI

15. Categorization from Current Standards
The start of an ontology, by identifying the widely-accepted categories.
Toward Ontology for SE-ROI

16. Categories in the Standards
Mission/Purpose Definition
Requirements Management
System Architecting
System Implementation
Technical Analysis
Technical Management/Leadership
Verification & Validation
CMMI
ANSI/EIA-632
MIL-STD-499C
IEEE-1220
ISO-15288
Colored boxes on following slides show the terminology used by each standard
Toward Ontology for SE-ROI

17. Mission/Purpose Definition
Define the mission or purpose of the new/changed system.
Typically described in the language of the system users rather than in technical language
CMMI
Develop customer requirements
ANSI/EIA-632
Not included
MIL-STD-499C
Not included
IEEE-1220
Define customer expectations
ISO-15288
Stakeholder needs definition
Toward Ontology for SE-ROI

18. Requirements Management
Creation and management of requirements
Efforts to define, analyze, validate, and manage the requirements
CMMI
Requirements development
Requirements mgmt
ANSI/EIA-632
System design
Requirements definition
MIL-STD-499C
System requirements analysis and validation
IEEE-1220
Requirements analysis
ISO-15288
Requirements analysis
Toward Ontology for SE-ROI

19. System Architecting
Define the system in terms of its component elements and their relationships
Diagrams that depict the system, its environment, components, and relationships
CMMI
Technical solution
ANSI/EIA-632
System design
Solution definition
MIL-STD-499C
System product technical requirements analysis and validation
Design or physical solution representation
IEEE-1220
Synthesis
ISO-15288
Architectural design
System life cycle mgmt
Toward Ontology for SE-ROI

20. System Implementation
Development/completion of the system
Specific system-level efforts in the standards are system integration and transition to use
ANSI/EIA-632
Product realization
Implementation
Transition to use
CMMI
Product integration
MIL-STD-499C
Not included
IEEE-1220
Not included
ISO-15288
Implementation
Integration
Transition
Toward Ontology for SE-ROI

21. Technical Analysis System-level technical analysis
Assessment of system performance
Trade-off analysis of alternatives
CMMI
Measurement and analysis
ANSI/EIA-632
Technical evaluation
System analysis
MIL-STD-499C
Functional analysis, allocations and validation
Assessments of system effectiveness, cost, schedule, and risk
Tradeoff analyses
IEEE-1220
Functional analysis
Requirements trade studies and assessments
Functional trade studies and assessments
Design trade studies and assessments
ISO-15288
Requirements analysis
Toward Ontology for SE-ROI

22. Technical Management/Leadership
Guiding the engineering teams involved in system design programs
Size/complexity of teams demands leadership
CMMI
Project planning
Project monitoring & control
Supplier agreement mgmt
Process/product quality assur.
Configuration mgmt
Integrated project mgmt
Decision analysis/resolution
Quantitative project mgmt
Risk mgmt
ANSI/EIA-632
Technical Mgmt
Planning
Assessment
Control
MIL-STD-499C
Planning
Monitoring
Decision making, control, and baseline maintenance
Risk mgmt
Baseline change control
Interface mgmt
Data mgmt
Subcontract mgmt
Technical reviews/audits
IEEE-1220
Technical mgmt
Track analysis data
Track requirements and design changes
Track performance
Track product metrics
Update specifications
Update architectures
Update plans
Maintain database
ISO-15288
Planning, Assessment, Control
Decision mgmt
Config mgmt
Acquisition, Supply
Resource mgmt
Risk mgmt
Toward Ontology for SE-ROI

23. Verification & Validation
Verification: comparison of the system with its requirements through objective evidence.
Validation: comparison of the system or requirements with the intended mission
ANSI/EIA-632
Technical Evaluation
Requirements validation
System verification
End products validation
CMMI
Verification
Validation
MIL-STD-499C
Design or physical solution verification and validation
IEEE-1220
Requirement verification
Functional verification
Design verification
ISO-15288
Verification
Validation
Quality mgmt
Toward Ontology for SE-ROI

24. Future Directions
Where is SE-ROI going?
Toward Ontology for SE-ROI

25. Project Advisory Group
Group of interested people/organizations
Communicate via web, telecon, meetings
Help define the data organization
Build public interest in the project
Provide access to real programs
View interim (protected) data as it develops
Current members come from:
AF Institute of Technology
Northrop Grumman
DOD Office of Secy of Defense
DRS
Johns Hopkins Univ
MIT
The Mitre Corp
NAVAIR
Raytheon
Rand Corporation
Rafael
Systems & Software Consortium
Univ of South Australia
USN Chief Engineer
For information, see
Toward Ontology for SE-ROI

26. Summary Systems engineering current state of knowledge is fragmented
Broadly-accepted ontology is needed
SE-ROI project needs categorization now
Structure the data to be correlated
Discover leading SE indicators
Identify SE best practices and methods
Categorization across standards helps develop the needed ontology
Toward Ontology for SE-ROI

27. Questions? Eric Honour +1 (850) 479-1985 ehonour@hcode.com
Dr. Ricardo Valerdi
+1 (617)
For information, see
Toward Ontology for SE-ROI

28. Survey of SE-ROI Knowledge
1992 Gruhl (NASA)
Project Definition – NASA
Program definition 10-15% reduces cost overruns
1990 Ancona/ Caldwell
Boundary Management Study
Boundary management averages 14%; more is better
2000 Miller (MIT)
Large Engineering Projects Study
Programs value cost over schedule over tech
Leadership important
1995 Franz (Boeing)
Impact of Systems Engineering on Quality and Schedule
Better SE led to significant cost reduction
2003 Barker (IBM)
Systems Engineering Effectiveness
Better SE reduced parametric costs by 30%
2004 Kludze (NASA)
Impact of Systems Engineering on Complex Systems
General belief that SE improves program cost
2004 Honour (SECOE)
Value of Systems Engineering
Greater SE 10-15% reduces cost/schedule overruns
Toward Ontology for SE-ROI

29. Next Steps Define interview data sheets
Use this categorization
Identify and interview trial projects
Obtain initial data
Evaluate the interview data sheets
Identify and interview projects
Ongoing effort for 2-3 years
Perform statistical correlation work
Ongoing effort for duration of project
Interim reports to participating organizations
Final report expected 2009
Toward Ontology for SE-ROI