Presentation is loading. Please wait.

Presentation is loading. Please wait.

Value of Systems Engineering; Summary Report 1/04 1 Value of Systems Engineering Summary report SECOE and related projects Eric Honour INCOSE Director,

Similar presentations


Presentation on theme: "Value of Systems Engineering; Summary Report 1/04 1 Value of Systems Engineering Summary report SECOE and related projects Eric Honour INCOSE Director,"— Presentation transcript:

1 Value of Systems Engineering; Summary Report 1/04 1 Value of Systems Engineering Summary report SECOE and related projects Eric Honour INCOSE Director, Sponsored Research

2 Value of Systems Engineering; Summary Report 1/04 2 Agenda Background Heuristic Claims of SE Value Gathered results on Value of SE NASA Tracking 1980s Boundary Management study Large Engineering Projects MIT study Impact of SE at NASA (SECOE 02-02) Impact of SE on Quality & Schedule Boeing SE Effectiveness IBM study Value of SE research (SECOE 01-03)

3 Value of Systems Engineering; Summary Report 1/04 3 Heuristic Claim of SE Better systems engineering leads to Better system quality/value Lower cost Shorter schedule SYSTEM DESIGN DETAIL DESIGN PRODUCTION INTEGRATION TEST Traditional Design Time Risk Saved Time/ Cost System Thinking Design Time Risk

4 Value of Systems Engineering; Summary Report 1/04 4 NASA Tracking 1980s Source Werner Gruhl NASA Comptrollers Office GRO76 OMV GALL IRAS TDRSS HST TETH LAND76 MARS MAG GOES I-M CEN ACT CHA.REC. SEASAT DE UARS SMM EDO ERB77 STS LAND78 COBE GRO82 ERB88 VOY EUVE/EP ULYS PIONVEN IUEISEE HEAO

5 Value of Systems Engineering; Summary Report 1/04 5 Boundary Management Study Study of 45 high-tech new product development teams...Ancona and Caldwell, Research Technology Management, 1990 Significant portion of time is spent at team boundaries Individual Time Spent n Outside Team14%* * Typically limited to few individuals n Within Team38% n Alone48%

6 Value of Systems Engineering; Summary Report 1/04 6 Boundary Management in Technical Teams Boundary management occurs in four different roles. Level of effort in each role changes with time. Ambassador Ambassador Buffering, building support, reporting, strategy Task Coordinator Task Coordinator Lateral group coordination, info transfer, planning, negotiating Scout Scout Obtain possibilities from outside - interface with marketing Guard Guard Withhold information, prevent disclosure CREATION DEVELOPMENT DIFFUSION

7 Value of Systems Engineering; Summary Report 1/04 7 Boundary Management Study Significant Findings: High-performing teams did more external interaction than low-performing teams System technical leadership creates greater success Internal team dynamics (goals, processes, individual satisfaction) did not correlate with performance Process definition is important but not sufficient.

8 Value of Systems Engineering; Summary Report 1/04 8 Cost Targets Schedule Targets Objective Targets 82% Percent of Projects Meeting: Large Engineering Projects Study of 60 LEPs (power generation, transportation, oil production, technology) The Strategic Management of Large Engineering Projects, MIT Press 2000 Evaluation by interviews and by objective and subjective quality measures. 72%45%18%37% Failed!

9 Value of Systems Engineering; Summary Report 1/04 9 Large Engineering Projects Significant Findings: Most important determinant was a coherent, well-developed organizational/team structure A structure of leadership creates greater success Technical difficulties, social disturbance, size were not statistically linked to performance All projects had turbulent events Technical excellence could not save a socially unacceptable project Process definition is important but not sufficient.

10 Value of Systems Engineering; Summary Report 1/04 10 Impact of SE at NASA (SECOE 02-02) Survey research within NASA Form with 38 questions, answers on graded scale Typical questions: On your most recent project, what percent of your total project cost was spent on Systems Engineering tasks? On your most recent project, did systems engineering shorten or lengthen the planned schedule? Aggressive survey pursuit with management push NASA: 250 sent, 54% valid response INCOSE: 650 sent, 38% valid response Engineering of Complex Systems – The Impact of Systems Engineering at NASA, A.K.P.Kludze, Jr. doctoral dissertation George Washington Univ. 2003

11 Value of Systems Engineering; Summary Report 1/04 11 Response Demographics Participating Organization NASA136 INCOSE243 Total379 Work Experience % % 21+22% Age Groups % % 60+10% Education Level Bachelor34% Master55% Doctor11% Job Titles SE56% PM17% Other27%...significant differences by organization

12 Value of Systems Engineering; Summary Report 1/04 12 Key Survey Results - Cost Percent Spent on SE 0-5%6-10% 11-15%16% + Respondents marked bracket to show percent of total cost spent on SE on last project. Mode at 6-10% of project Few projects spent 11-15% Unexplained bimodal response >16% (perhaps interpretation of project) Cost Benefit of SE Very PoorPoorFairGoodExcellent Respondents believe strongly in cost benefit of SE In secondary question, few respondents could quantify

13 Value of Systems Engineering; Summary Report 1/04 13 Key Survey Results - Schedule At What Stage is SE Most Effective? Very EarlyMidway EndNo MatterNo Need Vast majority of respondents believe that SE is most effective very early in a project. Impact of SE on Schedule ShortenLengthenDont Know INCOSE respondents believe SE shortened schedule on most recent project NASA respondents uncertain Secondary questions uncertain in the quantification....other results available in dissertation

14 Value of Systems Engineering; Summary Report 1/04 14 Impact of Systems Engineering on Quality and Schedule Empirical evidence obtained from three parallel (same time) projects Each developed a complex, robotic Universal Holding Fixture (UHF) Each used a different level of SE Results are compared TraitUHF1UHF2UHF3 Size10 x 408 x 506 x 14 Accuracy ±0.005±0.003 Contact SensorsNone57108 Vacuum Sensors Real-time checksNoYes Probe contoursNoYes NC interfaceNoYes Impact of Systems Engineering on Quality and Schedule – Empirical Evidence, W. Forrest Frantz, Boeing Corp. 1995

15 Value of Systems Engineering; Summary Report 1/04 15 Project Differences Project TraitUHF1UHF2UHF3 Sys mgmt experienceLowLow-Medium Subcontract approachDesign ReviewsFull-time SE on site Access to SE supportLowHigh, but not used High, used Requirements approachToken reqsComplete, detailed, integrated reqs Design approachHW/SW specs, multi-org approach Functional specs fully address HW/SW processes and interfaces Functional adherenceDesign docs took precedence; specs updated per design Specs followed, CCB control Design reviewsWeekly team reviews Formal internal; little external Formal internal and external Integration approachPatterned after design Drive by functional specs; defined early in life cycle Acceptance testingDefined in high- level plan Formal tests based on Reqs and Functional specs

16 Value of Systems Engineering; Summary Report 1/04 16 Impacts Use of better SE reduced Overall cycle time Time to create reqs Time to design/produce Time to test...even in the face of more complex, higher quality systems! Overall Development Time (weeks) Requirements to RFP (weeks)Design to Production (weeks)

17 Value of Systems Engineering; Summary Report 1/04 17 Systems Engineering Effectiveness Study of 8 software product development projects during upgrade of SE processes Determining Systems Engineering Effectiveness, Bruce Barker, IBM Commercial Products, Conference on Systems Integration, Stevens Institute 2003 Evaluation by cost and schedule against a standard estimating method. Identify affected components Evaluate Impact, Complexity Convert to points Estimate Cost, Schedule Historical Database, Cost per Point New Product Concept Product Line Architecture Costing method applies only to project management, business management, systems engineering, system integration, and delivery into production. Application development costs are not included. © Copyright IBM Corp 2003 Used With Permission

18 Value of Systems Engineering; Summary Report 1/04 18 Project Data $K/Point Averages Without SE With SE $1,350/pt $944/pt $1,454/pt $1,142/pt $818/pt Project 1 Project 2 Project 3 Project 4 Project 5 Project 6 Project 7 Project 8 12,934 1,223 10,209 8,707 4,678 5,743 14, ,191 2,400 11,596 10,266 5,099 5,626 10,026 1, ,406 1,962 1,136 1,179 1, ,739 YearProjectPointsCost ($K) SE Costs (%) $K/ Point © Copyright IBM Corp 2003 Used With Permission

19 Value of Systems Engineering; Summary Report 1/04 19 Timeline of Projects SE organization created SE Process documented SE Formal Training Started Project w/o SE Project With SE As the Systems Engineering process has been enabled and integrated through the organization, productivity has increased Yearly Avg Cost per Point © Copyright IBM Corp 2003 Used With Permission

20 Value of Systems Engineering; Summary Report 1/04 20 Systems Engineering Effectiveness Significant Findings: Impact and complexity provide an effective method to perform parametric costing. Early parametric costing works. © Copyright IBM Corp 2003 Used With Permission Preliminary data indicates that the use of Systems Engineering will improve project productivity when effectively combined with the Project Management and Test Processes. Systems engineering improves productivity. $K/Point Averages Without SE With SE $1,350/pt $944/pt $1,454/pt $1,142/pt $818/pt

21 Value of Systems Engineering; Summary Report 1/04 21 Value of SE (SECOE 01-03) Multi-year effort to obtain statistical data Correlate amount/quality of SE with project quality/success SE Effort Parameterized by Technical Size Technical Complexity Risk Level Development Quality (function of Technical Value, Cost, Schedule, Risk) 6-10% ?

22 Value of Systems Engineering; Summary Report 1/04 22 Respondent Data 43 respondents 1 project not completed Values $1.1M - $5.6B SE Cost 0.3% - 26% Cost, schedule, quality correlate better with Systems Engineering Effort: SEE = SE Qual * (SE Cost %)

23 Value of Systems Engineering; Summary Report 1/04 23 Cost Overrun vs. SE Effort Source: SECOE INCOSE 2003 Average Cost Overrun 90% Assurance (1.6)

24 Value of Systems Engineering; Summary Report 1/04 24 Schedule Overrun vs. SE Effort Source: SECOE INCOSE 2003 Average Schedule Overrun 90% Assurance (1.6)

25 Value of Systems Engineering; Summary Report 1/04 25 Test Hypothesis: Quality Source: SECOE INCOSE 2003 SE Effort Quality Hypothesis

26 Value of Systems Engineering; Summary Report 1/04 26 Conclusions Value of SE SE effort improves development quality Cost, schedule, subjective Hypothesis is supported by the data Optimum SE effort is 10-15% or more Cost, schedule overruns are minimized However, note wide dispersion of data Also note few data points at this level; most projects spent far less Quality of the SE effort matters Lower quality SE reduces effectiveness

27 Value of Systems Engineering; Summary Report 1/04 27 Company Participation Project benchmarking service funded by participants Aggregated data shared among participants Raw data protected by data blinding SECOE Blinded Raw Data Project Benchmarking data Statistical data Value of SE Reported results Participating Companies Public 1 month Participating Companies (all data) Quarterly Every two years

28 Value of Systems Engineering; Summary Report 1/04 28 Company Participation Data gathering Select 4-6 programs One day session per participating company every 4 months 1½-hour sessions with PM+SE of each program Data gathered by two SECOE researchers Forms & notes do not identify programs Reports Benchmark report within 30 days of each quarterly session, compares to all prior data 4-month reports to all participants with aggregated results from all data, all sources Participation price $??K for each year

29 Value of Systems Engineering; Summary Report 1/04 29 Questions? Eric Honour INCOSE Director for Sponsored Research Pensacola, FL, USA +1 (850)


Download ppt "Value of Systems Engineering; Summary Report 1/04 1 Value of Systems Engineering Summary report SECOE and related projects Eric Honour INCOSE Director,"

Similar presentations


Ads by Google