Presentation is loading. Please wait.

Presentation is loading. Please wait.

SE Effectiveness Working Group 23-Nov-2010 Inaugural Meeting Presenter:Joseph P. Elm Software Engineering Institute.

Similar presentations


Presentation on theme: "SE Effectiveness Working Group 23-Nov-2010 Inaugural Meeting Presenter:Joseph P. Elm Software Engineering Institute."— Presentation transcript:

1 SE Effectiveness Working Group 23-Nov-2010 Inaugural Meeting Presenter:Joseph P. Elm Software Engineering Institute

2 SE Effectiveness Working Group 23-Nov-2010 Agenda SE Effectiveness Working Group Initial Project: Support of NDIA / IEEE SE Effectiveness Study Background and History Current Efforts SEEWG Role

3 3 SE Effectiveness Working Group 23-Nov-2010 Purpose promote effective systems engineering by collecting and analyzing quantitative data on the impact of specific SE processes and practices on project performance. This data will contribute to the development of a stronger business case for systems engineering.

4 4 SE Effectiveness Working Group 23-Nov-2010 Objectives Identify principles and practices PROVEN to provide benefit to projects and programs Develop guidance, training, and tools to assist SE practitioners in applying these principles and practices assist system acquirers in ensuring that these principles and practices are applied to their projects Establish means of monitoring / tracking the application of SE principles and practices and their impacts with the intent of continuously improving project performance through better SE

5 5 SE Effectiveness Working Group 23-Nov-2010 Scope Address the collection, analysis, and distribution of quantitative evidence of the value of SE throughout the system life cycle Address this question from the perspectives of both suppliers and acquirers Seek data broadly across all market sectors Defense Aerospace Transportation Telecommunications etc.

6 6 SE Effectiveness Working Group 23-Nov-2010 Membership, Roles, and Responsibilities PositionResponsibilitiesIncumbentContact Info SEEWG Chair: Call and preside over WG meetings Manage and administer WG Report to AD-Processes Duties common to all WG members Joseph P. SEEWG Co-chair Preside over WB meetings in the absence of the Chair Assist the Chair in managing and administering the WG Duties common to all WG members Eric Sponsor Resource advocacy Status reporting to the INCOSE Technical Director, BOD, and external stakeholders.) Bob Swarz (AD-Processes) SEEWG Members Planning Research Analysis Technical writing Presenting Interviewing Bruce Elliott Joseph Elm Summer Fowler Eric Honour Steve Mazeika Nancy Roseberry Frank Sciulli Richard Sidley Dale Smith Tim Spencer

7 SE Effectiveness Working Group 23-Nov-2010 Agenda SE Effectiveness Working Group Initial Project: Support of NDIA / IEEE SE Effectiveness Study Background and History Current Efforts SEEWG Role

8 8 SE Effectiveness Working Group 23-Nov-2010 Agenda SE Effectiveness Working Group Initial Project: Support of NDIA / IEEE SE Effectiveness Study Background and History Current Efforts SEEWG Role Development Phasing Life Cycle Integration Systems Engineering Process Baselines Life Cycle Planning Integrated Teaming Systems Engineering Management

9 9 SE Effectiveness Working Group 23-Nov-2010 Background and History 1 The value of SE is appreciated by many, but disputed by some Quantitative evidence of the value of SE is sparse Greuhl, Walter: Lessons Learned, Cost/Schedule Assessment Guide. NASA Comptrollers Office, 1992 Honour, Eric; Understanding the Value of Systems Engineering NDIA SE Effectiveness Committee; A Survey of Systems Engineering Effectiveness - Initial Results. 2008

10 10 SE Effectiveness Working Group 23-Nov-2010 Background and History 2 In 2006, NDIA embarked on a project to collect quantitative evidence of SE Value NDIA formed the SE Effectiveness Committee (SEEC) The SEEC conducted the SE Effectiveness Study – Developed a survey collecting information from defense contractors Assessed SE capabilities by cataloging artifacts of SE processes Assessed project performance based on satisfaction of budget, schedule and requirements Assessed other factors influencing project performance – Received responses from 64 projects – Analyzed the data and identified the strength of relationships between SE activities and project performance – Results published results in 2007 and 2008 (http://www.sei.cmu.edu/reports/08sr034.pdf)http://www.sei.cmu.edu/reports/08sr034.pdf Showed valuable relationships between many (but not all) SE activities and project performance

11 11 SE Effectiveness Working Group 23-Nov-2010 Artifact-based assessment of SE Practices CMMI-SE/SW/IPPD v Process Areas 179 Goals 614 Practices 476 Work Products 14 Process Areas 31 Goals 87 Practices 199 Work Products Systems Engineering- related Filter 13 Process Areas 23 Goals 45 Practices 71 Work Products Size Constraint Filter Considered significant to Systems Engineering Survey content is based on a recognized standard (CMMI)

12 12 SE Effectiveness Working Group 23-Nov-2010 Assessment of Project Performance Assess TOTAL Project Performance Project Cost, Project Schedule, Project Scope Focus on commonly used measurements – EVMS, baseline management, requirements satisfaction, budget re- baselining and growth, milestone and delivery satisfaction Assessment of Other Factors Project Challenge – some projects are more complex than others Acquirer Capability – some acquirers are more capable than others Project Environment – projects executed in and deployed to different environments have different needs

13 13 SE Effectiveness Working Group 23-Nov-2010 NDIA SE Effectiveness Study The Bottom Line For the projects that did the most SE, 56% delivered the best project performance For the projects that did the least SE, only 15% delivered the best project performance. 39% 46% 15% 29% 59% 12% 31% 13% 56% Best Performance ( x > 3.0 ) Moderate Performance ( 2.5 x 3.0 ) Lower Performance ( x < 2.5 ) Lower Capability ( x 2.5 ) N = 13 Moderate Capability ( 2.5 < x < 3.0 ) N = 17 Higher Capability (x 3.0 ) N = 16 Gamma = 0.32 p = 0.04 PROJECT PERFORMANCE vs. TOTAL SE CAPABILITY

14 14 SE Effectiveness Working Group 23-Nov-2010 Composite Measures Summary of Relationships Strong Relationship Moderately Strong to Strong Relationship Moderately Strong Relationship Weak Relationship

15 15 SE Effectiveness Working Group 23-Nov-2010 Agenda SE Effectiveness Working Group Initial Project: Support of NDIA / IEEE SE Effectiveness Study Background and History Current Efforts SEEWG Role Development Phasing Life Cycle Integration Systems Engineering Process Baselines Life Cycle Planning Integrated Teaming Systems Engineering Management

16 16 SE Effectiveness Working Group 23-Nov-2010 Background and History 3 Presented study results and recommendations to OSD in 2007 Held discussions with IEEE in 2009 regarding extension of the study to a wider audience Briefed new OSD leadership in May-2010 Positive reception of findings Interested in using findings to improve value of SE processes within DoD Supportive of follow-on study So, Here we are today …

17 17 SE Effectiveness Working Group 23-Nov-2010 The Mission Assist the DoD and the SE community in achieving a quantifiable and persistent improvement in project performance through appropriate application of systems engineering principles and practices Publishing research results is not enough. We need to: – Identify principles and practices PROVEN to provide benefit This is an extension and a confirmation of the prior NDIA study – Assist DoD in developing the guidance to implement the principles and practices identified in study findings – Assist DoD in establishing a means of monitoring / tracking the results of efforts to improve SE – Assist DoD in institutionalizing these efforts so that they becomethe way DoD does business.

18 18 SE Effectiveness Working Group 23-Nov-2010 The Team National Defense Industrial Association Systems Engineering Division Institute of Electrical and Electronics Engineers Aerospace and Electronic Systems Society Carnegie Mellon University Software Engineering Institute (FFRDC) The Systems Engineering Effectiveness Committee

19 19 SE Effectiveness Working Group 23-Nov-2010 The Plan 1 SEES proven effective SE practices SE Effectiveness Study Business Case for SE SE framework Adoption by industry Recommendations for DoD Acquisition PolicyGuidanceTraining System AcquisitionSystem Development Data collection and monitoring Phase I Phase II Phase III Aids

20 20 SE Effectiveness Working Group 23-Nov-2010 The Plan 2 Phase I: Identify SE activities that show net benefit (i.e., value exceeding cost) to program performance. Expand previous SE Effectiveness study to survey a larger population Develop an SE Framework and a stronger Business Case for SE (BCSE) Phase II: Develop recommendations to OSD for policy guidance, and training to implement the findings of Phase I. Develop recommendations for OSD review Develop tools to implement policy and guidance –Suggested RFP language–Suggested Contract language –Program office training–SE assessment methods and tools (in collaboration with DAU, AFIT, et. al.)– SE artifact CDRLs Phase III: Establish a mechanism for continuous improvement of the BCSE Framework. On-going data collection from DoD programs Integration with program review processes

21 21 SE Effectiveness Working Group 23-Nov-2010 Survey Tenets All data will be submitted anonymously No data collected will identify the respondent, project, or organization\ All data will be handled confidentially Data will be submitted directly to a secure web site managed by the SEI – The SEI is a federally funded research and development center. It does not compete with any responding organizations, and frequently operates as a trusted broker in matters of confidential and proprietary information. Only authorized SEI staff will have access to the submitted data Only aggregated data will be released to the participants and the public No released data will be traceable to a project, person, or organization.

22 22 SE Effectiveness Working Group 23-Nov-2010 Schedule Phase I Jun-2010 – Mar-2012 Survey development, execution, and analysis BCSE Framework development Phase II Mar-2012 – Jun-2013 Recommendations to DoD for policy, guidance, and training Phase III Jun Oct-2013 Mechanisms for continued data collection and analysis

23 23 SE Effectiveness Working Group 23-Nov-2010 Agenda SE Effectiveness Working Group Initial Project: Support of NDIA / IEEE SE Effectiveness Study Background and History Current Efforts SEEWG Role Development Phasing Life Cycle Integration Systems Engineering Process Baselines Life Cycle Planning Integrated Teaming Systems Engineering Management

24 24 SE Effectiveness Working Group 23-Nov-2010 Task I: Manage participation of INCOSE Members 1 The survey target audience is Project Managers, Chief Engineers, Lead System Engineers, etc. of projects delivering products (not services) Not limited to defense industries – all industries are welcome Not limited to US companies – all are welcome Reaching potential respondents Grass roots approach – Reach our to organizational membership seeking participation Top down approach – Indentify and contact SE leadership at major companies represented in participating organizations

25 25 SE Effectiveness Working Group 23-Nov-2010 Task I: Manage participation of INCOSE Members 2 Grass roots approach Obtain INCOSE membership roster Broadcast an invitation to participate to members of INCOSE Top down approach Identify SE leadership at major companies – Work from INCOSE membership roster – Identify SE leaders within their companies – Contact them directly and solicit their support to: Identify potential respondents within their company Promote participation

26 26 SE Effectiveness Working Group 23-Nov-2010 Task II: Develop SE Framework and BSCW SEES proven effective SE practices SE Effectiveness Study Business Case for SE SE framework Adoption by industry Recommendations for DoD Acquisition PolicyGuidanceTraining System AcquisitionSystem Development Data collection and monitoring Phase I Phase II Phase III Aids INCOSE input

27 27 SE Effectiveness Working Group 23-Nov-2010 Task III: Support Industry Adoption SEES proven effective SE practices SE Effectiveness Study Business Case for SE SE framework Adoption by industry Recommendations for DoD Acquisition PolicyGuidanceTraining System AcquisitionSystem Development Data collection and monitoring Phase I Phase II Phase III Aids INCOSE input Tools Training Consulting

28 28 SE Effectiveness Working Group 23-Nov-2010 Lets Make this New Study a Success ! For more information, contact: Joseph P. Elm Software Engineering Institute


Download ppt "SE Effectiveness Working Group 23-Nov-2010 Inaugural Meeting Presenter:Joseph P. Elm Software Engineering Institute."

Similar presentations


Ads by Google