SE Effectiveness Committee 1 Systems Engineering Effectiveness Committee Activity Report NDIA SE Division Meeting 18 August 2010

SE Effectiveness Committee 2 Systems Engineering Effectiveness - Activity Since Last SE Division Meeting Form a subcommittee: to develop a research proposal for the next phase of the SE Effectiveness study. (Mike McLendon / Marvin Ebbert /Joe Elm) made up of IEEE and NDIA representatives to discuss SE Effectiveness study extension. (Joe Elm / Bob Lyons) to plan a workshop on system development performance measures focused on readiness to proceed at milestones. (Garry Roedler, Chris Miller) to organize a study and/or workshop on “Why good programs succeed and why bad programs fail”. (Al Brown, Chris Miller, Dave Griffith, Geoff Draper) to identify and collect available information on SEP and SEMP effectiveness. (Dona Lee, Bob Scheurer)

SE Effectiveness Committee 3 SE Effectiveness Study Extension (Business Case for Systems Engineering) – Status Situation: At the request of OSD, on May 6 we briefed Mr. Stephen Welby (DDR&E Director Systems Engineering) on the SE Effectiveness Study results -Study completed back in 2007 but it never gained much traction -Mr. Welby agreed that the info was good and he would be a conduit for making it known to DoD. -Mr. Welby saw the value of having data on more domains but also realized that if he did nothing to recognized the value of the work to date, we could not expect to get the additional data that he desires. -We need a solid and reasonable study plan with options to begin dialogue with OSD on next steps. -Mike McLendon, CTR OSD ATL, agreed to be engaged on this effort Action: Set-up Working Group with IEEE to expand SE Effectiveness study to a wider group Augmentation of action: Develop a research proposal detailing the next steps -In collaboration with IEEE and the DoD, expand and improve upon the NDIA SE Effectiveness Study, using survey to identify SE activities that show net benefit to program performance. -Develop recommendations to OSD for policy and guidance updates (if required) to implement the findings of the expanded survey. -Establish a means of structuring acquisition program reporting to facilitate the on-going collection of data on SE activity as a mechanism for continuous improvement Action Status Working group formed with the IEEE Aerospace & Electronic Systems Society (AES) to reach a far broader audience. -Also, Mr. Dan Goddard, Chief of AF Division, AFMC/EN, Mr. Clarence Gooden, Civ USAF SAF/AQRE, and Mr Marty Meth, former pentagon official, have joined the team Holding weekly LiveMeeting/Telecon -Reviewing NDIA SE Effectiveness survey documentation -Reviewing survey questions for suggested refinements / additions -Drafting to encourage NDIA and IEEE member participation -Developed a draft of research proposal for future work -Plan to report on progress at Annual SE Conference in San Diego

SE Effectiveness Committee 4 System Development Performance Measurement – Status Situation: WSARA required joint DT&E and SE tracking and reporting on MDAP achievement of measurable performance criteria -Goal is to enable objective, information-based decision making -Bring better measurement of performance criteria into Program Support Reviews (PSRs) Action: Plan a workshop on system development performance measures focused on readiness to proceed at milestones. This workshop should include OSD and the Services. Augmentation of action: Establish a plan to form a Working Group under the SEEC to address system development performance measures as required by WSARA and needed by OSD. -The plan should reflect a similar process as used for the Development Planning WG. -As part of the plan include the planning of a workshop focused on identifying information needs and potential measures. -The working group should include collaboration with PSM and participation from OSD and the Services. Action Status: Garry Roedler and Chris Miller developed a briefing on the planning and actions for establishing a special working group under the SEEC and preparation needed for a workshop on System Development Performance Measurement. The plan includes: -Formation of a Core Team -Analysis of industry resources -Development of a position paper -Invitations to and formation of the Working Group -Planning and conduct of the Workshop -Post-workshop actions – briefing OSD leadership, action plan, and working group report Collaboration with Practical Software and Systems Measurement (PSM) confirmed. Briefing on the planning and actions to be reviewed at the SEEC meeting, August 18

SE Effectiveness Committee 5 “Why Good Programs Succeed and why bad Programs Fail” – Status Situation: Strong interest in critical success and failure characteristics for programs -Propose NDIA SEEC do cross industry analysis on critical success/failure characteristics – work in conjunction with NDIA PM Committee –Leverage existing studies as basis and analyze for common threads (some from NDIA SED) Systemic Root Cause Analysis; SE Effectiveness Survey; GAO Studies; Army study (McGarry); NDIA Gulf Coast Chapter Study; and Other industry studies Consider full life cycle, especially pre-M/S A –Develop prioritized list of critical characteristics –Develop a set of recommendations Action: To organize a study and/or workshop on “Why good programs succeed and why bad programs fail”. Augmentation of action: The biggest concern is that over the past several years this question/task has been given to several different groups (SERC, MITRE, SE Effectiveness, Systematic Database, …). They all established achievable objectives and each one resulted in a credible product that met those objectives. Yet in the eyes of AT&L leadership none of them were ‘silver bullet’ solutions and therefore they were never embraced. The concern is not whether or not OSD (Jim Thompson) ‘wants’ this report, but one of skepticism that what is wanted is feasible. Chris Miller has agreed to confirm this task request with Jim, but the primary issue is that we don’t want our team to commit to a study that is dead on arrival (even before we begin). The subcommittee plans to discuss more at a side-bar during the August 18 th meeting. Action Status: This action is on hold, pending further confirmation/guidance from Jim Thompson and discussion by the sub- committee.

SE Effectiveness Committee 6 SEP and SEMP Effectiveness – Status Situation: Colonel Shawn P. Shanley, SAF/AQRE Director, Engineering and Technical Management Division, suggested the committee look into the effectiveness of SEPs and SEMPs Action: Identify and collect available information on SEP and SEMP effectiveness Augmentation of action: Dona Lee agree to investigate the background of this action item resulting from the April SE Division meeting to see what was intended as well as contact Col Shawn Shanley to see what the AF is doing in this area and report back to see whether there is anything further to do in this area. Action Status: Dona contacted Col Shanley and he sent her the Air Force SEP study but it is closehold. Dona has reviewed the document for discussion in our August 18 committee meeting. In a parallel effort, Bob Scheurer has developed a short draft questionnaire for soliciting SEP – SEMP development and utilization from NDIA industry members. It will be designed to ensure that responses are completely anonymous.

SE Effectiveness Committee 7 Situation: As a primary Committee Chair of the NDIA Systems Engineering Division, and/or a past Track Chair at the NDIA Systems Engineering Conference, Dona Lee and Al Brown are members of the Systems Engineering Conference 2010 Technical Program Committee (TPC), representing the Systems Engineering Effectiveness Committee. Action: NDIA SE Conference Paper Selection for the Systems Engineering Effectiveness track Augmentation of action: Provide the briefing order for the papers/panels assigned to you track. Added a SE Standards Panel (=3 papers) because we think standards represent best practices and there is a lot going on now referencing bringing some back Prioritized order of presentation of papers for the track. Action Status: Complete NDIA SE Conference 2010 Technical Program Committee, Abstract Review – Status