1. Proposed Approach to SERC EM Task: Assessing SysE Effectiveness in Major Defense Acquisition Programs (MDAPs) Barry Boehm, USC-CSSE 26 November 2008

2. Outline EM Task Statement of Work – Definition of “Systems Engineering Effectiveness” EM Task Schedule and Planned Results Candidate Measurement Methods, Evaluation Criteria, Evaluation Approach Survey and evaluation instruments Workshop objectives and approach 01/29/20092

3. EM Task Statement of Work Develop measures to monitor and predict system engineering effectiveness for DoD Major Defense Acquisition Programs – Define SysE effectiveness – Develop measurement methods for contractors, DoD program managers and PEOs, oversight organizations For weapons platforms, SoSs, Net-centric services – Recommend continuous process improvement approach – Identify DoD SysE outreach strategy Consider full range of data sources – Journals, tech reports, org’s (INCOSE, NDIA), DoD studies Partial examples cited: GAO, SEI, INCOSE, Stevens/IBM GFI: Excel version of SADB Deliverables: Report and presentation – Approach, sources, measures, examples, results, recommendations 01/29/20093

4. Measuring SysE Effectiveness - And measuring SysE effectiveness measures Good SysE correlates with project success – INCOSE definition of systems engineering, “An interdisciplinary approach and means to enable the realization of successful systems” Good SysE not a perfect predictor of project success – Project does bad SysE, but gets lucky at the last minute and finds a new COTS solution, producing a great success – Project does great SysE, but poor managers and developers turn it into a disaster Goodness of a candidate SysE effectiveness measure (EM) – Whether it can detect when a project’s SysE is leading the project more toward success than toward failure Heuristic for evaluating a proposed SysE EM – Role-play as underbudgeted, short-tenure project manager – Ask “How little can I do and still get a positive rating on this EM?” 01/29/20094

5. 5 RESL as Proxy for SysE Effectiveness: A COCOMO II Analysis Used on FCS

6. 01/29/20096 Added Cost of Minimal Software Systems Engineering

7. 01/29/20097 How Much Architecting is Enough? Percent of Project Schedule Devoted to Initial Architecture and Risk Resolution Added Schedule Devoted to Rework (COCOMO II RESL factor) Total % Added Schedule 10000 KSLOC 100 KSLOC 10 KSLOC Sweet Spot Sweet Spot Drivers: Rapid Change: leftward High Assurance: rightward

8. EM Task Schedule and Results 01/29/20098 PeriodActivityResults 12/8/08 – 1/28/09Candidate EM assessments, surveys, Interviews, coverage matrix Initial survey results, candidate EM assessments, coverage matrix 1/29-30/09SERC-internal joint workshop with MPTProgress report on results, gaps, plans for gap followups 2/1 – 3/27/09Sponsor feedback on results and plans; Sponsor identification of candidate pilot organizations; Execution of plans; suggested EMs for weapons platform (WP) pilots Identification of WP pilot candidate organizations at Contractor, PM/PEO. Oversight levels; Updated survey, EM evaluation, recommended EM results 3/30-4/3/09 weekSERC-external joint workshop with MPT, sponsors, collaborators, pilot candidates, potential EM users Guidance for refining recommended EMs; Candidates for pilot EM evaluations 4/7 – 5/1/09Tailor lead EM candidates for weapons platform (WP) pilots; SADB-based evaluations of candidate EMs Refined, tailored EM candidates for weapons platform (WP) pilots at Contractor, PM-PEO, oversight levels; Pilot evaluation guidelines 5/6-7/09Joint workshop with MPT, sponsors, collaborators, pilot candidates, stakeholders; Select WP EM pilots Selected pilots; Guidance for final preparation of EM candidates and evaluation guidelines 5/11 – 7/10/09WP EM pilot experiments; Analysis and evaluation of guidelines and results; Refinement of initial SADB evaluation results based on EM improvements EM pilot experience database and survey results; Refined SADB EM evaluations 7/13 – 8/14/09Analyze WP EM pilot and SADB results; Prepare draft report on results, conclusions, and recommendations Draft report on WP and general EM evaluation results, conclusions, and recommendations for usage and research/transition/education initiatives 8/17-18/09Workshop on draft report with sponsors, collaborators, WP EM deep-dive evaluators, stakeholders Feedback on draft report results, conclusions, and recommendations 8/19 - 9/4-09Prepare, present, and deliver final reportFinal report on WP and general EM evaluation results, conclusions, and recommendations for usage and research/transition/education initiatives

9. Outline EM Task Statement of Work – Definition of “Systems Engineering Effectiveness” EM Task Schedule and Planned Results Candidate Measurement Methods, Evaluation Criteria, Evaluation Approach Survey and evaluation instruments Workshop objectives and approach 01/29/20099

10. Candidate Measurement Methods NRC Pre-Milestone A & Early-Phase SysE top-20 checklist Air Force Probability of Program Success (PoPS) Framework INCOSE/LMCO/MIT Leading Indicators Stevens Leading Indicators (new; using SADB root causes) USC Anchor Point Feasibility Evidence progress UAH teaming theories NDIA/SEI capability/challenge criteria SISAIG Early Warning Indicators/ USC Macro Risk Tool 01/29/200910

11. Independent EM Evaluations and Resolution 01/29/200911 Candidate EM USC Stevens FC-MD UAH PoPS Leading Indicators X X X INCOSE LIs X X Stevens LIs X X X SISAIG LIs/ Macro Risk X X X NRC Top-20 List X X X SEI CMMI-Based LIs X X X USC AP-Feasibility Evidence X X X UAH Team Effectiveness X X X

12. Extended SEPP Guide EM Evaluation Criteria Quality Attributes Accuracy and Objectivity. Does the EM accurately and objectively identify the degree of SE effectiveness? Or is it easy to construct counterexamples in which it would be inaccurate? Level of Detail. How well does the EM cover the spectrum between strategic and tactical data collection and analysis? Scalability. How well does the EM cover the spectrum between simple weapons platforms and ultra-large systems of systems with deep supplier chains and multiple concurrent and interacting initiatives? Ease of Use/Tool Support. Is the EM easy to learn and apply by non-experts, and is it well supported by tools, or does it require highly specialized knowledge and training? Adaptability. Is the EM easy to adapt or extend to apply to different domains, levels of assessment, changing priorities, or special circumstances? Maturity. Has the EM been extensively used and improved across a number and variety of applications? Cost Attributes Cost To Collect Data Cost To Relate Data To Effectiveness Schedule Efficiency Key Personnel Feasibility Functional Attributes 1201/29/2009

13. Criteria validation (EM) - 2 1301/29/2009

14. Candidate EM Coverage Matrix 01/29/200914 SERC EM Task Coverage Matrix V1.0 NRC Probability of Success SE Leading Indicators LIPSF (Stevens) Anchoring SW Process (USC) PSSES (U. of Alabama) SSEE (CMU/SEI) Macro Risk Model/Tool Concept Dev Atleast 2 alternatives have been evaluated xxx x (w.r.t NPR) (x) Can an initial capability be achieved within the time that the key program leaders are expected to remain engaged in their current jobs (normally less than 5 years or so after Milestone B)? If this is not possible for a complex major development program, can critical subsystems, or at least a key subset of them, be demonstrated within that time frame? x(x)x x (5 years is not explicitly stated) (x) (seems to be inferrable from the conclusions) (x) (implies this) Will risky new technology mature before B? Is there a risk mitigation plan? xxx(x)xx Have external interface complexities been identified and minimized? Is there a plan to mitigate their risks? xxxxxx KPP and CONOPS At Milestone A, have the KPPs been identified in clear, comprehensive, concise terms that are understandable to the users of the system? x(x)x x (strongly implied) (x) (implied) xx At Milestone B, are the major system-level requirements (including all KPPs) defined sufficiently to provide a stable basis for the development through IOC? xx(x)xx (x) (There is no direct reference to this but is inferrable) x Has a CONOPS been developed showing that the system can be operated to handle the expected throughput and meet response time requirements? xx(x) x (x) (there is a mention of a physical solution. That's the closest in this regard) xx Legend: x = covered by EM (x) = partially covered (unless stated otherwise)

15. EM “Survey” Instruments Two web-/PDF-based “surveys” now in place EM set exposure, familiarity and use – Participants: industry, government, academia – Short survey, seeking experience with EM sets – Presently distributed only to limited audience – Finding individual metrics used, less so for EM sets EM evaluation form – Participants: Task 1 EM team members – Detailed survey, requires in-depth evaluation of EM’s – Plan follow-up interviews for qualitative assessments – Evaluation still in progress, too early to expect results

16. Exposure to EM sets Only two EM sets used (one by DoD, other by industry) Even in limited audience, EM sets not widely used Several have no exposure at all Some “union of EM sets” might be required

17. EM Task Schedule and Results 01/29/200917 PeriodActivityResults 12/8/08 – 1/28/09Candidate EM assessments, surveys, Interviews, coverage matrix Initial survey results, candidate EM assessments, coverage matrix 1/29-30/09SERC-internal joint workshop with MPTProgress report on results, gaps, plans for gap followups 2/1 – 3/27/09Sponsor feedback on results and plans; Sponsor identification of candidate pilot organizations; Execution of plans; suggested EMs for weapons platform (WP) pilots Identification of WP pilot candidate organizations at Contractor, PM/PEO. Oversight levels; Updated survey, EM evaluation, recommended EM results 3/30-4/3/09 weekSERC-external joint workshop with MPT, sponsors, collaborators, pilot candidates, potential EM users Guidance for refining recommended EMs; Candidates for pilot EM evaluations 4/7 – 5/1/09Tailor lead EM candidates for weapons platform (WP) pilots; SADB-based evaluations of candidate EMs Refined, tailored EM candidates for weapons platform (WP) pilots at Contractor, PM-PEO, oversight levels; Pilot evaluation guidelines 5/6-7/09Joint workshop with MPT, sponsors, collaborators, pilot candidates, stakeholders; Select WP EM pilots Selected pilots; Guidance for final preparation of EM candidates and evaluation guidelines 5/11 – 7/10/09WP EM pilot experiments; Analysis and evaluation of guidelines and results; Refinement of initial SADB evaluation results based on EM improvements EM pilot experience database and survey results; Refined SADB EM evaluations 7/13 – 8/14/09Analyze WP EM pilot and SADB results; Prepare draft report on results, conclusions, and recommendations Draft report on WP and general EM evaluation results, conclusions, and recommendations for usage and research/transition/education initiatives 8/17-18/09Workshop on draft report with sponsors, collaborators, WP EM deep-dive evaluators, stakeholders Feedback on draft report results, conclusions, and recommendations 8/19 - 9/4-09Prepare, present, and deliver final reportFinal report on WP and general EM evaluation results, conclusions, and recommendations for usage and research/transition/education initiatives

18. Target EM Task Benefits for DoD Identification of best available EM’s for DoD use – Across 3 domains; 3 review levels; planning and execution Early warning vs. late discovery of SysE effectiveness problems Identification of current EM capability gaps – Recommendations for most cost-effective enhancements, research on new EM approaches – Ways to combine EM strengths, avoid weaknesses Foundation for continuous improvement of DoD SysE effectiveness measurement – Knowledge base of evolving EM cost-effectiveness – Improved data for evaluating SysE ROI 01/29/200918

19. Candidate EM Discussion Issues Feedback on evaluation criteria, approach – Single-number vs. range asessment – Effectiveness may vary by domain, management level Experience with candidate EMs Additional candidate EMs – Dropped GAO, COSYSMO parameters, NUWC open-systems Feedback on EM survey and evaluation instruments – Please turn in your survey response Feedback on coverage matrix Other issues or concerns 01/29/200919

20. 6/18/08©USC-CSSE20 Macro Risk Model Interface 01/29/200920