1. 1 DoD Systems and Software Engineering Taking it to the Next Level 25 Oct 2006 Mark D. Schaeffer Director, Systems and Software Engineering Office of the Deputy Under Secretary of Defense (A&T)

2. 2 Office of the Under Secretary of Defense Acquisition, Technology and Logistics USD, Acquisition Technology & Logistics DUSD, Acquisition & Technology Technical Advisor, Interoperability Dir, Systems of Systems Management Dir, Systems and Software Engineering Dir, Portfolio Systems Acquisition Industrial Programs Defense Procurement and Acquisition Policy Small Business Programs Defense Contract Management Agency Defense Acquisition University Flatter, Leaner, Empowered!

3. 3 Systems and Software Engineering Mission Statement  Shape acquisition solutions and promote early technical planning  Promote the application of sound systems and software engineering, developmental test and evaluation, and related technical disciplines across the Department's acquisition community and programs  Raise awareness of the importance of effective systems engineering and drive the state-of-the-practice into program planning and execution  Establish policy, guidance, education and training in collaboration with academia, industry, and government communities  Provide technical insight to program managers and leadership to support decision making Evolving System Engineering Challenges

4. 4 Systems Engineering Revitalization Framework GuidanceE&T Policy Program Support Acquisition Community SE and T&E Communities Academic Community Industry Associations Driving Technical Excellence into Programs!

5. 5 Systems Engineering Revitalization Effort  Issued Department-wide Systems Engineering (SE) policy  Issued guidance on SE, T&E, and SE Plans (SEPs)  Instituted system-level assessments in support of DAB, OIPT, DAES, and in support of programs  Established SE Forum to ensure senior-level focus within DoD  Integrating DT&E with SE policy and assessment functions--focused on effective, early engagement of both  Instituting a renewed emphasis on modeling and simulation in acquisition  Working with Defense Acquisition University to revise curricula (SPRDE, T&E, PQM, LOG, PM, ACQ, FM, CONT)  Leveraging close working relationships with industry and academia Necessary but not sufficient!

6. 6 Vision for Systems Engineering and Software  Competencies Improved  Delivered Product Suite Courseware Policy/Guidance Program Support methods  Elevated Stature  Raised Awareness  Positive Influence  World class leadership  Broaden to Software Engineering, System Assurance, Complex Systems-of- Systems  Responsive and agile, proactive to changing customer needs  Focused technical assistance, guidance, and workforce education and training Systems and Software Engineering Centers of Excellence... the Technical Foundation that Enables Acquisition Excellence Systems Engineering Revitalization

7. 7 Why the Focus on Software…  Research investment has been static or declining DARPA computer science R&D funding 50% ↓ (’01 - ’04, universities)  Requirements growth 10X (% functionality) ’60s -’00s  Need vs. skilled/clearable workforce - gaps increasing  President’s Information Technology Advisory Committee Report, February 2005 Identifies SW as “major vulnerability” Recommends priority attention: “Secure Software Engineering and Software Assurance” and “Metrics, Benchmarks, and Best Practices”  Cost, schedule and performance issues Software is an increasingly, important factor

8. 8 Systems and Software Engineering in Programs Reduces Costly Mistakes $222.8B RDT&E FYDP** **SAR data for MAIS and MDAP programs under OSD Systems Engineering Oversight 33% historical RDT&E Cost Growth RDT&E Mistakes Under estimating engineering effort is Major source of error % of RDT&E Total (CG) Yields a Potential Applied to $73.52B RDT&E Cost Growth FYDP Source: DoD Cost Avoidance Study (CAIG) 10 year ongoing * SSE positive impact on just 1/3 of RDT&E mistakes (11%) $24.51B RDT&E Cost Avoidance FYDP Yields a Potential 1/3 * SSE impact

9. 9 Director, Systems & Software Engineering Mark Schaeffer SES Deputy Director Enterprise Development Bob Skalamera SES Deputy Director Developmental Test & Evaluation Chris DiPetto SES Deputy Director Software Engineering & System Assurance Mark Schaeffer (Acting) SES Deputy Director Assessments & Support Dave Castellano SES CORE COMPETENCIES SE Policy SE Guidance SE in Defense Acquisition Guidebook Technical Planning Risk Management Reliability & Maintainability Contracting for SE SoS SE Guide SE Education and Training DAU SE Curriculum SPRDE Certification Rqmt Corrosion R-TOC Value Engineering CORE COMPETENCIES DT&E Policy DT&E Guidance T&E in Defense Acquisition Guidebook TEMP Development Process DT&E Education and Training DAU DT&E Curriculum DT&E Certification Rqmt Joint Testing, Capabilities & Infrastructure Targets Oversight Acq Modeling & Simulation Energy DSOC/Acq Tech Task Force CORE COMPETENCIES SWE and SA Policy SWE and SA Guidance SoS, SA Guides SWE and SA Education and Training DAU SW Acq Curriculum Continuous Learning Modules for SWE, SoS, SA Software Engineering Acquisition Support Software Engineering Institute (SEI) Process Improvement CMMI Sponsor DoD/National Software Investment Strategy CORE COMPETENCIES Support of ACAT I and Other Special Interest Programs (MDAP, MAIS) Assessment Methodology (Program Support Reviews - PSRs) T&E Oversight and Assessment of Operational Test Readiness (AOTR) Systems Engineering and Developmental Test Planning and Support Lean/6-Sigma Training/Cert State of Systems Engineering Director, Systems & Software Engineering Acquisition program excellence through sound systems and software engineering Est. Aug 06

10. 10 DoD Software Center of Excellence Organizational Tenets  Support Acquisition Success Ensure effective and efficient software solutions across the acquisition spectrum of systems, SoS and capability portfolios  Improve the State-of-the-Practice of Software Engineering Advocate and lead software initiatives to improve the state-of- the-practices through transition of tools, techniques, etc.  Provide Leadership, Outreach and Advocacy Implement at Department and National levels, a strategic plan for meeting Defense software requirements  Foster Software Resources to meet DoD needs Enable the US and global capability to meet Department software needs, in an assured and responsive manner Promote World-Class Leadership for Defense Software Engineering

11. 11 What We’re Finding and What We’re Hearing

12. 12 Top 10 Emerging Systemic Issues Major contributors to poor program performance 1. ManagementIPT roles, responsibilities, authority, poor communication Inexperienced staff, lack of technical expertise 2. RequirementsCreep/stability Tangible, measurable, testable 3. Systems EngineeringLack of a rigorous approach, technical expertise Process compliance 4. StaffingInadequate Government program office staff 5. ReliabilityAmbitious growth curves, unrealistic requirements Inadequate “test time” for statistical calculations 6. Acquisition StrategyCompeting budget priorities, schedule-driven Contracting issues, poor technical assumptions 7. ScheduleRealism, compression 8. Test PlanningBreadth, depth, resources 9. SoftwareArchitecture, design/development discipline Staffing/skill levels, organizational competency (process) 10. Maintainability/LogisticsSustainment costs not fully considered (short-sighted) Supportability considerations traded

13. 13 Top 5 SE Issues NDIA Workshop – July 2006 1.Key systems engineering practices known to be effective are not consistently applied across all phases of the program life cycle. 2.Insufficient systems engineering is applied early in the program life cycle, compromising the foundation for initial requirements and architecture development. 3.Requirements are not always well-managed, including the effective translation from capabilities statements into executable requirements to achieve successful acquisition programs. 4.The quantity and quality of systems engineering expertise is insufficient to meet the demands of the government and the defense industry. 5.Collaborative environments, including SE tools, are inadequate to effectively execute SE at the joint capability, system of systems (SoS), and system levels. NDIA Systems Engineering Division - Top 5 SE Issues July 26-27, 2006

14. 14 Top 7 Software Issues NDIA/DoD Workshop – August 2006 1.The impact of requirements upon software is not consistently quantified and managed in development or sustainment. 2.Fundamental system engineering decisions are made without full participation of software engineering. 3.Software life-cycle planning and management by acquirers and suppliers is ineffective. 4.The quantity and quality of software engineering expertise is insufficient to meet the demands of government and the defense industry. 5.Traditional software verification techniques are costly and ineffective for dealing with the scale and complexity of modern systems. 6.There is a failure to assure correct, predictable, safe, secure execution of complex software in distributed environments. 7.Inadequate attention is given to total lifecycle issues for COTS/NDI impacts on lifecycle cost and risk. NDIA Systems Engineering Division - Top SW Issues August 24-25, 2006

15. 15 Defense Software Strategy Summit  OSD led Summit co-hosted by NDIA Oct 18-19, 2006 Build on the Top 7 SW Issue Workshop Findings  Nearly 100 participants from across DoD, industry and academia  Identified numerous issues, barriers and actionable recommendations across: Software acquisition and sustainment Software policy Human capital Software engineering practices  Next steps: Final report forthcoming Prioritize work, develop action plans, and move out

16. 16 Challenges Remain  Implementing a DoD vision and strategy for software  Component and Industry adoption and effective implementation of sound SE practices as early as possible in the system life cycle  SE Working Integrated Product Teams (SE WIPTs)  Retention and development of technical acumen in an aging and shrinking acquisition workforce  Meeting all requests for technical support to programs  SE support to Acquisition Initiatives stemming from the QDR  Continue to evolve “high visibility” initiatives: - Energy - System-of-Systems - CMMI- Modeling & Simulation - Defense Safety - System Assurance Oversight Council

17. 17 DoD Centers of Excellence How do we get there?

19. 19 CMMI: New Release and Next Steps Issues:  Integrity of CMMI appraisals  Misperception and misuse of the CMMI by acquirers Actions:  Implemented changes to the CMMI v1.2 product suite to ensure: Integrity of appraisals Quality of the product suite Education of acquirers Opportunities for streamlining where appropriate  Developing a CMMI model for Acquirer process improvement Partnership with General Motors Stakeholders cross DoD, Govt Agencies and Industry  Writing a CMMI guidebook Help acquirers understand what CMMI is and is not  Conducting study of actual process implementation post-Level 5

20. 20 System Assurance Issues:  Vulnerability of our systems to malicious tampering or access  Numerous assurance, protection and safety initiatives that are not aligned Actions:  Developing a comprehensive System Assurance strategy  Promoting nationwide collaboration  Identifying standards activities to address system vulnerabilities  Developing a Handbook for Engineering System Assurance Guidance for PMs and Engineers on how Systems Engineering practice can be applied to mitigate system vulnerability to malicious control/tampering

21. 21 System-of-Systems Engineering  DUSD (A&T) directed OSD-led effort to develop and publish System-of-Systems (SoS) Systems Engineering guide 6-month effort addressing areas of agreement across community Initial focus on SoS with stated requirements and organizations responsible for execution Addresses DAG technical process and considerations for technical management across system life cycle -Focused on systems engineering challenges characteristics of SoS and suggested approaches Audience: Program Managers and Lead/Chief Engineers for SoS acquisition programs, legacy systems, and constituent programs Draft of initial version of guide is out for review