Where have SE Standards (and Models) come from…and where are they going? INCOSE - WMA Donald J. Gantzer SAIC gantzerd@saic.com 703-412-3668 May 15 , 2007

Objectives To give an overall summary of key SE process standards and models To illustrate a top level comparison of them To indicate trends and uses (in DoD) To briefly address one key process – Technical Planning Disclaimer: The views and information presented here are the author’s alone and do not necessarily represent SAIC or DoD views.

Agenda SE Standards and Models Overview Evolution of Standards & Models ISO - 15288 EIA - 632 IEEE - 1220 CMMI - Dev Defense Acquisition Guide/SE Correlations across Application to Tech Planning Summary Some Key References

Oh…wither models & standards?! “The Process Standards / Models Quagmire”. Source: Systems and Software Consortium {SSCI}

SE Standards and Models Trends CMMI-Acq (07) SA-CMM CMMI-AM CMM for SW SW-CMM(97) CMMII v1.1(02) CMMI-Dev v2.0(06) EIA-731 [SECM](99) SE-CMM SECAM MilStd 499 A/B EIA/IS-632 GEIA-632 (99) GEIA-632A (07) ISO 15288(02) 08? 06+ IEEE-15288 (04) IEEE-1220(05) IEEE-1220 IEEE-1220(99) CMM- Capability Maturity Mode ISO -15288: SE-System Life Cycle Processes EIA - 632:Process for Engineering a System IEEE-1220: Application & Management of the SE Process INCOSE SE HB v3 (06) INCOSE SE HB v2a (04) 95 00 05 Sources: INCOSE, SEI

Overview of SE Standards + CMMI ISO/IEC - 15288 – SE – System life cycle processes “Establish a common framework for describing the life cycle of systems” – 25 processes; incl 7 project, 11 technical, and 7 agreement and enterprise; 62 pgs - it is a hi-level framework [descriptive] EIA - 632 – Processes for Engineering a System “Provide an integrated set of fundamental processes to aid a developer in the engineering or re-engineering of a system” - 33 Requirements in 13 processes; 120 pgs - it is less prescriptive, broader scope [e.g., incl enterprise environment] IEEE - 1220 – Application and Management of the SE Process “Defines the requirements for an enterprise’s total technical effort related to the development of products and processes that will provide life cycle support for the products. It prescribes an integrated technical approach to engineering a system and requires the application of management of the systems process throughout the product life cycle.” - 28 requirements 28; incl 14 general, 6 by LC stages, 8 in SE Process ; 87 pgs – CMMI - DEV – Capability Maturity Model Integration® - 27 process areas:11 Management, 6 Engineering, 5 Support, and 5 Process; 573 pgs

SE Standards & Models Life Cycle Phases ISO 15288 EIA 632 IEEE 1220 CMMI (informal) DoD/DAG [DoDI 5000.2] Concept Pre-system Definition concept, exploration, vision Development System Def., Subsystem design, Detailed design Detailed design; FAIT feasibility, design, development Tech Development; System Development.& Demonstration Production End product, physical, IT&E production, manufacturing, delivery Production & Deployment: LRIP Utilization operations Operations & Support [O&S]: FRIP Support support, maintenance, sustainment O&S: Sustainment Retirement disposal, phase out O&S:Disposal

(source: S. Sheard, SPC and J. Lake, SMi; 2004) Scope of SE Standards IEEE 1220 EIA 632 Level of Detail MIL-STD-499B EIA/IS 632 ISO/IEC 15288 As is to be expected, SE standards have a lower level of detail and a larger scope than individual discipline standards, such as MIL-STD-498. The first three (and so far only released) standards have the most detail and least breadth of scope of the SE standards, as they try to provide information on how to do systems engineering, and mostly confine themselves to engineering (plus some project management). The newer SE standards, which do not even recognize the term ”SE”, take a broader approach, adding first the emphasis on Supplier-Customer agreement and then all the other enterprise processes that must be in place to build systems. Breadth of Scope (source: S. Sheard, SPC and J. Lake, SMi; 2004) 11

SE/SW Standards Harmonization Road Map [source: SSCI] EIA 632 ISO/IEC JTC1/SC7 IEEE EIA Software Engineering … ISO/IEC 15288 IEEE 1220 ISO/IEC 12207 ISO/IEC 15504 An Harmonization road map Systems SC7 Internal Harmonization Joint Standards Body Harmonization Assessment Adapted from: SC7 WG7 N0560 Systems Engineering Study Group Report 11JAN02

Simple Generic SE Process source: Mil Std 499A/B and DAU guidance Inputs: System Analysis & Control Requirements Analysis Functional Analysis/allocation Synthesis Outputs: Note: Applied to AFIT/CSE SE Case Studies; http://www.afit.edu/cse/

IEEE 1220: Systems Engineering Process – (2005) Note: includes detailed flows for each activity; and an example SEMP toc Clause 4 - General Requirements Clause 6 – The SE Process SE process Polices & procedures for SE Planning the technical effort: Prepare/update engineering plan; schedule; tech plans. Development strategies Modeling & prototyping Integrated repository: data, tools. Integrated data package: HW, SW, LC processes, human. Specification tree Drawing tree System breakdown structure Integration of the SE effort: concurrent engr., Int. teams. Technical reviews Quality management Product and process improvement: re- engr., self-assess., LL. 1. Requirements Analysis 2. Requirements Validation 3. Functional Analysis 4. Functional Verification 5. Synthesis 6. Design Verification *Requirements/ Functional /Design trade studies & assessments 7. Systems Analysis* 8. Control [source:SSCI + mods from IEEE 1220]

EIA 632: Processes for Engineering a System (1999; reaffirmed 2003) Technical Management Planning Process Control Process Assessment Process Plans, Directives, & Status Outcomes & Feedback Supply Process Acquisition Process Acquisition and Supply CONOPS & Requirements System Design Requirements Definition Process Solution Definition Process* System Products Acquisition Request Architectures/Designs Product Realization Implementation Process Transition to Use Process Note: provides detailed activities and outcomes for each process Note: under revision to ‘harmonize’ w ISO 15288 Products Technical Support Product Verification Process Product Validation Process Requirements Validation Process* Systems Analysis Process* (source: graph from INCOSE SE Handbook v2)*

Draft update to EIA-632 (out for internal review) EIA - 632a *{source: GEIA report; R. Harwell, 11/05}; 4 domains; Systems Management, System Creation, System Evaluation, Enterprise Environment Planning Assessment Progress Assessment Control Requirements Definition Concept Definition Solution Definition System Definition Product Realization [Implementation/Transition] System Realization Systems Analysis Mission & Systems Analysis Requirements & End products Val., System ver. System V&V Supply & Acquisition Customer & Supplier Relationship Management Enterprise Support Resources & Infrastructure Governance Enterprise support LC Portfolio Mngt.

ISO 15288: System Life-Cycle Processes (2002) Source: INCOSE SE Handbook, v3.0 Each process has purpose, outcomes, and activities

CMMI Issues DoD Sponsor Report Programs execute at lower maturity levels than their organizations have achieved and advertised High-maturity practices are not consistently applied at the project level after contract award How to ensure new projects will incorporate CMMI processes Appraisal sampling procedures – how to ensure adequate coverage of the organizational unit Appraiser quality – training, consistency Lack of agreement on what constitutes Levels 4 and 5 Need to converge to a single representation Content of appraisal disclosure statements is lacking Inadequate training and education for acquirers Should CMMI be used for source selection? Source: NDIA SE Conf., 10/06 K. Baldwin

Summary Improvements in CMMI-Dev v1.2 [August, 2006] New material added to CMMI v1.2 to ensure processes used at project start-up Appraisal Disclosure Statement (ADS) improved Sampling rules improved High maturity appraiser certification instituted Additionally… CMMI guide for Acquisition Orgs to leverage Supplier’s efforts – 3/07 CMMI – Acq in final draft [summer 07?] More improvements are necessary Source: NDIA SE Conf., 10/06 K. Baldwin

Capability Maturity Model Integrated (CMMI- Dev) CMMI Core SE Related Examples Integrated Product and Process Development Supplier Sourcing SW Organizational Goal (OPD) Project Goal (IPM) IPPD SE Related Examples Hardware Related Examples SW Related Examples CMMI Core (now includes SS) * Sources incl EIA-731(SECM) Source: SEI

CMMI-Dev 1.2 Process Areas— Continuous Representation Category Process Area Project Management [added Acquisition Management] Project Planning Project Monitoring and Control Supplier Agreement Management [becomes Solicitation & Supplier Agreement Management ?] Integrated Project Management) Risk Management Quantitative Project Management (4) (bold is for Levels 2&3 capability; […] addresses correlation with draft CMMI-Acq ) Configuration Management Process and Product Quality Assurance Measurement and Analysis Decision Analysis and Resolution Causal Analysis and Resolution (5) Support [Acq] Requirements Management [Acq] Requirements Development [Acq] Technical Solution Product Integration [Acq drops this?] Verification Validation Engineering Organizational Process Focus; Definition Organizational Training; Process Performance (4) Organizational Innovation and Deployment(5) Process Management (Hi maturity level)

Defense Acquisition Guide (DAG) [11/04] – Chapter 4 on SE [Note: DAG/SE used ISO 15288, EIA- 632, IEEE 1220, and DAU 2001 SE Handbook as key references]

Core Technical Processes: Example Systems Engineering Process Architecture (SE Processes taken from DAG, Chapter 4) Technical Management Processes: Tech Planning Tech Decision Risk Requirements Assessment Analysis Management Management Core Technical Processes: Requirements Development Legend: Management Engineering Support Logical Analysis Design Solution Feedback from each process Implementation Integration Note: this illustration was adopted from some PI work done for DISA_by author Transition Technical Support Processes: Verification & Validation Interface Management Configuration & Data Management

SE Standards/Models Mapping - Management ISO-15288 EIA-632 IEEE-1220 CMMI-DEV DAG/SE Chap. 4 Project Planning Planning Planning tech effort, SE integration, of SE effort Project Planning; Integrated PM; Quantitative PM Technical Planning; Interface Mngt. Project Assessment Assessment Control Project Monitoring & Control; M&A; Technical Assessment; Project Control Project Monitoring & Control Decision Making Systems Analysis Measurement & Analysis (M&A); Decision Analysis & Resolution; Quantitative PM; Causal Analysis & Resolution; Decision Analysis Risk Mngt. Risk Mngt. Configuration Management (CM) CM CM; Int. repository/ data pkg. Requirements Mngt. Information Mngt. Control: info dissemination Control; Integrated DB/pkg. Integrated PM; Technical Data Mngt. Agreement : Acquisition & Supply Acquisition & Supply Supplier Agreement Mngt.; see also draft CMMI-Acq See other DAG chapters Enterprise: Environment, Life Cycle, Resource, & Quality Mngt. Environment & Enterprise Support [e.g., investment, resource, process mngt.] QM; Product & Process Improvement Organizational processes; Process & Product QA

SE Standard/Models Mapping - Technical ISO-15288 EIA-632 IEEE-1220 CMMI-Dev DAG/ Chap. 4 Stakeholder Requirements Definition Requirements Definition Requirements Analysis Requirements Development Systems Analysis (SA) Requirements & Functional Analysis; SA Logical Analysis Architectural Design Solution Definition Synthesis; SA; modeling, spec tree, SBS Technical Solution Design Solution Implementation Implementation; production prototyping, fabrication, assembly, production Integration integration Product Integration Verification System Verification Functional and Design Verification; Tech reviews Verification; [+Chap 9 – IT&E] Validation Requirements and End Products Validation Requirements Validation; Tech reviews, test Validation; [+Chap, 9 – IT&E] Transition Transition to Use Product Integration; Org. Innovation & Deployment; Org. Training Operation; Maintenance; and Disposal field support support operations, support, maintenance, sustainment, disposal See other DAG chapters.

Example: Apply practices to Technical Planning ISO - 15288 EIA - 632 IEEE - 1220 CMMI - Dev PM BoK & DoD(AT&L)/SSE Tech Planning considerations

ISO 15288 Project Planning activities “Purpose is to produce and communicate effective and workable project plans” Identify the project objectives and constraints Define the project scope as established in the agreement Establish a WBS based on evolving system architecture Define and maintain a project schedule based on project objectives and work estimates Project achievement criteria for the life cycle stage decision gates, delivery dates and major dependencies on external inputs or outputs Define the project costs and plan a budget Establish the structure of authorities and responsibilities for project work Define the infrastructure and services required by the project Plan the acquisition of materials, goods and enabling system services supplied from outside the project Generate and communicate a plan for technical mgmt. of the project, including the reviews Define the project measures to be generated and the associated data to be collected, validated and analyzed Generate a project quality plan Source: Summarized from ISO 15288

EIA-632 – Technical Planning Process Implementation Strategy stakeholders, applicable docs, process approaches, LC phases, integration, reporting requirements, implementation Technical Effort Definition Requirement types, db, risk mngt. process metrics, metrics/quality, cost objectives, TPMs, tasks, methods & tools, technology Schedule & Organization Event& calendar based schedules, resources, staffing/disciplines, team/ org structure Technical Plans Engineering, Risk mngt., Tech Review, V &V, other Work Directives Work packages, work authorizations Source: EIA-632

IEEE-1220 Planning the Technical Effort “Prepare and Implement the technical plans and schedules to guide the project toward accomplishment of its objectives and proper conclusion.” Engineering plan [example SEMP content] Master and Detail Schedules Technical plans Developmental strategies Modeling & prototyping Integrated repository, Data, Tools, and Integrated data package Hw, SW, Humans Life cycle processes Spec and Drawing trees; SBS Integration the SE effort Tech Reviews Quality management Product & process improvement Source: IEEE - 1220

CMMI – Project Planning Purpose is to supply and maintain plans that define project activities. Establish Estimates Estimate scope Establish Estimates of work products/attributes Define life cycle Determine effort & cost estimates Develop Project Plan Establish budget & schedule Identify risks Plan for data management, Plan for resources; Needed knowledge & skills Plan stakeholder involvement Establish the Plan Obtain commitment to the Plan Review plans that affect project Reconcile work & resource levels Obtain commitment Other key process area relationships: – Requirements Development, Monitoring & Control, Supplier Agreement Mngt. Integrated PM, Risk Mngt., Measurement & Analysis, … Source: CMMI-Dev

CMMI Generic Practices for Planning process area Perform the planning process Establish & maintain an Org policy for planning process Plan the planning process Provide resources Assign responsibility Train people Manage configurations Identify and involve relevant stakeholders Monitor and control the planning process Objectively evaluate adherence to the planning process Review status with higher level management [PEO?] …

PM – Book of Knowledge (PM-BoK) (Project Management) Scope* Integration (charter, scope statement, PMP) Communication* Risk* Quality* Human Resources* Time (definition, sequencing, estimation) Cost (estimation, budgeting) Procurement (purchase, acquisition, contracting) * Apply Planning, Execution & Control to each area DoD PMBoK Extension (2003): SE SW Acquisition Logistics T&E Manufacturing Source: www.PMI.org ; 3rd Edition, 2004

Planning Process example [INCOSE SE Handbook v3]

DoD/SSE Technical Planning emphasis Manage a Comprehensive Set of Requirements Define project scope w key stakeholders [FoS, SoS] Formulate, assess, select the preferred system concept Develop explicit and testable system/project requirements Develop a WBS [products & process] Resource & Staffing to the Technical Plan Organize and staff the project team [TA, PM, LSE, IPTs] Estimate the time and resource requirements [IMS, EVMS] Develop a project critical path Develop a project budget Develop and Managing Technical Baselines Identify, manage, and mitigate project risks [technical] Manage project changes and customer expectations Managing Event-based Technical Reviews Integrating Tech Planning into overall Program Planning & Management Context [IMP/IMS, EVMS, program Risks] Note: DoD is updating DAG/SE, SEP Prep Guide, and DoDI 5000.2 Source: Dinesh Verma; DoD/SSE Workshop

… and in summary ISO - 15288 is becoming a SE process ‘reference’ model IEEE - 1220 updated per ISO–15288; IEEE ‘adopted’ the 15288 w elaboration; further updates planned EIA - 632 is being ‘harmonized’ to the ISO – 15288 & CMMI CMMI used EIA 632, 731 as sources; next version will also look at 15288 and 1220 ISO 12207 (SW Engineering processes) is being harmonized with 15288; additionally a ”Life Cycle Management” standard being drafted both [in draft review INCOSE SE Handbook v3.0, 2006 is based on 15288; SE Certification will be based on it; It is to be released as a ISO tech report Coordination also underway w ISO 9001 DoD SE Revitalization: DAG SE now being revised; one area being ‘beefed up’ is Software; DAU is implementing a series of SE courses DoD SoSE Guide being piloted uses the DAG/Chapter 4 CMMI Guide for Acquirers; SE in Acquisition Contracts Guide NDIA-SE draft Systems Assurance Guide uses ISO 15288 and EIA-632 IEEE working on an SE guidance & certification (by specialties)

SE Sources Links: ISO 15288: http://www.15288.com/ EIA 632: http://www.geia.org/index.asp?bid=552 IEEE 1220: http://www.software.org/quagmire/descriptions/ieee1220.asp SEI-CMMI: http://www.sei.cmu.edu/cmmi/models/models.html#modules Systems & Software Consortium: http://www.systemsandsoftware.org/ DAU-DAG: http://akss.dau.mil/dag/ DoD(AT&L) SE: http://www.acq.osd.mil/se/index.html AF Center for SE; http://www.afit.edu/cse/ INCOSE – Standards TC http://www.incose.org/practice/techactivities/standards.aspx References: “SE Standards & Models Compared”; J. Lake (SMi) and S. Sheard (SPC), INCOSE 2004 “Evolution of a Standard EIA-632”; R. Harwell, INCOSE 2006 “Cross-Cultural Issues Associated with the Application of ISO/IEC Standard”; T. Ferris, INCOSE 2006 “INCOSE Insight: Special Feature: Standards in SE”, April 2007; see particularly K. Crowder, D. Kitterman, T. Doran, R. Harwell, and S. Arnold articles.

Backup slides

Some SE Process References ISO/IEC 15288: System Engineering – System Life Cycle Processes; 2002 EIA/IS 632: Processes for Engineering a System; 1998 &? IEEE 1220: Application and Management of the Systems Engineering Process; 2005 CMMI© – Capability Maturity Model Integration for Systems Engineering, Software Engineering, and Integrated Product and Process Development, 1.2 (2006) Defense Acquisition Guide, Chapter 4 - Systems Engineering; Defense Acquisition University, 2004 CMMI – Acq: Adapting CMMI for Acquisition Organizations: A Preliminary Report; 2006 Understanding and Leveraging a Supplier’s CMMI Efforts; 2007 INCOSE Systems Engineering Handbook, v 3, 2006

NDIA Top 5 SE Issues 2003 2006 Lack of awareness of SE importance Source: NDIA SE Conference 10/06; M. Schaffer DUSD(A&T) SSE Lack of awareness of SE importance Lack of adequate qualified resources Insufficient SE tools and environments Inconsistent requirements definition Poor initial program formulation 2003 Inconsistent SE practices across all life cycle phases Insufficient quantity and quality of SE expertise Requirements not well managed or translated Insufficient SE early in the life cycle Inadequate tools and collaborative environments 2006

Taking SE to the Next Level Source: Honorable James L. Finely, DUSD(A&T), NDIA SE, 10/06 My challenge to you: Instilling upfront and continuous technical rigor Understanding Systems-of-Systems SE Fielding capability to the warfighter sooner Increasing combat systems fuel efficiency Addressing growing software concerns Sincere thanks to our partners Industry, in particular, NDIA Service and Agency Engineering Staffs “THE WILL TO CHANGE …”

DUSD(A&T) Vision for SE . . . Source: Honorable James L. Finely, DUSD(A&T), NDIA SE, 10/06 Systems and Software Engineering Center of Excellence Systems Engineering Revitalization World class leadership Broaden Software Engineering and System Assurance, expanding to Systems- of- Systems Responsive and agile, proactive to changing customer needs Focused technical assistance, guidance, and workforce education and training Competencies Impaired Elevated Stature Raised Awareness Positive Influence Delivered Product Suites Customer focused… Program Support is number one priority Credentials… Domain expertise Human capital (advanced degrees, DAWIA Level III certifications, lean/six sigma green belts) World Class… Outreach to a broad communities base Recognition and respect; can “compete” in a global marketplace (e.g., NATO, international symposia, advice sought by MODs, etc.) Organizational stature… Postured for success (forward-thinking, innovative, etc.) Credibility and rank . . . the Technical Foundation that Enables Acquisition Excellence

Sound technical planning to support warfighting capabilities Systems Engineering MS A MS B MS C CD B1 Joint Concepts JCD EOA Rpt TD & RR CDD CPD SPG FAA FNA EOA SDD PD O&S FRP DR OSD/ JCS COCOM FCB Operational Support Capability Based Assessments Demonstrate capabilities meet user needs OSD/JCS COCOM Sound technical planning to support warfighting capabilities Define relationships with related capabilities, architectures (e.g., GIG) Enterprise Assess portfolio performance (CAR) Identify alternatives; trade cost, sched, perf SoS Integrate SoS; assess cost, sched, perf Determine system performance parameters and verification plans System Assess system performance against capability needs Identify incremental, system specifications Components Integrate and test SE is the technical foundation for building acquisition knowledge over time Fully integrated SE approach: technical maturity, cost realism, risk mitigation Develop, test, and assess increments of capability Source: Honorable James L. Finely, DUSD(A&T), NDIA SE, 10/06