1. University of Southern California Center for Systems and Software Engineering Barry Boehm and Jo Ann Lane University of Southern California Center for Systems and Software Engineering http://csse.usc.edu Presented at SEPG 2009 Avoid V-Nosediving or Spiraling Out of Control with the Incremental Commitment Model ----Tutorial----

2. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE2 Goals of Tutorial Describe –Current and new challenges with respect to the development of software-intensive systems –How the Incremental Commitment Model (ICM) and its capabilities can help developers meet these challenges –Use of risk patterns to determine which system elements can be done in an agile fashion and which should use more formal processes Review several case studies that illustrate challenges and the power of the ICM process decision table Conduct exercises to develop ICM model for various special cases

3. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE3 Tutorial Outline Challenges for developing next- generation software systems Overview of ICM Risk-based balance of agility and assurance ICM process decision table Guidance and examples for using the ICM ICM tailoring exercises

4. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE4 What are Current Lifecycle Concerns?

5. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE5 Current System Acquisition Methods V-Model 1 Spiral Model 2 High level guidance assumes that acquirers have extensive acquisition experience... Without experience, too easy to misinterpret and auger in with disastrous results... 1 http://en.wikipedia.org/wiki/V-Model 2 http://en.wikipedia.org/wiki/Spiral_modelhttp://en.wikipedia.org/wiki/V-Modelhttp://en.wikipedia.org/wiki/Spiral_model

6. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE6 Typical Acquisition Process Military pilot coming off a fighter plane is assigned to manage/oversee the acquisition of a new aircraft subsystem –Excellent understanding of aircraft personnel needs –No experience with system/software development –Conditioned to plan the flight and fly the plan –Will interpret V-model diagram sequentially –Will interpret spiral diagram as one-size-fits-all

7. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE7 Additional Future Process Challenges-I Multi-owner, multi-mission systems of systems –Integrated supply chain: strategic planning, marketing, merchandising, outsourcing, just-in-time manufacturing, logistics, finance, customer relations management –Over 50 separately evolving external systems –Need to satisfice among multiple stakeholders Rapid pace of change –In competition, mission priorities, technology, Commercial Off-the-Shelf (COTS), environment –Need incremental development to avoid obsolescence –Need concurrent vs. sequential processes –Need both prescience and rapid adaptability Software important; humans more important

8. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE8 Example: SoSE Synchronization Points

9. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE9 Priorities for Rapid Adaptation to Change Software is more adaptable than hardware People are generally more adaptable than software Need systems and software that better help people to adapt

10. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE10 Additional Future Process Challenges-II Emergence and human-intensiveness –Requirements not pre-specifiable –Need for evolutionary growth –Need to manage uncertainty and risk Brownfield vs. Greenfield development –Need to provide legacy continuity of service –Need to accommodate legacy, OTS constraints Always-on, never-fail systems –Need well-controlled, high-assurance processes –Need to synchronize and stabilize concurrency –Need to balance assurance and agility

11. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE11 The Broadening Early Cone of Uncertainty (CU) Need greater investments in narrowing CU –Mission, investment, legacy analysis –Competitive prototyping –Concurrent engineering –Associated estimation methods and management metrics Larger systems will often have subsystems with narrower CU’s Global Interactive, Brownfield Batch, Greenfield Local Interactive, Some Legacy

12. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE12 Lifecycle Evolution Motivators as Identified by CSSE Sponsors and Affiliates Current lifecycle models and associated practices –Based on outdated assumptions –Don’t always address today’s system development challenges, especially as systems become more complex Need better integration of –Systems engineering –Software engineering –Human factors engineering –Specialty engineering (e.g. information assurance, safety) Need to better identify and manage critical issues and risks up front (Young memo) Additional details found in Backup Charts…

13. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE13 An omniscient authority pre-establishes the requirements, preferred system concept, etc. (A4.1, 4.2, 4.4) Technical solutions are mapped and verified with respect to these (A4.1, 4.2, 4.4) They and technology do not change very often (A4.2, 4.5, 6.2) –Emphasis on rigor vs. adaptability The program has stable and controllable external interfaces (A4.5) Project organization is function-hierarchical and WBS-oriented (A3.1, 3.2) All requirements are equally important (A4.2) Reviews event/product-based vs. evidence-based (A5.2) The program’s critical path can be defined up front (A6.1) Can achieve optimum readiness at minimum life-cycle cost (A6.5) –No slack for contingencies Current SysE-SwE Guidance: Outdated Assumptions Example: SysE Plan Preparation Guide Sometimes OK for hardware; generally not for software

14. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE14 System/Software Architecture Mismatches - Maier, 2006 System Hierarchy –Part-of relationships; no shared parts –Function-centric; single data dictionary –Interface dataflows –Static functional- physical allocation Software Hierarchy –Uses relationships; layered multi-access –Data-centric; class- object data relations –Interface protocols; concurrency challenges –Dynamic functional- physical migration

15. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE15 Fractionated, incompatible sensor data management “Touch Football” interface definition earned value –Full earned value taken for defining interface dataflow –No earned value left for defining interface dynamics Joining/leaving network, publish-subscribe, interrupt handling, security protocols, exception handling, mode transitions –Result: all green EVMS turns red in integration Examples of Architecture Mismatches … Sensor 1 SDMS1 Sensor 2 SDMS2 Sensor 3 SDMS3 Sensor n SDMSn ……

16. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE16 Effect of Software Underrepresentation Software risks discovered too late Slow, buggy change management Recent large project reorganization SW (WBS-based) PM Sys EngrSoftwareC4ISR SensorsNetworks SW New

17. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE17 Tutorial Outline Challenges for developing next-generation software systems Overview of ICM Risk-based balance of agility and assurance ICM process decision table Guidance and examples for using the ICM ICM tailoring exercises

18. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE18 What is the ICM? Risk-driven framework for tailoring system life- cycle processes Integrates the strengths of phased and risk- driven spiral process models Synthesizes together principles critical to successful system development –Commitment and accountability of system sponsors –Success-critical stakeholder satisficing –Incremental growth of system definition and stakeholder commitment –Concurrent engineering –Iterative development cycles –Risk-based activity levels and anchor point milestones Principles trump diagrams… Used by 60-80% of CrossTalk Top-5 projects, 2002-2005

19. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE19 ICM Nature and Origins Integrates hardware, software, and human factors elements of systems engineering –Concurrent exploration of needs and opportunities –Concurrent engineering of hardware, software, human aspects –Concurrency stabilized via anchor point milestones Developed in response to DoD-related issues –Clarify “spiral development” usage in DoD Instruction 5000.2 Initial phased version (2005) –Explain Future Combat System of systems spiral usage to GAO Underlying process principles (2006) –Provide framework for human-systems integration National Research Council report (2007) Integrates strengths of current process models –But not their weaknesses ©USC-CSSE

20. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE20 ICM Integrates Strengths of Current Process Models—But not their Weaknesses V-Model: Emphasis on early verification and validation –But not ease of sequential, single-increment interpretation Spiral Model: Risk-driven activity prioritization –But not lack of well-defined in-process milestones RUP and MBASE: Concurrent engineering stabilized by anchor point milestones –But not software orientation Lean Development: Emphasis on value-adding activities –But not repeatable manufacturing orientation Agile Methods: Adaptability to unexpected change –But not software orientation, lack of scalability ©USC-CSSE

21. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE21 12/31/2007 ©USC-CSSE 21 Incremental Commitment in Gambling Total Commitment: Roulette –Put your chips on a number E.g., a value of a key performance parameter –Wait and see if you win or lose Incremental Commitment: Poker, Blackjack –Put some chips in –See your cards, some of others’ cards –Decide whether, how much to commit to proceed

22. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE22 12/31/2007 ©USC-CSSE 22 Scalable Remotely Controlled Operations

23. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE23 12/31/2007 ©USC-CSSE 23 Total vs. Incremental Commitment – 4:1 RPV Total Commitment –Agent technology demo and PR: Can do 4:1 for $1B –Winning bidder: $800M; PDR in 120 days; 4:1 capability in 40 months –PDR: many outstanding risks, undefined interfaces –$800M, 40 months: “halfway” through integration and test –1:1 IOC after $3B, 80 months Incremental Commitment [with a number of competing teams] –$25M, 6 mo. to VCR [4]: may beat 1:2 with agent technology, but not 4:1 –$75M, 8 mo. to FCR [3]: agent technology may do 1:1; some risks –$225M, 10 mo. to DCR [2]: validated architecture, high-risk elements –$675M, 18 mo. to IOC [1]: viable 1:1 capability –1:1 IOC after $1B, 42 months

24. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE24 The Incremental Commitment Life Cycle Process: Overview Anchor Point Milestones Synchronize, stabilize concurrency via FEDs Risk patterns determine life cycle process

25. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE25 ICM Activity Levels for Complex Systems

26. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE26©USC-CSSE Anchor Point Feasibility Evidence Descriptions Evidence provided by developer and validated by independent experts that: If the system is built to the specified architecture, it will –Satisfy the requirements: capability, interfaces, level of service, and evolution –Support the operational concept –Be buildable within the budgets and schedules in the plan –Generate a viable return on investment –Generate satisfactory outcomes for all of the success-critical stakeholders All major risks resolved or covered by risk management plans Serves as basis for stakeholders’ commitment to proceed Can be used to strengthen current schedule- or event-based reviews

27. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE27©USC-CSSE 27 15 July 2008 Rapid Change Creates a Late Cone of Uncertainty – Need incremental vs. one-shot development Uncertainties in competition, technology, organizations, mission priorities

28. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE28 The Incremental Commitment Life Cycle Process: Overview Anchor Point Milestones Concurrently engr. OpCon, rqts, arch, plans, prototypes Concurrently engr. Incr.N (ops), N+1 (devel), N+2 (arch)

29. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE29 Incremental Commitment In Systems and Life: Anchor Point Milestones Common System/Software stakeholder commitment points –Defined in concert with Government, industry organizations –Initially coordinated with Rational’s Unified Software Development Process Exploration Commitment Review (ECR) –Stakeholders’ commitment to support initial system scoping –Like dating Validation Commitment Review (VCR) –Stakeholders’ commitment to support system concept definition and investment analysis –Like going steady

30. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE30 Incremental Commitment In Systems and Life: Anchor Point Milestones (continued) Foundations Commitment Review (FCR) –Stakeholders’ commitment to support system architecting –Like getting engaged Development Commitment Review (DCR) –Stakeholders’ commitment to support system development –Like getting married Incremental Operational Capabilities (OCs) –Stakeholders’ commitment to support operations –Like having children

31. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE31 ICM Anchor Point Milestone Content (1) (Risk-driven level of detail for each element) Milestone ElementFoundations Commitment ReviewDevelopment Commitment Review Definition of Operational Concept Top-level system objectives and scope –System boundary –Environment parameters and assumptions –Evolution parameters Operational concept –Operations and maintenance scenarios and parameters –Organizational life-cycle responsibilities (stakeholders) Elaboration of system objectives and scope of increment Elaboration of operational concept by increment System Prototype(s) Exercise key usage scenarios Resolve critical risks Exercise range of usage scenarios Resolve major outstanding risks Definition of System Requirements Top-level functions, interfaces, quality attribute levels, including –Growth vectors and priorities –Prototypes Stakeholders’ concurrence on essentials Elaboration of functions, interfaces, quality attributes, and prototypes by increment Identification of TBD’s (to-be- determined items) Stakeholders’ concurrence on their priority concerns

32. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE32 ICM Anchor Point Milestone Content (2) (Risk-driven level of detail for each element) Milestone ElementFoundations Commitment ReviewDevelopment Commitment Review Definition of System and Software Architecture Top-level definition of at least one feasible architecture –Physical and logical elements and relationships –Choices of COTS and reusable software elements Identification of infeasible architecture options Choice of architecture and elaboration by increment –Physical and logical components, connectors, configurations, constraints –COTS, reuse choices –Domain-architecture and architectural style choices Architecture evolution parameters Definition of Life- Cycle Plan Identification of life-cycle stakeholders –Users, customer, developers, maintainers, interoperators, general public, others Identification of life-cycle process model –Top-level stages, increments Top-level WWWWWHH* by stage Elaboration of WWWWWHH* for Initial Operational Capability (IOC) –Partial elaboration, identification of key TBD’s for later increments Feasibility Evidence Assurance of consistency among elements above –Via analysis, measurement, prototyping, simulation, etc. –Business case analysis for requirements, feasible architectures Assurance of consistency among elements above All major risks resolved or covered by risk management plan *WWWWWHH: Why, What, When, Who, Where, How, How Much

33. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE33 Risk-Driven Scalable Spiral Model: Increment View Rapid Change High Assurance Short, Stabilized Development Of Increment N Increment N Transition/O&M Increment N Baseline Short Development Increments Foreseeable Change (Plan) Stable Development Increments

34. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE34 Risk-Driven Scalable Spiral Model: Increment View Agile Rebaselining for Future Increments Short, Stabilized Development of Increment N Verification and Validation (V&V) of Increment N Deferrals ArtifactsConcerns Rapid Change High Assurance Future Increment Baselines Increment N Transition/ Operations and Maintenance Future V&V Resources Increment N Baseline Current V&V Resources Unforeseeable Change (Adapt) Short Development Increments Foreseeable Change (Plan) Stable Development Increments Continuous V&V

35. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE35 Acquisition Via Spiral OODA Loops Life Cycle Architecture Milestone for Cycle Decide on next-cycle capabilities, architecture upgrades, plans Stable specifications, COTS upgrades Development, integration, V&V, risk management plans Feasibility evidence Act on plans, specifications Keep development stabilized Change impact analysis, preparation for next cycle (mini- OODA loop) Orient with respect to stakeholders priorities, feasibility, risks Risk/Opportunity analysis Business case/mission analysis Prototypes, models, simulations Observe new/updated objectives, constraints, alternatives Usage monitoring Competition, technology, marketplace ISR Operate as current system Accept new system — C4ISR vs. fruitcake metaphor

36. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE36 Agile Change Processing and Rebaselining Assess Changes, Propose Handling Stabilized Increment-N Development Team Change Proposers Future Increment Managers Agile Future- Increment Rebaselining Team Negotiate change disposition Formulate, analyze options in context of other changes Handle Accepted Increment-N changes Discuss, resolve deferrals to future increments Propose Changes Discuss, revise, defer, or drop Rebaseline future-increment Foundations packages Prepare for rebaselined future-increment development Defer some Increment-N capabilities Recommend handling in current increment Accept changes Handle in current rebaseline Proposed changes Recommend no action, provide rationale Recommend deferrals to future increments

37. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE37 Example ICM Commercial Application: Symbiq Medical Infusion Pump Winner of 2006 HFES Best New Design Award Described in NRC HSI Report, Chapter 5

38. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE38 Symbiq IV Pump ICM Process - I Exploration Phase –Stakeholder needs interviews, field observations –Initial user interface prototypes –Competitive analysis, system scoping –Commitment to proceed Valuation Phase –Feature analysis and prioritization –Display vendor option prototyping and analysis –Top-level life cycle plan, business case analysis –Safety and business risk assessment –Commitment to proceed while addressing risks

39. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE39 Symbiq IV Pump ICM Process - II Architecting Phase –Modularity of pumping channels –Safety feature and alarms prototyping and iteration –Programmable therapy types, touchscreen analysis –Failure modes and effects analyses (FMEAs) –Prototype usage in teaching hospital –Commitment to proceed into development Development Phase –Extensive usability criteria and testing –Iterated FMEAs and safety analyses –Patient-simulator testing; adaptation to concerns –Commitment to production and business plans

40. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE40 ICM and Brownfield Development Many process models are Greenfield- oriented –Requirements→Design→Develop→Test→Operate Failed Greenfield project example –Corporate central financial system –To replace spaghetti-code collection of COBOL programs Improved ICM Brownfield approach –Concurrent new-system definition and legacy system re-engineering

41. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE41 Failed Greenfield Corporate Financial System Used waterfall approach –Gathered requirements –Chose best-fit ERP system –Provided remaining enhancements Needed to ensure continuity of service –Planned incremental phase-in of new services Failed due to inability to selectively phase out legacy services –Dropped after 2 failed tries at cost of $40M

42. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE42 Budgeting Legacy Systems Patched, Highly Coupled Financial and Non-Financial Services Legacy Business Services Contract Services Project Services Deliverables Management Billing Staffing Work Breakdown Structure Subcontracting Scheduling Progress Tracking Change Tracking Reqs, Configuration Management Earned Value Management

43. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE43 ICM Approach to Brownfield Engineering Understanding needs –Analysis of legacy system difficulties Envisioning opportunities –Concurrently decouple legacy financial and non-financial services, explore new system phase-in and architecture options System scoping and architecting –Extract legacy financial, non-financial services –Prioritize, plan for incremental financial services phase-in/out Feasibility evidence development –Successful examples of representative service extractions –Evidence of cost, schedule, performance feasibility

44. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE44 Legacy Business Services Contract ServicesProject Services Result of Legacy Re-engineering Contract Financial Services Billing Subcontract payments... Contract Non- Financial Services Deliverables mgmt. Terms compliance... General Financial Services Accounting Budgeting Earned value Payroll... General Non- Financial Services Progress tracking Change tracking... Project Financial Services WBS Expenditure categories... Project Non- Financial Services Scheduling Staffing Reqs CM...

45. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE45 ICM Summary Current processes not well matched to future challenges –Emergent, rapidly changing requirements –High assurance of scalable performance and qualities Incremental Commitment Model addresses challenges –Assurance via evidence-based milestone commitment reviews, stabilized incremental builds with concurrent V&V Evidence shortfalls treated as risks –Adaptability via concurrent agile team handling change traffic and providing evidence-based rebaselining of next-increment specifications and plans –Use of critical success factor principles: stakeholder satisficing, incremental growth, concurrent engineering, iterative development, risk- based activities and milestones Major implications for funding, contracting, career paths

46. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE46 Tutorial Outline Challenges for developing next-generation software systems Overview of ICM Risk-based balance of agility and assurance ICM process decision table Guidance and examples for using the ICM ICM tailoring exercises

47. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE47 Using Risk to Balance Assurance and Agility - Overview

48. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE48 Risks Used in Risk Comparison Environmental risks –Technology uncertainties –Many stakeholders –Complex system-of-systems –Legacy compatibility Agility risks –Scalability –Use of simple design –Personnel turnover –Too-frequent releases –Not enough agile-capable people Plan-driven risks –Rapid change –Need for rapid results –Emergent requirements –Not enough discipline-capable people

49. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE49 Four Examples Three agent-based systems –Medium – SupplyChain.com application –Small – Event Managers application –Large – National Information System for Crisis Management (NISCM) application Medical case Each example results in a development strategy based on risk analyses

50. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE50 SupplyChain.com Profile Turnkey agent-based supply chain management systems Distributed, multi-organization teams of about 50 people Parts of applications are relatively stable, while others are highly volatile Architectures are driven by a few key COTS packages that are also continually evolving

51. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE51 SupplyChain.com Home Ground Chart

52. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE52 SupplyChain.com Risks

53. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE53 SupplyChain.com Mitigations

54. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE54 SupplyChain.com Strategy LCA LCO

55. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE55 Event Managers Project Small startup company Diverse set of smaller agent-based planning applications This project: support for managing the infrastructure and operations of conferences and conventions Widening variety of options and interdependencies, makes an agent-based approach highly attractive

56. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE56 Event Managers Home Ground Chart

57. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE57 Event Managers Risks

58. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE58 Event Managers Mitigations

59. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE59 Event Managers Strategy LCA

60. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE60 NISCM Profile Broad coalition of government agencies and critical private-sector organizations Support cross-agency and public-private sector coordination of crisis management activities Adaptive mobile network, virtual collaboration, information assurance, and information integration technology Private-sector system-of-systems integration contractor

61. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE61 NISCM Home Ground Chart

62. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE62 NISCM Risks

63. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE63 NISCM Mitigations

64. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE64 NISCM Strategy LCALCO

65. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE65 Risk Profiles Across Agent-Based Examples

66. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE66 Medical Case Study -- USA 1400 software people; 7M SLOC; 7 sites –4 in Europe, 2 in India 500 medical applications; 500 financial; others Survivability-critical software problems –Reliability, productivity, performance, interoperability –Sarbanes-Oxley requirements –Management receptive to radical change Some limited experimental use of agile methods –Led by top software technologist/manager Committed to total change around Scrum and XP

67. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE67 Medical-USA Adoption Profile July 2004 - July 2005 –Recruit top people from all sites into core team(s) –Get external expert help –Develop architecture –Early Scrum successes with infrastructure –Revise policies and practices –Train, reculture everyone –Manage expectations July 2005 – July 2006 –Begin full-scale development –Core teams as mentors

68. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE68 Key Practices – USA Medical Include customers and marketers –New roles; do’s/don’ts/opportunities; CRACK personnel; full collaboration and teamwork; expectations management Scrum; most XP practices; added company practices –6-12 person teams with team rooms, dedicated servers –Hourly smoke test; nightly build and regression test –Just-in-time analysis; story-point estimates; fail fast; detailed short-term plans; company architecture compliance –Embrace change in applications and practices –Global teams: wikis, daily virtual meetings, act as if next-door Release management –2-12 week architecting Sprint Zero; 3-10 1-month Sprints; Release Sprint; 1-6 month beta test –Next Sprint Zero concurrent with Release Sprint Initiative manager and team –Define practices; evolve infrastructure; provide training; guide implementation; evaluate compliance/usage; continuous improvement

69. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE69 Tutorial Outline Challenges for developing next-generation software systems Overview of ICM Risk-based balance of agility and assurance ICM process decision table Guidance and examples for using the ICM ICM tailoring exercises

70. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE70 The Incremental Commitment Life Cycle Process: Overview Anchor Point Milestones Synchronize, stabilize concurrency via FEDs Risk patterns determine life cycle process

71. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE71 The ICM as Risk-Driven Process Generator Stage I of the ICM has 3 decision nodes with 4 options/node –Culminating with incremental development in Stage II –Some options involve go-backs –Results in many possible process paths Can use ICM risk patterns to generate frequently-used processes –With confidence that they fit the situation Can generally determine this in the Exploration phase –Develop as proposed plan with risk-based evidence at VCR milestone –Adjustable in later phases

72. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE72 03/19/2008 ©USC-CSSE 72 Different Risk Patterns Yield Different Processes

73. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE73 The ICM Process Decision Table: Key Decision Inputs Product and project size and complexity Requirements volatility Mission criticality Nature of Non-Developmental Item (NDI)* support –Commercial, open-source, reused components Organizational and Personnel Capability * NDI Definition [ DFARS] : a) any product that is available in the commercial marketplace; b) any previously developed product in use by a U.S. agency (federal, state, or local) or a foreign government that has a mutual defense agreement with the U.S.; c) any product described in the first two points above that requires only modifications to meet requirements; d) any product that is being produced, but not yet in the commercial marketplace, that satisfies the above criteria.

74. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE74 The ICM Process Decision Table: Key Decision Outputs Key Stage I activities: incremental definition Key Stage II activities: incremental development and operations Suggested calendar time per build, per deliverable increment

75. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE75 Common Risk-Driven Special Cases of the ICM (Cases 1-4) Case 1: Use NDI Example: Small accounting system Size, Complexity: Size variable, complexity low Typical Change Rate/Month: Negligible Criticality: n/a NDI Support: Complete Organizational Personnel Capability: NDI-experienced (medium) Key Stage I Activities (Incremental Definition): Acquire NDI Key Stage II Activities (Incremental Development/Operations): Use NDI Time/Build: n/a Time/Increment: Vendor-driven Case 2: Agile Example: E-services Size, Complexity: Low Typical Change Rate/Month: 1-30 Criticality: Low to medium NDI Support: Good, in place Organizational Personnel Capability: Agile-ready, medium-high experience Key Stage I Activities (Incremental Definition): Skip Valuation and Architecting phases Key Stage II Activities (Incremental Development/Operations): Scrum plus agile methods of choice Time/Build: <= 1 day Time/Increment: 2-6 weeks Case 3: Architected Agile Example: Business data processing Size, Complexity: Medium Typical Change Rate/Month: 1-10 Criticality: Medium to high NDI Support: Good, most in place Organizational Personnel Capability: Agile-ready, medium to high experience Key Stage I Activities (Incremental Definition): Combine Valuation, Architecting phases. Complete NDI preparation. Key Stage II Activities (Incremental Development/Operations): Architecture-based Scrum of Scrums Time/Build: 2-4 weeks Time/Increment: 2-6 months Case 4: Formal Methods Example: Security kernel; Safety-critical LSI chip Size, Complexity: Low Typical Change Rate/Month: 0.3 Criticality: Extra high NDI Support: None Organizational Personnel Capability: Strong formal methods experience Key Stage I Activities (Incremental Definition): Precise formal specification Key Stage II Activities (Incremental Development/Operations): Formally-based programming language; formal verification Time/Build: 1-5 days Time/Increment: 1-4 weeks

76. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE76 Common Risk-Driven Special Cases of the ICM (Cases 5-8) Case 5: Hardware with Embedded Software Component Example: Multi-sensor control device Size, Complexity: Low Typical Change Rate/Month: 0.3 - 1 Criticality: Medium to very high NDI Support: Good, in place Organizational Personnel Capability: Experienced, medium-high Key Stage I Activities (Incremental Definition): Concurrent hardware/software engineering. CDR-level ICM DCR Key Stage II Activities (Incremental Development/Operations): IOC development, LRIP, FRP. Concurrent version N+1 engineering Time/Build: Software 1-5 days Time/Increment: Market-driven Case 6: Indivisible IOC Example: Complete vehicle platform Size, Complexity: Medium to high Typical Change Rate/Month: 0.3 - 1 Criticality: High to very high NDI Support: Some in place Organizational Personnel Capability: Experienced, medium to high Key Stage I Activities (Incremental Definition): Determine minimum- IOC likely, conservative cost. Add deferrable software features as risk reserve Key Stage II Activities (Incremental Development/Operations): Drop deferrable features to meet conservative cost. Strong award free for features not dropped. Time/Build: Software: 2-6 weeks Time/Increment: Platform: 6-18 months Case 7: NDI-Intensive Example: Supply chain management Size, Complexity: Medium to high Typical Change Rate/Month: 0.3 - 3 Criticality: Medium to very high NDI Support: NDI-driven architecture Organizational Personnel Capability: NDI-experienced, medium to high Key Stage I Activities (Incremental Definition): Thorough NDI-suite life cycle cost-benefit analysis, selection, concurrent requirements/architecture definition Key Stage II Activities (Incremental Development/Operations): Pro- active NDI evolution influencing, NDI upgrade synchronization Time/Build: Software: 1-4 weeks Time/Increment: Systems: 6-18 months Case 8: Hybrid Agile/Plan-Driven System Example: C4ISR system Size, Complexity: Medium to very high Typical Change Rate/Month: Mixed parts; 1-10 Criticality: Mixed parts; Medium to very high NDI Support: Mixed parts Organizational Personnel Capability: Mixed parts Key Stage I Activities (Incremental Definition): Full ICM, encapsulated agile in high change, low-medium criticality parts (Often HMI, external interfaces) Key Stage II Activities (Incremental Development/Operations): Full ICM, three-team incremental development, concurrent V&V, next- increment rebaselining Time/Build: 1-2 months Time/Increment: 9-18 months

77. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE77 Common Risk-Driven Special Cases of the ICM (Cases 9-11) Case 9: Multi-Owner Directed System of Systems Example: Net-centric military operations Size, Complexity: Very high Typical Change Rate/Month: Mixed parts; 1-10 Criticality: Very high NDI Support: Many NDIs, some in place Organizational Personnel Capability: Related experience, medium to high Key Stage I Activities (Incremental Definition): Full ICM; extensive multi-owner team building, negotiation Key Stage II Activities (Incremental Development/Operations): Full ICM; large ongoing system/software engineering effort Time/Build: 2-4 months Time/Increment: 18-24 months Case 10: Family of Systems Example: Medical device product line Size, Complexity: Medium to very high Typical Change Rate/Month: 1-3 Criticality: Medium to very high NDI Support: Some in place Organizational Personnel Capability: Related experience, medium to high Key Stage I Activities (Incremental Definition): Skip Valuation and Architecting phases Key Stage II Activities (Incremental Development/Operations): Scrum plus agile methods of choice Time/Build: 1-2 months Time/Increment: 9-18 months Case 11: Brownfield Example: Incremental legacy phaseout Size, Complexity: High to very high Typical Change Rate/Month: 0.3-3 Criticality: Medium-high NDI Support: NDI as legacy replacement Organizational Personnel Capability: Legacy re-engineering Key Stage I Activities (Incremental Definition): Re-engineer/refactor legacy into services Key Stage II Activities (Incremental Development/Operations): Incremental legacy phaseout Time/Build: 2-6 weeks/refactor Time/Increment: 2-6 months

78. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE78 Common Risk-Driven Special Cases of the ICM (Cases 12a/b) Case 12a: Net-Centric Services – Community Support Example: Community services or special interest group Size, Complexity: Low to medium Typical Change Rate/Month: 0.3-3 Criticality: Low to medium NDI Support: Tailorable service elements Organizational Personnel Capability: NDI-experienced Key Stage I Activities (Incremental Definition): Filter, select, compose, tailor NDI Key Stage II Activities (Incremental Development/Operations): Evolve tailoring to meet community needs Time/Build: <= 1 day Time/Increment: 2-12 months Case 12b: Net-Centric Services – Quick Response Decision Support Example: Response to competitor initiative Size, Complexity: Medium to high Typical Change Rate/Month: 3-30 Criticality: Medium to high NDI Support: Tailorable service elements Organizational Personnel Capability: NDI-experienced Key Stage I Activities (Incremental Definition): Filter, select, compose, tailor NDI Key Stage II Activities (Incremental Development/Operations): Satisfy quick response; evolve or phase out Time/Build: <= 1 day Time/Increment: Quick response-driven LEGEND C4ISR: Command, Control, Computing, Communications, Intelligence, Surveillance, Reconnaissance. CDR: Critical Design Review. DCR: Development Commitment Review. FRP: Full-Rate Production. HMI: Human-Machine Interface. HW: Hard ware. IOC: Initial Operational Capability. LSI: Large Scale Integration. LRIP: Low-Rate Initial Production. NDI: Non-Development Item. SW: Software

79. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE79 Tutorial Outline Motivation Overview of ICM Risk-based balance of agility and assurance ICM process decision table Guidance and examples for using the ICM process decision table Group Exercise: Case study using ICM process decision table Team Exercise: Applying ICM to 10 cases

80. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE80 Case 1: Use NDI Exploration phase identifies NDI opportunities NDI risk/opportunity analysis indicates risks acceptable –Product growth envelope fits within NDI capability –Compatible NDI and product evolution paths –Acceptable NDI volatility, some open-source components highly volatile –Acceptable usability, dependability, interoperability –NDI available or affordable –Example: Small accounting system Size/complexity: Low Anticipated change rate (% per month): Low Criticality: Low NDI support: Complete Organizational and personnel capability: NDI-experienced Key Stage I activities: Acquire NDI Key Stage II activities: Use NDI Time/build: Driven by time to initialize/tailor NDI Time/increment: Driven by NDI upgrades

81. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE81 Case 1 Elaboration Suppose you have an application for which an appropriate NDI (COTS, open source, reuse library, customer-furnished package) solution is available, and other options of developing perhaps a better version yourself or outsourcing such a development. Even if you produce a better solution (frequently not the case), you will generally incur more expense and take longer to begin to capitalize on its benefits. And you will often find that the NDI package has features that you hadn’t realized you would need, but are there when you need them. On the other hand, there are risks that may disqualify some NDIs. They may be overly complex for your needs, incompatible with your other applications, or highly volatile. See the discussion in Section 33.1 of Software Engineering Economics (Boehm, 1981) for more about the pros and cons of NDI solutions.

82. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE82 Case 2: Pure Agile Methods Exploration phase determines –Low product and project size and complexity –Fixing increment defects in next increment acceptable –Existing hardware and NDI support of growth envelope –Sufficient agile-capable personnel –Need to accommodate rapid change, emergent requirements, early user capability Example: E-services Size/complexity: Low Anticipated change rate (% per month): 1-30% Criticality: Low to medium NDI support: Good; in place Organizational and personnel capability: Agile-ready, medium to high capability Key Stage I activities: Skip Valuation and Architecting phases Key Stage II activities: Scrum plus agile methods of choice Time/build: Daily Time/increment: 2-6 weeks

83. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE83 Case 2 Elaboration If your project is small (less than 10 people) and its criticality involves the loss of discretionary vs. essential funds, a pure agile method such as Extreme Programming (Beck, 1999), Scrum (Schwaber, 2002), or Crystal (Cockburn, 2002) is generally best if you have relatively high-capability, agile-ready personnel. The risks of using a more formal, plan-driven approach are less adaptability to change and belated feedback on the product’s capabilities. The biggest risk is to try to develop the application all at once for several months, and then find that it is a mismatch to the users’ needs. Agile developers have found that the best way to address this risk is to organize the project into short (2-6 week) delivery increments that may be incomplete, but provide early useful capabilities. Any flaws can then be detected early and fixed in the next increment (for very high criticality applications, this would not be acceptable). On a small project it is also easy to set up a daily build and regression test structure that identifies integration problems early when they are easier to fix. Some risks of using the agile approach is that it is harder to write a contract for what is being developed; that it may sub optimize for early success by using unscalable capabilities (fourth-generation languages, running all in main memory); or high personnel turnover. See (Boehm and Turner, 2004), pp. 121-128 for an example risk analysis of an agent-based event planning system, ending with a decision to go agile.

84. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE84 Case 3: Architected Agile Exploration phase determines –Need to accommodate fairly rapid change, emergent requirements, early user capability –Low risk of scalability up to 100 people –NDI support of growth envelope –Nucleus of highly agile-capable personnel –Moderate to high loss due to increment defects Example: Business data processing Size/complexity: Medium Anticipated change rate (% per month): 1-10% Criticality: Medium to high NDI support: Good, most in place Organizational and personnel capability: Agile-ready, med-high capability Key Stage I activities: Combined Valuation and Architecting phase, complete NDI preparation Key Stage II activities: Architecture-based scrum of scrums Time/build: 2-4 weeksTime/increment: 2-6 months

85. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE85 Case 3 Elaboration For medium-size (20-80 people), medium complexity (reasonably mature and scalable technology; largely compatible shareholders), agile methods can be scaled using an Architected Agile approach with early investment in a largely change-prescient architecture and user/developer/customer team building. For relatively stable projects (0.3-1% change/month), plan-driven methods can be used with low risk. But for higher rates of changes (1-10%/month), a more agile approach is less risky. A risk analysis of a 50-person, medium sized architecture-based agile supply chain management project is provided on pages 106-121 of (Boehm and Turner, 2004). A number of organizations in such areas as corporate infrastructure, medical, aerospace, and ERP applications have reported significant gains in adaptability and quality of the Architected Agile approach over plan-driven methods for such projects. However, others that had less capable and agile-ready people, less management and customer commitment, and less up-front architecture investment have not. (Boehm, 2007)

86. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE86 Case 4: Formal Methods Biggest risks: Software/hardware does not accurately implement required algorithm precision, security, safety mechanisms, or critical timing Example: Security kernel or safety-critical LSI chip Size/complexity: Low Anticipated change rate (% per month): 0.3% Criticality: Extra high NDI support: None Organizational and personnel capability: Strong formal methods experience Key Stage I activities: Precise formal specification Key Stage II activities: Formally-based programming language; formal verification Time/build: 1-5 days Time/increment: 1-4 weeks

87. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE87 Case 4 Elaboration Formal methods involve the development and verification of a precise mathematical specification of the behavior of a hardware and/of software systems; an implementation of the system in formal-semantics-based languages; and a mathematical proof that the implementation is exactly equivalent to the specification (no more; no less). Such methods are expensive relative to standard commercial practice and require scarce high-capability personnel, but are inexpensive relative to a massive product recall, expensive lawsuits, or a massive loss of valuable and confidential information. Current formal methods generally require repeating the expensive mathematical proof process whenever the specification or implementation is changed, making the approach less viable for systems with highly volatile requirements. In general, non-developmental items are not precisely specified and verified enough to be safely used in such systems. Also, formal methods have limited scalability; almost all fully-verified software systems have less than 10,000 lines of code. However, some progress is being made toward modularizing such systems so that implementations and proofs can be built up incrementally via lower-level lemmas and theorems.

88. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE88 Case 5: Hardware Component with Embedded Software Biggest risks: Device recall, lawsuits, production line rework, hardware-software integration –DCR carried to Critical Design Review level –Concurrent hardware-software design Criticality makes Agile too risky –Continuous hardware-software integration Initially with simulated hardware Low risk of overrun –Low complexity, stable requirements and NDI –Little need for risk reserve –Likely single-supplier software

89. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE89 Case 5: Hardware Component with Embedded Software (continued) Example: Multi-sensor control device Size/complexity: Low Anticipated change rate (% per month): 0.3-1% Criticality: Medium to very high NDI support: Good, in place Organizational and personnel capability: Experienced; medium to high capability Key Stage I activities: Concurrent hardware and software engineering; CDR-level ICM DCR Key Stage II activities: IOC Development, LRIP,FRP, concurrent version N+1 engineering Time/build: 1-5 days (software) Time/increment: Market-driven

90. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE90 Case 5 Elaboration The application classes above have been mostly software-intensive. The differences in economic and risk patterns for hardware-intensive projects (Boehm and Lane, 2007) will create different risk-based special cases of the ICM. Once a project commits to a particular manufacturing approach and infrastructure, the hardware cost of change will be much higher. And the primary costs at risk are in development and manufacturing, particularly if the component is cheaper to replace than to fix or if fixes can be accomplished by software workarounds. Thus, the ICM Stage I activities of producing and validating detailed specifications and plans are much more cost-effective than for the agile cases above. They will often go beyond the usual level of detail (Critical Design Review versus evidence- based Preliminary Design Review) for an ICM Development Commitment Review (DCR), since so many of the details are likely to be major sources of rework expenditures and delay if wrong. And after Initial Operational Capability (IOC) development, the rework risks generally dictate an incremental low rate initial production phase before a full-rate production phase in Stage II. In case the component is planned to evolve to subsequent hardware-software reconfigurations, the ICM approach of having a concurrent systems engineering team developing specifications and plans for the “N+1st “ increment could be adopted. The length of these later increments will be driven by the product’s marketplace or competitive situation.

91. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE91 Case 6: Indivisible IOC Biggest risk: Complexity, NDI uncertainties cause cost- schedule overrun –Similar strategies to case 5 for criticality (CDR, concurrent HW- SW design, continuous integration) –Add deferrable software features as risk reserve Adopt conservative (90% sure) cost and schedule Drop software features to meet cost and schedule Strong award fee for features not dropped –Likely multiple-supplier software makes longer (multi-weekly) builds more necessary

92. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE92 Case 6: Indivisible IOC (continued) Example: Complete vehicle platform Size/complexity: Medium to high Anticipated change rate (% per month): 0.3-1% Criticality: High to very high NDI support: Some in place Organizational and personnel capability: Experienced, medium to high capability Key Stage I activities: Determine minimum-IOC likely, conservative cost; Add deferrable software features as risk reserve Key Stage II activities: Drop deferrable features to meet conservative cost; Strong award fee for features not dropped Time/build: 2-6 weeks (software) Time/increment: 6-18 months (platform)

93. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE93 Case 6 Elaboration More complex hardware intensive systems, such as aircraft, may have a great deal of software that can be incrementally developed and tested, but may have an indivisible hardware IOC that must be developed as a unit before it can be safely tested (e.g., an automobile or aircraft’s braking system or full set of safety-critical vehicle controls). Relative to the cost and duration of the software increments, the indivisible hardware IOC has two significant sources of risk: –It cannot fit into smaller software increments –It cannot drop required hardware features to meet IOC schedule/cost/quality as independent variable. –The first can be addressed by synchronizing the hardware IOC with the Nth software increment (e.g., (Rechtin and Maier, 1997) osculating orbits for hardware and software). If some combination of schedule/cost/quality is truly the project IOC’s independent variable, then one does not want to commit to a most-likely combination of schedule/cost/quality, as there will be a roughly 50% chance of an overrun or quality shortfall in completing the indivisible IOC. Rather, it is better to determine a conservative IOC cost and schedule for meeting the quality objective, and use the difference between the conservative cost and schedule and the most- likely cost and schedule as a risk reserve. The best way to do this is to use the conservative cost and schedule as the IOC target, and to determine a set of desired but non-essential, easy to drop software features that could be developed within the risk reserve. Then, if the most-likely indivisible IOC capability begins to overrun, some desired software features can be dropped without missing the scheduled delivery time and cost. There should also be a strong award-fee incentive for the developers to minimize the number of features that need to be dropped.

94. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE94 Case 7: NDI-Intensive Biggest risks: incompatible NDI; rapid change, business/mission criticality; low NDI assessment and integration experience; supply chain stakeholder incompatibilities Example: Supply chain management Size/complexity: Medium to high Anticipated change rate (% per month): 0.3-3% Criticality: Medium to very high NDI support: NDI-driven architecture Organizational and personnel capability: NDI-experienced; medium to high capability Key Stage I activities: Thorough NDI-suite life cycle cost-benefit analysis, selection, concurrent requirements and architecture definition Key Stage II activities: Pro-active NDI evolution influencing, NDI upgrade synchronization Time/build: 1-4 weeks (software) Time/increment: 6-18 months (systems)

95. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE95 Case 7 Elaboration Our experiences in developing USC web-service applications between 1996 and 2004 was that they went from 28% of the application’s functionality being delivered by NDI components to 80% (Yang et al, 2005). A similar trend was identified by the 2001 Standish Report, which reported that 53% of the functionality of commercial software applications was being delivered by NDI components in 2000 (Standish, 2001). The economics of NDI-intensive systems dictates a bottom-up versus a top-down approach to system development, in which the capability envelope of the NDI determines the affordable requirements, rather than a top-down requirements-to- capability approach. A large supply-chain management system may need to choose among several NDI candidates each for such functions as inventory control, trend analysis, supplier/customer relations management, transportation planning, manufacturing control, and financial transactions; and evaluate not only the candidates’ cost/performance aspects, but also their interoperability with each other and with the corporation’s legacy infrastructure. Besides NDI assessment, other significant sources of effort can be NDI tailoring, NDI integration, and effect of NDI version volatility and obsolescence; see (Yang et al, 2005). A particular challenge in Stage II is the effect of NDI volatility and obsolescence. Surveys have indicated that commercial NDI products have a new release about every 10 months, and that old releases are supported by the vendor for about 3 releases. Some large systems have had about 120 NDI components, indicating that about 12 components will have new releases each month, and that not upgrading will leave each component unsupported in about 30 months. In such cases, a great deal of attention needs to be paid to upgrade synchronization, and to pro-active NDI evolution influencing. Some large organizations synchronize their NDI upgrades to their major re-training cycles of about 12-18 months. For additional NDI best practices, see (Wallnau et al, 2002).

96. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE96 Case 8: Hybrid Agile/Plan-Driven System Biggest risks: large scale, high complexity, rapid change, mixed high/low criticality, partial NDI support, mixed personnel capability Example: C4ISR system Size/complexity: Medium to very high Anticipated change rate (% per month): Mixed parts; 1-10% Criticality: Mixed parts; medium to very high NDI support: Mixed parts Organizational and personnel capability: Mixed parts Key Stage I activities: Full ICM; encapsulated agile in high changed; low-medium criticality parts (often HMI, external interfaces) Key Stage II activities: Full ICM, three-team incremental development, concurrent V&V, next-increment rebaselining Time/build: 1-2 months Time/increment: 9-18 months

97. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE97 Case 8 Elaboration Some large, user-intensive systems such as C4ISR systems, air traffic control systems, and network control systems have a mix of relatively stable, high- criticality elements (sensors and communications; key business logic) and more volatile, more moderately critical elements such as GUI displays, electronic warfare countermeasures, and interfaces with externally evolving systems. As many acquisitions of this nature have shown, it is highly risky to apply one-size- fits-all processes and incentive structures to these differing classes of elements. Instead, in Stage I, it is important to determine which system elements belong in which class along with the other functions of understanding needs, envisioning opportunities, NDI assessment, scoping the system, and determining feasible requirements and increments performed for complex systems in Stage I; to architect the system to encapsulate the volatile parts for development by agile teams; and to organize to use plan-driven development for the other elements of the overall system. In Stage II, the cycles for the agile teams are likely to be significantly shorter than those for the plan-driven teams.

98. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE98 Case 9: Multi-Owner System of Systems Biggest risks: all those of Case 8 plus –Need to synchronize, integrate separately-managed, independently-evolving systems –Extremely large-scale; deep supplier hierarchies –Rapid adaptation to change extremely difficult Example: Net-centric military operations or global supply chain management Size/complexity: Very high Anticipated change rate (% per month): Mixed parts; 1-10% Criticality: Very high NDI support: Many NDIs; some in place Organizational and personnel capability: Related experience, medium to high Key Stage I activities: Full ICM; extensive multi-owner teambuilding, negotiation Key Stage II activities: Full ICM; large ongoing system/software engineering effort Time/build: 2-4 monthsTime/increment:18-24 months

99. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE99 Case 9 Elaboration In this situation, your goal is to integrate a set of existing systems (or guide and evolve the integration of a set of existing systems). These systems are primarily developed, owned, and maintained by an organization other than the one that is attempting to manage and guide the set of systems as a system of systems. Because of the independence of these constituent systems, the SoS organization has little or no formal control over the processes used to maintain and evolve the constituent systems. The SoS may be an enterprise wide business SoS, with the constituents being primarily COTS products along with some legacy applications. Or it may be Department of Defense (DoD) warfighting SoS, where the constituent legacy systems are integrated to increase capabilities on the battle field. Traditional systems engineering (SE) activities are typically tailored for the SoS case to define an SoS architecture, better coordinate the activities of multiple systems in migrating to the SoS architecture, and provide synchronization points for the SoS. In (OUSD AT&L, 2008), pilot studies have shown that many DoD SoS have re-organized the traditional SE activities into a set of seven core elements: 1) translating capability objectives, 2) understanding systems and relationships, 3) assessing performance to capability objectives, 4) developing, evolving, and maintaining SoS design, 5) monitoring and assessing changes, 6) addressing new requirements and options, and 7) orchestrating updates to SoS. Further analysis shows, that these elements map fairly well to the hybrid agile/plan-driven case (Case 8) at the SoS level. What makes this case different from Case 8, is that each of the constituent systems is using their own processes, which could be any of the above cases, depending on the scope, complexity, and characteristics of the constituent system. What is key is that the Case 9 extends Case 8 to include information or participation of the constituent systems in the agile planning activities and lets the “battle rhythm” of the constituent system increments guide the SoS plan-driven and V&V activities.

100. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE100 Case 10: Family of Systems Biggest risks: all those of Case 8 plus –Need to synchronize, integrate separately-managed, independently- evolving systems –Extremely large-scale; deep supplier hierarchies –Rapid adaptation to change extremely difficult Example: Medical device product line Size/complexity: Medium to very high Anticipated change rate (% per month): 1-3% Criticality: Medium to very high NDI support: Some in place Organizational and personnel capability: Related experience, medium to high capability Key Stage I activities: Full ICM; full stakeholder participation in product line scoping; strong business case Key Stage II activities: Full ICM; extra resources for first system, version control, multi-stakeholder support Time/build: 1-2 monthsTime/increment: 9-18 months

101. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE101 Case 10 Elaboration Families of systems are typically a set of systems that belong to a product line and can be easily used to customize a solution for a given need. This might be a suite of medical devices or a suite of applications to customer support. This is often the set of systems developed by a vendor that become the NDI components for CASE 7 above. Again, the rigor required for the SoS case is present here. However, in this situation, the family of systems is typically owned and evolved by a single organization/vendor and presents a case where the owning organization has much more control over the evolution of the components of the family of systems, thus possibly reducing some risks and allowing the ICM process to be a little more streamlined.

102. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE102 Case 11: Brownfield Example: Incremental legacy phaseout Size/complexity: High to very high Anticipated change rate (% per month): 0.3-3 Criticality: Medium-high NDI support: NDI as legacy replacement Organizational and personnel capability: Legacy re-engineering Key Stage I activities: Re-engineer/refactor legacy into services Key Stage II activities: Incremental legacy phaseout Time/build: 2-6 week/refactor Time/increment: 2-6 months

103. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE103 Case 11 Elaboration A Brownfield counterexample involved a major US corporation that used a Greenfield systems engineering and development approach to develop a new Central Corporate Financial System to replace a patched-together collection of strongly-coupled and poorly documented COBOL business data processing programs. The system included an early Enterprise Resource Planning (ERP) system that was well matched to the corporation's overall financial needs, but not to its detailed business processes, which included various workarounds to compensate for difficulties with the legacy software. The new system was well organized to support incremental implementation, but was dropped at a cost of $40 million after two failed tries to provide continuity of service, due primarily to the infeasibility of incrementally phasing out the legacy software and business processes compatibly with the new system's incremental capabilities. The ICM can be used to organize systems engineering and acquisition processes in ways that better support Brownfield legacy software re-engineering so that organizations can better provide continuity of services. In particular, its concurrent activities of Understanding Needs, Envisioning Opportunities, System Scoping and Architecting, Feasibility Evidence Development, and Risk/Opportunity Assessment enable projects to focus specifically on their legacy system constraints and on opportunities to deal with them. The application of the ICM to the Brownfield corporate counterexample situation would have avoided the failures of the Greenfield development. Its Understanding Needs activity would have determined the ways in which the legacy system had intertwined financial and other business services. For examples, Project Services included budgeting and scheduling, work breakdown system accounting, earned value management intertwined with requirements, version and configuration management; Contract Services included expenditure category management, billing, and receivables management intertwined with deliverables management and engineering change proposal tracking; with similar intertwining in Personnel Services and Marketing Services. The ICM's Envisioning Opportunities activity would have identified opportunities to use large-scale refactoring methods to decouple the financial and non-financial elements of these services in ways that would make it feasible for them to interoperate with a Financial Services element. Its System Scoping and Architecting activity would have repackaged the legacy software and developed the new architecture around financial services that would support incremental phaseout, and its Feasibility Evidence Development activity would have assessed any outstanding risks and covered them with risk management plans that would be tracked to ensure project success. A good example of a Brownfield methodology with several case study examples is provided in (Hopkins and Jenkins, 2008).

104. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE104 Case 12a: Net-Centric Services – Community Support Example: Community services or special interest group Size/complexity: Low to medium Anticipated change rate (% per month): 0.3-3 Criticality: Low to medium NDI support: Tailorable service elements Organizational and personnel capability: NDI-experienced Key Stage I activities: Filter, select, compose, tailor NDI Key Stage II activities: Evolve tailoring to meet community needs Time/build: <= 1 day Time/increment: 2-12 months

105. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE105 Case 12a Elaboration Net-Centric Services for community support tend to fall into two categories. One involves community service organizations providing needed services for child care, elder care, handicapped, homeless, or jobless people. Their information processing service needs include tracking of clients, volunteers, donors, donations, events, and provision of news and communication services. These used to require considerable programming to realize, but now can be realized by combining and tailoring NDI packages offering such services. The other category of net-centric services for community support involves special interest groups (professionals, hobbyists, committees, ethnic groups) with similar needs for tracking members, interests, subgroups, events, and provision of news and communication services. Development of such capabilities involves much more prototyping and much less documentation than is needed for more programming- oriented applications; for example, the internals of the NDI packages are generally unavailable for architectural documentation, and the resulting system capabilities are more driven by NDI capabilities than by prespecified requirements documents. A good example of how the ICM provides support for such net-centric services is provided in (Boehm and Bhuta, 2008) and the entire journal provides additional approaches and case studies of net-centric services development.

106. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE106 Case 12b: Net-Centric Services – Quick Response Decision Support Example: Response to competitor initiative Size/complexity: Medium to high Anticipated change rate (% per month): 3-30 Criticality: Medium to high NDI support: Tailorable service elements Organizational and personnel capability: NDI-experienced Key Stage I activities: Filter, select, compose, tailor NDI Key Stage II activities: Satisfy quick response; evolve or phase out Time/build: <= 1 day Time/increment: Quick response-driven

107. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE107 Case 12b Elaboration Another form of net-centric services involves rapidly configuring a capability to analyze alternative decision options in response to a competitor's initiative. These might involve a competitor beginning to penetrate a business's home territory, and might involve the need to evaluate the relative costs and benefits of alternative strategies of expanding services, expanding service locations, or repricing products or services. NDI packages to help analyze geographical, logistical, human resources, and financial implications of alternative competitive response decisions can be rapidly configured to provide a quick-response capability for converging on a decision. Once the decision is made, the resulting capability may be phased out, or evolved into a more general and maintainable capability for similar future situations. Again, the November/December 2008 special issue of IEEE Software on "Opportunistic Software Systems Development" provides additional perspectives on this area of net-centric services development.

108. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE108 Frequently Asked Question Q: Having all that ICM generality and then using the decision table to come back to a simple model seems like an overkill. –If my risk patterns are stable, can’t I just use the special case indicated by the decision table? A: Yes, you can and should – as long as your risk patterns stay stable. But as you encounter change, the ICM helps you adapt to it. –And it helps you collaborate with other organizations that may use different special cases.

109. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE109 Tutorial Outline Challenges for developing next-generation software systems Overview of ICM Risk-based balance of agility and assurance ICM process decision table Guidance and examples for using the ICM ICM tailoring exercises

110. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE110 ICM Tailoring Exercise Outline 1.Break into 5 groups: one to analyze each of the following projects Symbiq medical infusion pump SupplyChain.com application Event Managers application National Information System for Crisis Management (NISCM) application 2.Analyze project and determine key risks/issues 3.Based on key risks/issues, identify candidate ICM special cases for development 4.Select special case and provide Rationale for final selection Planned risk mitigation activities Probable decision at each anchor point milestone

111. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE111©USC-CSSE References - I Beck, K., Extreme Programming Explained, Addison Wesley, 1999. Boehm, B., “Some Future Trends and Implications for Systems and Software Engineering Processes”, Systems Engineering 9(1), pp. 1-19, 2006. Boehm, B., Brown, W., Basili, V., and Turner, R., “Spiral Acquisition of Software-Intensive Systems of Systems, CrossTalk, Vol. 17, No. 5, pp. 4-9, 2004. Boehm, B. and Lane J., "21st Century Processes for Acquiring 21st Century Software-Intensive Systems of Systems." CrossTalk: Vol. 19, No. 5, pp.4-9, 2006. Boehm, B., and Lane, J., “Using the ICM to Integrate System Acquisition, Systems Engineering, and Software Engineering,” CrossTalk, October 2007, pp. 4-9. Boehm, B., and Lane, J., “A Process Decision Table for Integrated Systems and Software Engineering,” Proceedings, CSER 2008, April 2008. Boehm, B., “Future Challenges and Rewards for Software Engineers,” DoD Software Tech News, October 2007, pp. 6-12. Boehm, B. et al., Software Cost Estimation with COCOMO II, Prentice Hall, 2000. Boehm, B., Software Engineering Economics, Prentice Hall, 2000. Carlock, P. and Fenton, R., "System of Systems (SoS) Enterprise Systems for Information-Intensive Organizations," Systems Engineering, Vol. 4, No. 4, pp. 242-26, 2001. Carlock, P., and J. Lane, “System of Systems Enterprise Systems Engineering, the Enterprise Architecture Management Framework, and System of Systems Cost Estimation”, 21st International Forum on COCOMO and Systems/Software Cost Modeling, 2006. Checkland, P., Systems Thinking, Systems Practice, Wiley, 1980 (2nd ed., 1999). Department of Defense (DoD), Defense Acquisition Guidebook, version 1.6, http://akss.dau.mil/dag/, 2006.http://akss.dau.mil/dag/ Department of Defense (DoD), Instruction 5000.2, Operation of the Defense Acquisition System, May 2003. Department of Defense (DoD), Systems Engineering Plan Preparation Guide, USD(AT&L), 2004. Electronic Industries Alliance (1999); EIA Standard 632: Processes for Engineering a System Galorath, D., and Evans, M., Software Sizing, Estimation, and Risk Management, Auerbach, 2006. Hall, E.T., Beyond Culture, Anchor Books/Doubleday, 1976.

112. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE112©USC-CSSE References -II Highsmith, J., Adaptive Software Development, Dorset House, 2000. International Standards Organization, Information Technology Software Life Cycle Processes, ISO/IEC 12207, 1995 ISO, Systems Engineering – System Life Cycle Processes, ISO/IEC 15288, 2002. Jensen, R. “An Improved Macrolevel Software Development Resource Estimation Model,” Proceedings, ISPA 5, April 1983, pp. 88-92. Krygiel, A., Behind the Wizard’s Curtain; CCRP Publication Series, July, 1999, p. 33 Lane, J. and Boehm, B., "System of Systems Cost Estimation: Analysis of Lead System Integrator Engineering Activities", Information Resources Management Journal, Vol. 20, No. 2, pp. 23-32, 2007. Lane, J. and Valerdi, R., “Synthesizing SoS Concepts for Use in Cost Estimation”, Proceedings of IEEE Systems, Man, and Cybernetics Conference, 2005. Lientz, B., and Swanson, E.B., Software Maintenance Management, Addison Wesley, 1980. Madachy, R., Boehm, B., Lane, J., "Assessing Hybrid Incremental Processes for SISOS Development", USC CSSE Technical Report USC-CSSE-2006-623, 2006. Maier, M., “Architecting Principles for Systems-of-Systems”; Systems Engineering, Vol. 1, No. 4 (pp 267-284). Maier, M., “System and Software Architecture Reconciliation,” Systems Engineering 9 (2), 2006, pp. 146-159. Northrop, L., et al., Ultra-Large-Scale Systems: The Software Challenge of the Future, Software Engineering Institute, 2006. Pew, R. W., and Mavor, A. S., Human-System Integration in the System Development Process: A New Look, National Academy Press, 2007. Putnam, L., “A General Empirical Solution to the Macro Software Sizing and Estimating Problem,” IEEE Trans SW Engr., July 1978, pp. 345-361. Rechtin, E. Systems Architecting, Prentice Hall, 1991. Schroeder, T., “Integrating Systems and Software Engineering: Observations in Practice,” OSD/USC Integrating Systems and Software Engineering Workshop, http://csse.usc.edu/events/2007/CIIForum/pages/program.html, October 2007. http://csse.usc.edu/events/2007/CIIForum/pages/program.html

113. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE113©USC-CSSE List of Acronyms B/LBaselined C4ISRCommand, Control, Computing, Communications, Intelligence, Surveillance, Reconnaissance CDConcept Development CDRCritical Design Review COTSCommercial Off-the-Shelf DCRDevelopment Commitment Review DIDevelopment Increment DoDDepartment of Defense ECRExploration Commitment Review EVMSEarned Value Management System FCRFoundations Commitment Review FEDFeasibility Evidence Description FMEAFailure Modes and Effects Analysis FRPFull-Rate Production GAOGovernment Accountability Office GUIGraphical User Interface

114. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE114©USC-CSSE List of Acronyms (continued) HMIHuman-Machine Interface HSIHuman-System Interface HWHardware ICMIncremental Commitment Model IOCInitial Operational Capability IRRInception Readiness Review IS&SEIntegrating Systems and Software Engineering LCOLife Cycle Objectives LRIPLow-Rate Initial Production MBASEModel-Based Architecting and Software Engineering NDINon-Developmental Item NRCNational Research Council OCOperational Capability OCROperations Commitment Review OO&DObserve, Orient and Decide OODAObserve, Orient, Decide, Act O&MOperations and Maintenance

115. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE115©USC-CSSE List of Acronyms (continued) PDRPreliminary Design Review PMProgram Manager PRPublic Relations PRRProduct Release Review RUPRational Unified Process SoSSystem of Systems SoSESystem of Systems Engineering SSESystems and Software Engineering SWSoftware SwESoftware Engineering SysESystems Engineering Sys EngrSystems Engineer S&SESystems and Software Engineering USD (AT&L)Under Secretary of Defense for Acquisition, Technology, and Logistics VCRValidation Commitment Review V&VVerification and Validation WBSWork Breakdown Structure WMIWarfighter-Machine Interface