USC CSSE Workshop Overview: Top 3 Software-Intensive Systems Risk Items Barry Boehm, USC-CSSE February 14,

2 Outline: Top-3 SIS Risks Workshop  Working group guidelines  Risk survey results and survey update(?)  The top three risks –Architecture complexity; system quality tradeoffs –Requirements volatility; rapid change –Acquisition and contracting process mismatches  Architecture complexity and system quality tradeoffs –Architecture complexity phenomenology –Nature of system quality –Quality tradeoff perspectives

3 Working Group Guidelines  Product: briefing, preferably with notes  Topics should include: –Most critical success factors in each area –Current best practices for addressing them –Areas for further research  Rated 0-10 on value and difficulty of research

4 Research Topics: Agile Methods 1.Relationship between plan driven and agility a.For individuals b.For organizations 2.Differences between agile and plan driven outcomes 3.Effect of Gurus 4.Mismatches between development approach and acquisition practices 5.How do you measure quality in an agile environment? 6.Data collection; agile experience base 7.Team of teams 8.Agile Development and Evolutionary Prototyping 9.Shared Code and/or module ownership 10.Architecture: when, how much, how to express 11.Lack of user consensus 12.Dynamic Homegrounds

5 SIS Risk Survey 2006: Statistics  Number of Surveys: 25  Average Experience: ~28 years (6 years – 51 years)  Area Distribution: –Software: 20 –Systems: 17 –Hardware: 0  Business Domain Distribution: –Aerospace: 18 –Software Infrastructure: 5 –Business: 4 –Telecom: 3 –Others: Secure Apps (1); Safety Critical Apps (1); C4ISR (1)

6 Risk Survey 2006: Nominees  Acquisition and contracting process mismatches  Architecture complexity; quality tradeoffs  Budget and schedule constraints  COTS and other independently evolving systems  Customer-developer-user team cohesion  Migration complexity  Personnel shortfalls  Process maturity  Requirements mismatch  Requirements volatility; rapid change  Technology maturity  User interface mismatch

7 Risk Survey 2006 Results

8 SIS Risk Grouping #Risk ItemΣRanks 1Architecture complexity, quality tradeoffs142 2Requirements volatility Acquisition and contracting process mismatches Customer-developer-user Budget and schedule Requirements mismatch Personnel shortfalls99 8COTS77 9Migration complexity User interface mismatch Technology maturity Process maturity46.83

9 Survey 2007: Early Statistics  Number or Surveys: 41  Average Experience: ~27 years (6 years – 51 years)  Area Distribution: –Software: 33 –Systems: 34 –Hardware: 0  Business Domain Distribution: –Aerospace: 32 –Software Infrastructure: 7 –Business: 6 –Telecom: 5 –Others: Secure Apps (1); Safety Critical Apps (1); C4ISR (1); Network and Protocols (1); Defense (1); Program and Risk Management (1)

10 Survey Results:

11 SIS Risk Grouping #Risk Item Previou s Rank ΣRanks 1.Architecture complexity, quality tradeoffs↔ Requirements volatility↔ Acquisition and contracting process mismatches ↔ Budget and schedule↑ Customer-developer-user↓ Requirements mismatch↔ Personnel shortfalls↔ COTS↔ Technology maturity↑ Migration complexity↓ User interface mismatch↓ Process maturity↔

12 Outline: Top-3 SIS Risks Workshop  Working group guidelines  Risk survey results and survey update(?)  The top three risks –Architecture complexity; system quality tradeoffs –Requirements volatility; rapid change –Acquisition and contracting process mismatches  Architecture complexity and system quality tradeoffs –Architecture complexity phenomenology –Nature of system quality –Quality tradeoff perspectives

13 SIS Architecture Complexity: Future Combat Systems

14 Requirements Volatility: Ripple Effects of Changes - Breadth, Depth, and Length Platform N Platform 1 Infra C4ISR Command and Control Situation Assessment Info Fusion Sensor Data Management Sensor Data Integration Sensors Sensor Components : … Breadth Length Depth DOTMLPF Legend: DOTMLPF Doctrine, Organization, Training, Materiel, Leadership, Personnel, Facilities C4ISR Command, Control, Communications, Computers, Intelligence, Surveillance, and Reconnaissance

15 Average Change Processing Time: 2 Systems of Systems  Average workdays to process changes

16 Acquisition/Contracting Mismatches: Fitness Landscapes  Role of Fitness Landscapes in Complex Adaptive Systems (CAS) –S. Kauffman, At Home in the Universe, Oxford University Press, 1995  CSoS Acquisition Challenges –B. Boehm, “Some Future Trends and Implications for Systems and Software Engineering Processes”, Systems Engineering 9(1), 2006, pp  A Candidate Three-Agent Acquisition Fitness Landscape –D. Reifer and B. Boehm, “Providing Incentives for Spiral Development: An Award Fee Plan”, Defense Acquisition Review 13(1), 2006, pp

17 Role of Fitness Landscapes in CAS  Incentive structures for local behavior  Induce global behavior via adaptation to change Fitness Landscape UniformRandom Survival- Related Global Result GridlockChaosEdge of Chaos Acronym (Metaphor) OWHITS (Ostriches with Heads in the Sand) TRAW (Turkeys Running Around Wild) NOSUFAS (No One-Size- Uniformly-Fits-All Solutions Acquisition Example MIL-STD-1521B Waterfall, Fixed Price, Build-to-Spec Recursive Acquisition Reform, Total Systems Performance Responsibility Candidate for Discussion: 3-Agent Model

18 Complex Systems Acquisition Challenges Objective Candidate Solution ExampleChallenges Avoid Obsolescence Plan-Driven Rapid Development 4-Hour HouseInflexible Adapt to Rapid Change Agile Methods Extreme Programming Unscalable; Buggy Releases Assure Resilience Independent V&VFormal MethodsExpensive

19 Candidate Approach: 3-Agent Model Agent ObjectiveAgent Approach Fitness Landscape/ Incentive Criteria* Build Current Increment Rapid, Stable, Schedule-As- Independent Variable (SAIV), Build to Specs and Plans Meet Milestones/Exercise SAIV; Deliver on Time; Collaboration with Other Agents Assure Resilience Integrated, Independent Verification and Validation Priority-Weighted Identification of Risks and Concerns; Collaboration with Other Agents Prepare for Build of Next Increment Observe, Orient, Decide on Proof-Carrying Rebaselined Specs and Plans Risk/Opportunity Management; Rebasline Proof Thoroughness; Collaboration with Other Agents

20 Risk-Driven Scalable Spiral Model: Increment View

21 Outline: Top-3 SIS Risks Workshop  Working group guidelines  Risk survey results and survey update(?)  The top three risks –Architecture complexity; system quality tradeoffs –Requirements volatility; rapid change –Acquisition and contracting process mismatches  Architecture complexity and system quality tradeoffs –Architecture complexity phenomenology –Nature of system quality –Quality tradeoff perspectives

22 Larger Systems Need More Architecting: COCOMO II Analysis Percent of Project Schedule Devoted to Initial Architecture and Risk Resolution Added Schedule Devoted to Rework (COCOMO II RESL factor) Total % Added Schedule KSLOC 100 KSLOC 10 KSLOC Sweet Spot Sweet Spot Drivers: Rapid Change: leftward High Assurance: rightward

23 Architecture-Breakers are the Biggest Source of Rework % of Software Problem Reports (SPR’s) TRW Project A 373 SPR’s TRW Project B 1005 SPR’s % of Cost to Fix SPR’s Major Rework Sources: Off-Nominal Architecture-Breakers A - Network Failover B - Extra-Long Messages

24 Best Architecture is a Discontinuous Function of Quality Level $100M $50M Arch. A: Custom many cache processors Arch. B: Modified Client-Server Response Time (sec) Original Spec After Prototyping

25 The Nature of Quality: Participant Survey Which figure best symbolizes quality improvement?

Holistic Approach

Lean Approach

Analytic Approach

Preoccupation with Booze and Sex

30 There is No Universal Quality-Value Metric  Different stakeholders rely on different value attributes –Protection: safety, security, privacy –Robustness: reliability, availability, survivability –Quality of Service: performance, accuracy, ease of use –Adaptability: evolvability, interoperability –Affordability: cost, schedule, reusability  Value attributes continue to tier down –Performance: response time, resource consumption (CPU, memory, comm.)  Value attributes are scenario-dependent –5 seconds normal response time; 2 seconds in crisis  Value attributes often conflict –Most often with performance and affordability

31 Overview of Stakeholder/Value Dependencies Attributes Stakeholders ** * * * * * * ** * Protection Robustness Quality of Service Adaptability Affordability Developers, Acquirers Mission Controllers, Administrators Info. Consumers Info. Brokers Info. Suppliers, Dependents  Strength of direct dependency on value attribute **- Critical ; *-Significant; blank-insignificant or indirect

32 Implications for Quality Engineering  There is no universal quality metric to optimize  Need to identify system’s success-critical stakeholders –And their quality priorities  Need to balance satisfaction of stakeholder dependencies –Stakeholder win-win negotiation –Quality attribute tradeoff analysis  Need value-of-quality models, methods, and tools

33 Tradeoffs Among Cost, Schedule, and Reliability: COCOMO II Want 10K hour MTBF within $5.5M, 20 months -- Cost/Schedule/RELY: “pick any two” points (RELY, MTBF (hours)) For 100-KSLOC set of features Can “pick all three” with 77-KSLOC set of features

34 Agenda : Wednesday, Feb 14  8:15 – 10:00 am: Architecture Complexity and Quality Tradeoffs; Elliot Axelband (RAND), Chair –Overview, Issues and Approaches; Barry Boehm (USC) –From Dependable Architectures To Dependable Systems; Nenad Medvidovic (USC) –Architecture Tradeoff Analysis: Towards a Disciplined Approach to Balancing Quality Requirements; Azad Madni (Intelligent Systems Technology)  10:00 – 10:30 am: Break  10:30 am – 12:30 pm: Requirements Volatility; George Friedman (USC), Chair –Process Synchronization and Stabilization; Rick Selby, Northrop Grumman –Disciplined Agility; Rich Turner (SSCI) –Using Anchor Point Milestones; Tom Schroeder, BAE Systems  12:30 – 1:30 pm: Lunch  1:30 – 3:30 pm Acquisition and Contracting Mismatches; Rick Selby (NGC), Chair –Acquisition Assessment Analyses; Kristen Baldwin (OSD/AT&T/S&SE) –Commercial Acquisition Practices; Stan Rifkin (Master Systems Inc.) –Space Program Acquisition: Systems Engineering & Programmatic Improvements; Marilee Wheaton (Aerospace Corporation)  3:30 – 4:00 pm: Break  4:00 – 5:00 pm: General Discussion: Working Group Formation; Barry Boehm, Chair