A study of the Causes of Requirements Volatility and its Impact on Systems Engineering Effort COSYSMO Workshop Center for Software and Systems Engineering, Annual Research Review March 11, 2010 Mauricio E. Peña – Ph.D. Student, USC Industrial and Systems Engineering Department

2 Motivation “Requirements are the foundation of the project. They form the basis for design, manufacture, test and operations….changes in requirements later in the development cycle can have a significant cost impact, possibly resulting in cancellation.” INCOSE Systems Engineering Handbook (2006)

3 Objectives Provide and overview of requirements volatility and its importance to engineering projects Introduce the research questions & research plan Discuss the state of the art and implications to COSYSMO Solicit feedback: –On a requirements volatility causal model –On a requirements volatility survey

4 Requirements Volatility Overview Requirements volatility is the change in requirements (added, deleted, and modified) over a given time interval [MIL-STD-498, 1994, Ferreira, 2002] Also known as: Requirements creep: An increase in scope and number of system requirements Requirements churn: Instability in the requirements set – requirements are modified or re-worked without necessarily resulting in an increase in the total number of requirements CDR [Hammer et al, 1998]

5 Importance of Understanding Requirements Volatility Requirements volatility has been identified by numerous research studies as a risk factor and cost-driver of systems engineering projects [Ferreira 2002, Boehm 1991] Requirements changes are costly, particularly in the later stages of the lifecycle process because the change may require rework of the design, verification and deployment plans [Kotonya and Sommerville, 1995] Requirements volatility trends are considered leading indicators of project performance as defined by the Lean Advancement Initiative Systems Engineering Leading Indicators Guide (2007) The Government Accountability Office (GAO) concluded in a 2004 report on the DoD’s acquisition of software-intensive weapons systems that missing, vague, or changing requirements are a major cause of project failure

6 Research Questions Research Question 1: What causes requirements volatility? Research Question 2: How does requirements volatility impact systems engineering effort?

7 Research Plan Literature Review Statistical Analysis Observations Survey Results Considerations for requirements volatility Causal Model Pilot RV survey Delphi Survey Industry Data Tested Hypothesis Updated Causal Model (hypothesis) RV survey COSYSMO Workshop Discussion LAI Knowledge Exchange Delphi Survey CompletedIn-progressPlanned

8 Literature Background Most of the requirements volatility research to date has been focused on software systems Various research methods have been utilized to investigate the causes and effects of requirement volatility – a methodological breakdown of the studies reviewed to date is below However, there still a lack of empirical data to determine the quantitative impact of requirements volatility on systems engineering effort for a broader base of engineering projects

9 Implications to COSYSMO COSYSMO Size Drivers Effort Multipliers Effort Calibration # Requirements # Interfaces # Scenarios # Algorithms - Application factors -8 factors - Team factors -6 factors Reuse Categories Volatility Factor [Fortune, 2009] Volatility was identified as a size driver adjustment factor in COSYSMO [Valerdi, 2005] It is currently a placeholder in the model

10 Impact of volatility after the requirements have been baselined SRR Delta SE effort SRR Planned # Req Actual # Req Planned # Req Actual # Req PROJECT WITH VOLATILITY PROJECT WITH LOW VOLATILITY # Req. SE Hours

11 Observations 1.Requirements volatility is correlated with an increase in the size drivers of systems engineering effort [Valerdi, 2005; Houston,2000] 2.Requirements Volatility is expected during the early stages of a project (conceptualize / requirements phase). It becomes a concern when it ocurrs after the requirements phase is complete because it is likely to result in re-work of engineering products [Kotonya and Sommerville (1995); Rosenberg and Hyatt (1996); Ferreira, (2002)] 3.Requirements volatility is correlated to increases in cost, schedule duration and re- work [Ferreira et al, 2009; Zowghi and Nurmuliani, 2002] 4.Requirements volatility may have a compounding effect on systems engineering effort because the changes may not only cause rework but also trigger increases in the number of lower-level requirements [Kulk and Verhoef, 2008] 5.The use of volatility metrics and thresholds can help determine the adequacy of SE processes and staff and trigger corrective actions to mitigate further volatility (Roedler and Rhodes, 2007) 6.Requirements volatility is caused by a number of factors that are both external (e.g. environmental, customer priority changes) and internal (process capability, organizational policies) Based on the review of the literature a causal model was developed to investigate both research questions

12 Changes in org. structure and process Requirements volatility ++ +/- + Re-work +/- - SE Productivity Number of Sys Requirements Project Schedule SE Project Effort SE CostProject Cost Quality Customer Satisfaction Customer-Driven Changes SE Process Maturity Technology Maturity Requirements. Re-use - Experienced staff - Dependent Variable Independent Variable Contextual / Environmental changes +/- (Valerdi 2005) (Ferreira, et al 2009) (Valerdi 2005) (Houston, 2000) (Ferreira 2002) (Charette et al,2003) (GAO, 2004) (Ferreira 2002) (Charette et al,2003) (Houston, 2000) Zowghi et al. (2000) (Ferreira, et al 2009) - (Houston, 2000) (Ferreira 2002) (Charette et al,2003) (Ferreira, 2002) (GAO, 2004) + (Jones 1994) (Curtis et al. 1988) (Kotonya and Sommerville 1998) (Jones 1994) (Curtis et al. 1988) Zowghi et al. (2000) (Ferreira, 2002) (Ferreira, et al 2009) Causal Model (normative) Case 1 research question: what causes requirements volatility? (Ferreira 2002) (Kotonya and Sommerville 1998) +/- (Valerdi 2005) +/-

13 Causal Model (normative) Case 2 research question: how does requirements volatility impact systems engineering effort? Changes in org. structure and process Requirements volatility ++ +/- + Re-work +/- - SE Productivity Number of Sys Requirements Project Schedule SE Project Effort SE CostProject Cost Quality Customer Satisfaction Customer-Driven Changes SE Process Maturity Technology Maturity Requirements. Re-use - Experienced staff - Dependent Variable Independent Variable Contextual / Environmental changes +/- (Valerdi 2005) (Ferreira, et al 2009) (Valerdi 2005) (Houston, 2000) (Ferreira 2002) (Charette et al,2003) (GAO, 2004) (Ferreira 2002) (Charette et al,2003) (Houston, 2000) Zowghi et al. (2000) (Ferreira, et al 2009) - (Houston, 2000) (Ferreira 2002) (Charette et al,2003) (Ferreira, 2002) (GAO, 2004) + (Jones 1994) (Curtis et al. 1988) (Kotonya and Sommerville 1998) (Jones 1994) (Curtis et al. 1988) Zowghi et al. (2000) (Ferreira, 2002) (Ferreira, et al 2009) (Ferreira 2002) (Kotonya and Sommerville 1998) +/- (Valerdi 2005) +/-

14 Requirements volatility +/- Number of Requirements Project Schedule SE Project Effort SE Cost SE Process Maturity - Experienced staff Dependent Variable Independent Variable +/- (Valerdi 2005) (Ferreira, et al 2009) (Houston, 2000) Zowghi et al. (2000) (Ferreira, et al 2009) - (Ferreira, 2002) (GAO, 2004) Moderating Variable (Ferreira 2002) Moderating impact of expected volatility / thresholds Volatility Metrics / Thresholds Org Improvement Actions ++ (Kulk and Verhoef 2008 Zowghi and Nurmuliani (1998) +/- (Kulk and Verhoef 2008) +/-

15 Questions for discussion 1.Are there other important causes of volatility missing in the model? 2.In what cases is the relationship between requirements volatility and # of systems requirements a positive one, and in what cases is it a negative one? 3.Should the impact of requirements volatility on the # of system requirements be adjusted based on the criticality/coupling of the requirements that are added/modified/deleted? 4.Does volatility have a direct impact on productivity? 5.Should volatility thresholds vary depending on the size and duration of a project? 6.How does the lifecycle phase affect the expected level of volatility? 7.Feedback on the survey -

16 Next Steps Utilize recommendations/suggestions from this workshop to update the causal model and survey Administer the survey to LAI workshop participants Need to develop an approach to generate cost driver weight factors based on requirements volatility Start the industry outreach to reach agreement on data-sharing

17 References Boehm, B. (1991). Software Risk Management: Principles and Practices. IEEE Software 8, 1. Pp Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009). “Understanding the effects of requirements volatility in software engineering by using analytical modeling and software process simulation.” The Journal of Systems and Software. Vol. 82, pp Fortune, J. (2009). Estimating systems engineering reuse with the constructive systems engineering cost model (COSYSMO 2.0). Doctoral Dissertation. University of Southern California, Industrial and Systems Engineering Department. GAO (2004). Report to the Committee on Armed Services, U.S. Senate. Defense Acquisitions. Stronger Management Practices Are Needed to Improve DOD’s Software- Intensive Weapon Acquisitions Hammer, T., Huffman, L., and Rosenberg, L. (1998). “Doing requirements right the first time.” Crosstalk, the Journal of Defense Software Engineering. Pp Houston, Dan X. (2000). A Software Project Simulation Model for Risk Management, Ph.D. Dissertation, Arizona State University INCOSE Systems Engineering Handbook, Version 3, INCOSE, June 2006 ISO/IEC (2008). ISO/IEC 15288:2008 (E) Systems Engineering - System Life Cycle Processes. Kotonya, G., Sommerville, I., (1998). Requirements Engineering: Processes and Techniques. John Wiley and Sons, Ltd. Kulk, G, and Verhoef, C. (2008). “Quantifying requirements volatility effects”. Science of Computer Programming. Vol 72. pp 136–175 MIL-STD Software Development and Documentation. U.S. Department of Defense. Roedler, G. and Rhodes, D. (2007). Systems engineering leading indicators guide. Version 1. Massachusetts Institute of Technology, INCOSE, and PSM. Rosenberg, L. and Hyatt L., (1996). A Software Quality Model and Metrics for Identifying Project Risks and Assessing Software Quality. (retrieved from Schunn, C. (2008) Engineering Educational Design. Educational Designer, 1(1) Valerdi, R. (2005). The constructive systems engineering cost model (COSYSMO). Doctoral Dissertation. University of Southern California, Industrial and Systems Engineering Department. Zowghi, D. and Nurmuliani, N. (2002). A Study of the Impact of Requirements Volatility on Software Project Performance. Proceedings of the Ninth Asia-Pacific Software Engineering Conference