1. SQM - 1DCS - ANULECTURE 11-2005 Software Quality Management Software Quality Management Processes V & V of Critical Software & Systems Ian Hirst

2. SQM - 2DCS - ANULECTURE 11-2005 Agenda review review risk based V & V risk based V & V software & systems criticality software & systems criticality criticality analysis & risk assessment (CARA) method criticality analysis & risk assessment (CARA) method impact & risk driver categories & values impact & risk driver categories & values CARA steps CARA steps CARA implementation recommendations CARA implementation recommendations CARA benefits CARA benefits

3. SQM - 3DCS - ANULECTURE 11-2005 Review many software projects fail to meet objectives many software projects fail to meet objectives lack of objective quality evidence is common lack of objective quality evidence is common a complex solution requires significant test planning and management effort and a complex set of testing activities a complex solution requires significant test planning and management effort and a complex set of testing activities complete testing is usually not possible complete testing is usually not possible testing should be focused on assisting with evaluation of the success of the project and the ‘quality’ of the delivered solution…. testing should be focused on assisting with evaluation of the success of the project and the ‘quality’ of the delivered solution….

4. SQM - 4DCS - ANULECTURE 11-2005 Risk based V & V Risk = probability of occurrence x impact Risk = probability of occurrence x impact V & V is primarily a risk management activity V & V is primarily a risk management activity risks can be associated with both product & processes risks can be associated with both product & processes high impact elements are critical elements high impact elements are critical elements V & V is aimed at reducing or eliminating uncertainty by the provision of evidence of the capability & quality of software & systems V & V is aimed at reducing or eliminating uncertainty by the provision of evidence of the capability & quality of software & systems V & V on all software is not necessary or financially feasible V & V on all software is not necessary or financially feasible risk based V & V is a targeted activity…. risk based V & V is a targeted activity….

5. SQM - 5DCS - ANULECTURE 11-2005 Software & systems criticality Criticality is a measure of the impact of errors on: Criticality is a measure of the impact of errors on: system performance and operations system performance and operations safety / security safety / security cost & schedule cost & schedule Risk is a measure of the likelihood of errors based on: Risk is a measure of the likelihood of errors based on: complexity complexity maturity of technology maturity of technology requirements definition & stability requirements definition & stability testability testability developer experience…. developer experience….

6. SQM - 6DCS - ANULECTURE 11-2005 The Criticality Analysis & Risk Assessment (CARA) method CARA is a formalised methodology which evolved from a US Air Force V & V initiative. CARA is a formalised methodology which evolved from a US Air Force V & V initiative. CARA provides a systematic procedure for rank ordering development program elements with respect to well defined scoring factors associated with criticality & risks drivers CARA provides a systematic procedure for rank ordering development program elements with respect to well defined scoring factors associated with criticality & risks drivers CARA is a means for evaluating the risk exposure for software or systems and sizing the V & V effort CARA is a means for evaluating the risk exposure for software or systems and sizing the V & V effort CARA has been applied to space systems including : CARA has been applied to space systems including : space shuttle software and critical mission support software space shuttle software and critical mission support software space station flight software…. space station flight software….

7. SQM - 7DCS - ANULECTURE 11-2005 The criticality & risk driver categories & values error impact categories & values error impact categories & values Catastrophic (4) Catastrophic (4) Critical (3) Critical (3) Marginal (2) Marginal (2) Negligible (1) Negligible (1) risk driver categories & values risk driver categories & values Complexity (high=3, moderate=2, low =1) Complexity (high=3, moderate=2, low =1) Maturity of technology (high=3, moderate=2, low =1) Maturity of technology (high=3, moderate=2, low =1) Requirements definition & stability (high=3, moderate=2, low=1) Requirements definition & stability (high=3, moderate=2, low=1) Testability (high=3, moderate=2, low=1)…. Testability (high=3, moderate=2, low=1)…. refer tables 14.3 & 14.4

8. SQM - 8DCS - ANULECTURE 11-2005 The criticality categories & values

9. SQM - 9DCS - ANULECTURE 11-2005 The risk driver categories & values

10. SQM - 10DCS - ANULECTURE 11-2005 CARA steps Step 1: Identify software functions Step 1: Identify software functions Step 2: Establish the evaluation team Step 2: Establish the evaluation team Step 3: Develop the CARA evaluation criteria Step 3: Develop the CARA evaluation criteria Step 4: Perform criticality analysis & risk assessment Step 4: Perform criticality analysis & risk assessment Step 5: Set V&V analysis level (VAL) thresholds Step 5: Set V&V analysis level (VAL) thresholds Step 6: Estimate software size Step 6: Estimate software size Step 7: Generate V&V effort estimates Step 7: Generate V&V effort estimates Step 8: Evaluate effort estimate results Step 8: Evaluate effort estimate results Step 9: Revise V&V scope…. Step 9: Revise V&V scope….

11. SQM - 11DCS - ANULECTURE 11-2005 Step 1: Identify software functions Collect systems information (from CONOPS,specs, business cases) Collect systems information (from CONOPS,specs, business cases) Identify the required software capabilities (functions and performance) Identify the required software capabilities (functions and performance) Build a scoring matrix Build a scoring matrix A solution decomposition/ PBS based structure is useful A solution decomposition/ PBS based structure is useful Values may be assigned at any appropriate level of abstraction (requirements, requirements groups, components, sub systems, etc) Values may be assigned at any appropriate level of abstraction (requirements, requirements groups, components, sub systems, etc) Identify related systems domains / areas of specialisation…. Identify related systems domains / areas of specialisation….

12. SQM - 12DCS - ANULECTURE 11-2005 Step 2: Establish the evaluation team Engage system domain experts Engage system domain experts Engage development process experts Engage development process experts Establish management team & processes…. Establish management team & processes….

13. SQM - 13DCS - ANULECTURE 11-2005 Step 3: Develop the CARA evaluation criteria Collect evaluation criteria from similar domains Collect evaluation criteria from similar domains Develop an understanding of the mission the system is to perform Develop an understanding of the mission the system is to perform Tailor criticality evaluation criteria in terms of what is catastrophic, critical, or of moderate impact to users, customers, and acquirers of the system Tailor criticality evaluation criteria in terms of what is catastrophic, critical, or of moderate impact to users, customers, and acquirers of the system Tailor risk evaluation criteria ( by inclusion of additional drivers) Tailor risk evaluation criteria ( by inclusion of additional drivers) Identify criticality area or risk driver weightings, if necessary Identify criticality area or risk driver weightings, if necessary Review criteria with the customer…. Review criteria with the customer….

14. SQM - 14DCS - ANULECTURE 11-2005 Step 4: Perform criticality analysis & risk assessment Perform criticality analysis Perform criticality analysis consider the systems components, their interactions, failure modes & effects, and concepts of operations consider the systems components, their interactions, failure modes & effects, and concepts of operations rate the functions according to criteria and scoring rationale rate the functions according to criteria and scoring rationale Perform risk analysis Perform risk analysis review software & system development, test and verification plans review software & system development, test and verification plans review development methods, testing approach, reuse plans, organisational interfaces, integration requirements, risks & risk mitigation techniques review development methods, testing approach, reuse plans, organisational interfaces, integration requirements, risks & risk mitigation techniques rate the functions according to criteria and scoring rationale rate the functions according to criteria and scoring rationale Calculate CARA scores (n = Criticality[xW] x Risk[xW]) Calculate CARA scores (n = Criticality[xW] x Risk[xW]) Rank elements in score order…. Rank elements in score order….

15. SQM - 15DCS - ANULECTURE 11-2005 Step 5: Set V&V analysis level (VAL) thresholds Functions with higher CARA scores receive higher VALs Functions with higher CARA scores receive higher VALs Example VAL thresholds: Example VAL thresholds: CARA scoreVAL 1<CARA<2None 2<CARA<5Limited 5<CARA<8Focused 8<CARA<12Comprehensive e.g. 1: Safety impact of 4, complexity risk of 3, n = 12 (unweighted) e.g. 2: Cost impact of 1, maturity risk of 2, n = 2 (unweighted)….

16. SQM - 16DCS - ANULECTURE 11-2005 Step 6: Estimate software size This step can be performed anytime before step 7 This step can be performed anytime before step 7 The size measurement is the V & V workload: The size measurement is the V & V workload: V&V work = f (no. of requirements, external interfaces, output products) An alternative size measure is the developer software size estimate (e.g. SLOC) An alternative size measure is the developer software size estimate (e.g. SLOC)

17. SQM - 17DCS - ANULECTURE 11-2005 Step 7: Generate V & V effort estimates (including an independent V & V effort estimate) Apply V&V productivity factors to size estimates (these may vary according to VALs, software complexity & size, development methods, development types (initial production, block update), domains, developer maturity and experience, V&V agent experience) Apply V&V productivity factors to size estimates (these may vary according to VALs, software complexity & size, development methods, development types (initial production, block update), domains, developer maturity and experience, V&V agent experience) Apply program project management (schedule and effort estimation) standards & conventions …. Apply program project management (schedule and effort estimation) standards & conventions ….

18. SQM - 18DCS - ANULECTURE 11-2005 Step 8: Evaluate effort estimate results Review the results with the customer Review the results with the customer If the prescribed VALs for the software functions and associated costs are acceptable, generate the critical functions list (this defines the V&V scope and priorities) …. If the prescribed VALs for the software functions and associated costs are acceptable, generate the critical functions list (this defines the V&V scope and priorities) ….

19. SQM - 19DCS - ANULECTURE 11-2005 Step 9: Revise V&V scope If the results are not acceptable, use the independent V&V estimate to re-scope the effort If the results are not acceptable, use the independent V&V estimate to re-scope the effort VAL threshold adjustments may be used to aid breadth v. depth tradeoffs VAL threshold adjustments may be used to aid breadth v. depth tradeoffs VAL selective exceptions/adjustments may be used e.g. for safety critical functions with a score of 3 or more - apply focused V&V (instead of limited).... VAL selective exceptions/adjustments may be used e.g. for safety critical functions with a score of 3 or more - apply focused V&V (instead of limited)....

20. SQM - 20DCS - ANULECTURE 11-2005 CARA implementation recommendations CARA scoring must be done by domain experts CARA scoring must be done by domain experts CARA scoring must be done in a peer review environment CARA scoring must be done in a peer review environment Project management should participate in scoring activities Project management should participate in scoring activities CARA training should be done with all personnel CARA training should be done with all personnel Automation tools are necessary for large projects Automation tools are necessary for large projects Capturing the scoring rationale is important Capturing the scoring rationale is important CARA should be repeated at least once every major development milestone…. CARA should be repeated at least once every major development milestone….

21. SQM - 21DCS - ANULECTURE 11-2005 CARA benefits The results can be used to support V&V planning & management : The results can be used to support V&V planning & management : assessment of overall and relative risks assessment of overall and relative risks allocation of fixed resources across a set of V&V objectives and tasks allocation of fixed resources across a set of V&V objectives and tasks assessment of the need for future V&V resources assessment of the need for future V&V resources establishing V&V importance levels / focus points/ priorities establishing V&V importance levels / focus points/ priorities prioritisation of items for work sequencing prioritisation of items for work sequencing CARA establishes a structured approach to V&V which increased customer visibility into risk, risk mitigation and V&V activities…. CARA establishes a structured approach to V&V which increased customer visibility into risk, risk mitigation and V&V activities….

22. SQM - 22DCS - ANULECTURE 11-2005 Review V & V is primarily a risk management activity V & V is primarily a risk management activity V & V on all software is not necessary or financially feasible V & V on all software is not necessary or financially feasible Risk based V & V is a targeted activity Risk based V & V is a targeted activity The criticality analysis & risk assessment (CARA) method may be used to analyse, plan and justify a structured risk based V & V program The criticality analysis & risk assessment (CARA) method may be used to analyse, plan and justify a structured risk based V & V program

23. SQM - 23DCS - ANULECTURE 11-2005 References Sommerville edition 7: chapter 20: Critical Systems Development chapter 24: Critical Systems Validation handout: Determining the Required Level of IV&V Program [Boughton] p 349-357, 2001, IEEE Computer Society Marvin V. Zelkowitz and Iona Rus, Understanding IV & V in a safety critical and complex evolutionary environment: the NASA space shuttle program, ICSE 23 (23 rd International Conference on Software Engineering) p 349-357, 2001, IEEE Computer Society