1. 1 COSC 4406 Software Engineering COSC 4406 Software Engineering Haibin Zhu, Ph.D. Dept. of Computer Science and mathematics, Nipissing University, 100 College Dr., North Bay, ON P1B 8L7, Canada, haibinz@nipissingu.ca, http://www.nipissingu.ca/faculty/haibinz haibinz@nipissingu.ca

2. 2 Lecture 15 Risk Management and Quality Management Ref. Chap. 28, 15 & 16 (7 th Ed.)

3. 3 Project Risks What can go wrong? What is the likelihood? What will the damage be? What can we do about it?

4. 4 Reactive Risk Management project team reacts to risks when they occur project team reacts to risks when they occur mitigation—plan for additional resources in anticipation of fire fighting mitigation—plan for additional resources in anticipation of fire fighting fix on failure—resource are found and applied when the risk strikes fix on failure—resource are found and applied when the risk strikes crisis management—failure does not respond to applied resources and project is in jeopardy crisis management—failure does not respond to applied resources and project is in jeopardy

5. 5 Proactive Risk Management formal risk analysis is performed formal risk analysis is performed organization corrects the root causes of risk organization corrects the root causes of risk TQM concepts and statistical SQA TQM concepts and statistical SQA examining risk sources that lie beyond the bounds of the software examining risk sources that lie beyond the bounds of the software developing the skill to manage change developing the skill to manage change

6. 6 Seven Principles Maintain a global perspective—view software risks within the context of system and the business problem Maintain a global perspective—view software risks within the context of system and the business problem Take a forward-looking view—think about the risks that may arise in the future; establish contingency plans Take a forward-looking view—think about the risks that may arise in the future; establish contingency plans Encourage open communication—if someone states a potential risk, don’t discount it. Encourage open communication—if someone states a potential risk, don’t discount it. Integrate—a consideration of risk must be integrated into the software process Integrate—a consideration of risk must be integrated into the software process Emphasize a continuous process—the team must be vigilant throughout the software process, modifying identified risks as more information is known and adding new ones as better insight is achieved. Emphasize a continuous process—the team must be vigilant throughout the software process, modifying identified risks as more information is known and adding new ones as better insight is achieved. Develop a shared product vision—if all stakeholders share the same vision of the software, it likely that better risk identification and assessment will occur. Develop a shared product vision—if all stakeholders share the same vision of the software, it likely that better risk identification and assessment will occur. Encourage teamwork—the talents, skills and knowledge of all stakeholder should be pooled Encourage teamwork—the talents, skills and knowledge of all stakeholder should be pooled

7. 7 RISK Risk Management Paradigm control identify analyze plan track

8. 8 Risk Identification Product size—risks associated with the overall size of the software to be built or modified. Product size—risks associated with the overall size of the software to be built or modified. Business impact—risks associated with constraints imposed by management or the marketplace. Business impact—risks associated with constraints imposed by management or the marketplace. Customer characteristics—risks associated with the sophistication of the customer and the developer's ability to communicate with the customer in a timely manner. Customer characteristics—risks associated with the sophistication of the customer and the developer's ability to communicate with the customer in a timely manner. Process definition—risks associated with the degree to which the software process has been defined and is followed by the development organization. Process definition—risks associated with the degree to which the software process has been defined and is followed by the development organization. Development environment—risks associated with the availability and quality of the tools to be used to build the product. Development environment—risks associated with the availability and quality of the tools to be used to build the product. Technology to be built—risks associated with the complexity of the system to be built and the "newness" of the technology that is packaged by the system. Technology to be built—risks associated with the complexity of the system to be built and the "newness" of the technology that is packaged by the system. Staff size and experience—risks associated with the overall technical and project experience of the software engineers who will do the work. Staff size and experience—risks associated with the overall technical and project experience of the software engineers who will do the work.

9. 9 Assessing Project Risk-I  Have top software and customer managers formally committed to support the project?  Are end-users enthusiastically committed to the project and the system/product to be built?  Are requirements fully understood by the software engineering team and their customers?  Have customers been involved fully in the definition of requirements?  Do end-users have realistic expectations?

10. 10 Assessing Project Risk-II  Is project scope stable?  Does the software engineering team have the right mix of skills?  Are project requirements stable?  Does the project team have experience with the technology to be implemented?  Is the number of people on the project team adequate to do the job?   Do all customer/user constituencies agree on the importance of the project and on the requirements for the system/product to be built?

11. 11 Risk Components performance risk—the degree of uncertainty that the product will meet its requirements and be fit for its intended use. performance risk—the degree of uncertainty that the product will meet its requirements and be fit for its intended use. cost risk—the degree of uncertainty that the project budget will be maintained. cost risk—the degree of uncertainty that the project budget will be maintained. support risk—the degree of uncertainty that the resultant software will be easy to correct, adapt, and enhance. support risk—the degree of uncertainty that the resultant software will be easy to correct, adapt, and enhance. schedule risk—the degree of uncertainty that the project schedule will be maintained and that the product will be delivered on time. schedule risk—the degree of uncertainty that the project schedule will be maintained and that the product will be delivered on time.

12. 12 Risk Projection Risk projection, also called risk estimation, attempts to rate each risk in two ways Risk projection, also called risk estimation, attempts to rate each risk in two ways the likelihood or probability that the risk is real the likelihood or probability that the risk is real the consequences of the problems associated with the risk, should it occur. the consequences of the problems associated with the risk, should it occur. There are four risk projection steps: There are four risk projection steps: establish a scale that reflects the perceived likelihood of a risk establish a scale that reflects the perceived likelihood of a risk delineate the consequences of the risk delineate the consequences of the risk estimate the impact of the risk on the project and the product, estimate the impact of the risk on the project and the product, note the overall accuracy of the risk projection so that there will be no misunderstandings. note the overall accuracy of the risk projection so that there will be no misunderstandings.

13. 13 Building a Risk Table RiskMitigationMonitoring&Management 1:Catastrophic 2: Critical 3: marginal 4: negligible

14. 14 Building the Risk Table Estimate the probability of occurrence Estimate the probability of occurrence Estimate the impact on the project on a scale of 1 to 4, where Estimate the impact on the project on a scale of 1 to 4, where 4 = low impact on project success 4 = low impact on project success 1= catastrophic impact on project success 1= catastrophic impact on project success Sort the table by probability and impact Sort the table by probability and impact

15. 15 Risk Exposure (Impact) The overall risk exposure, RE, is determined using the following relationship [HAL98]: RE = P x C where P is the probability of occurrence for a risk, and C is the cost to the project should the risk occur.

16. 16 Risk Exposure Example Risk identification. Only 70 percent of the software components scheduled for reuse will, in fact, be integrated into the application. The remaining functionality will have to be custom developed. Risk identification. Only 70 percent of the software components scheduled for reuse will, in fact, be integrated into the application. The remaining functionality will have to be custom developed. Risk probability. 80% (likely). Risk probability. 80% (likely). Risk impact. 60 reusable software components were planned. If only 70 percent can be used, 18 components would have to be developed from scratch (in addition to other custom software that has been scheduled for development). Since the average component is 100 LOC and local data indicate that the software engineering cost for each LOC is $14.00, the overall cost (impact) to develop the components would be 18 x 100 x 14 = $25,200. Risk impact. 60 reusable software components were planned. If only 70 percent can be used, 18 components would have to be developed from scratch (in addition to other custom software that has been scheduled for development). Since the average component is 100 LOC and local data indicate that the software engineering cost for each LOC is $14.00, the overall cost (impact) to develop the components would be 18 x 100 x 14 = $25,200. Risk exposure. RE = 0.80 x 25,200 ~ $20,200. Risk exposure. RE = 0.80 x 25,200 ~ $20,200.

17. 17 mitigation—how can we avoid the risk? mitigation—how can we avoid the risk? monitoring—what factors can we track that will enable us to determine if the risk is becoming more or less likely? monitoring—what factors can we track that will enable us to determine if the risk is becoming more or less likely? management—what contingency plans do we have if the risk becomes a reality? management—what contingency plans do we have if the risk becomes a reality? Risk Mitigation, Monitoring, and Management

18. 18 Risks due to Product Size Product Size Business Impact Business Impact the Customer the Customer Process Maturity Process Maturity Technology Technology Staff/People Staff/People

19. 19 Risk Due to Product Size estimated size of the product in LOC or FP? estimated size of the product in LOC or FP? estimated size of product in number of programs, estimated size of product in number of programs, files, transactions? percentage deviation in size of product from percentage deviation in size of product from average for previous products? size of database created or used by the product? size of database created or used by the product? number of users of the product? number of users of the product? number of projected changes to the requirements number of projected changes to the requirements for the product? before delivery? after delivery? amount of reused software? amount of reused software? Attributes that affect risk:

20. 20 Risk Due to Business Impact affect of this product on company revenue? affect of this product on company revenue? visibility of this product by senior management? visibility of this product by senior management? reasonableness of delivery deadline? reasonableness of delivery deadline? number of customers who will use this product interoperability constraints sophistication of end users? number of customers who will use this product interoperability constraints sophistication of end users? amount and quality of product documentation that must be produced and delivered to the customer? amount and quality of product documentation that must be produced and delivered to the customer? governmental constraints costs associated with late delivery? governmental constraints costs associated with late delivery? costs associated with a defective product? costs associated with a defective product? Attributes that affect risk:

21. 21 Risks Due to the Customer Have you worked with the customer in the past? Have you worked with the customer in the past? Does the customer have a solid idea of requirements? Does the customer have a solid idea of requirements? Has the customer agreed to spend time with you? Has the customer agreed to spend time with you? Is the customer willing to participate in reviews? Is the customer willing to participate in reviews? Is the customer technically sophisticated? Is the customer technically sophisticated? Is the customer willing to let your people do their Is the customer willing to let your people do their job—that is, will the customer resist looking over your shoulder during technically detailed work? Does the customer understand the software Does the customer understand the software engineering process? Questions that must be answered:

22. 22 Risks Due to Process Maturity Have you established a common process framework? Have you established a common process framework? Is it followed by project teams? Is it followed by project teams? Do you have management support for Do you have management support for software engineering Do you have a proactive approach to SQA? Do you have a proactive approach to SQA? Do you conduct formal technical reviews? Do you conduct formal technical reviews? Are CASE tools used for analysis, design and Are CASE tools used for analysis, design and testing? Are the tools integrated with one another? Are the tools integrated with one another? Have document formats been established? Have document formats been established? Questions that must be answered:

23. 23 Technology Risks Is the technology new to your organization? Is the technology new to your organization? Are new algorithms, I/O technology required? Are new algorithms, I/O technology required? Is new or unproven hardware involved? Is new or unproven hardware involved? Does the application interface with new software? Does the application interface with new software? Is a specialized user interface required? Is a specialized user interface required? Is the application radically different? Is the application radically different? Are you using new software engineering methods? Are you using new software engineering methods? Are you using unconventional software development Are you using unconventional software development methods, such as formal methods, AI-based approaches, artificial neural networks? Are there significant performance constraints? Are there significant performance constraints? Is there doubt the functionality requested is "do-able?" Is there doubt the functionality requested is "do-able?" Questions that must be answered:

24. 24 Staff/People Risks Are the best people available? Are the best people available? Does staff have the right skills? Does staff have the right skills? Are enough people available? Are enough people available? Are staff committed for entire duration? Are staff committed for entire duration? Will some people work part time? Will some people work part time? Do staff have the right expectations? Do staff have the right expectations? Have staff received necessary training? Have staff received necessary training? Will turnover among staff be low? Will turnover among staff be low? Questions that must be answered:

25. 25 Project: Embedded software for XYZ system Risk type: schedule risk Priority (1 low... 5 critical): 4 Risk factor: Project completion will depend on tests which require hardware component under development. Hardware component delivery may be delayed Probability: 60 % Impact: Project completion will be delayed for each day that hardware is unavailable for use in software testing Monitoring approach: Scheduled milestone reviews with hardware group Scheduled milestone reviews with hardware group Contingency plan: Modification of testing strategy to accommodate delay using Modification of testing strategy to accommodate delay using software simulation software simulation Estimated resources: 6 additional person months beginning 7-1-96 Recording Risk Information

26. 26 Quality The American Heritage Dictionary defines quality as The American Heritage Dictionary defines quality as “a characteristic or attribute of something.” “a characteristic or attribute of something.” For software, two kinds of quality may be encountered: For software, two kinds of quality may be encountered: Quality of design encompasses requirements, specifications, and the design of the system. Quality of design encompasses requirements, specifications, and the design of the system. Quality of conformance is an issue focused primarily on implementation. Quality of conformance is an issue focused primarily on implementation. user satisfaction = compliant product + good quality + delivery within budget and schedule user satisfaction = compliant product + good quality + delivery within budget and schedule

27. 27 Software Quality Conformance to explicitly stated functional and performance requirements, explicitly documented development standards, and implicit characteristics that are expected of all professionally developed software.

28. 28 Cost of Quality Prevention costs include Prevention costs include quality planning quality planning formal technical reviews formal technical reviews test equipment test equipment Training Training Internal failure costs include Internal failure costs include rework rework repair repair failure mode analysis failure mode analysis External failure costs are External failure costs are complaint resolution complaint resolution product return and replacement product return and replacement help line support help line support warranty work warranty work

29. 29 Software Quality Assurance Formal Technical Reviews Test Planning & Review Measurement Analysis & Reporting Process Definition & Standards

30. 30 Role of the SQA Group-I Prepares an SQA plan for a project. Prepares an SQA plan for a project. The plan identifies The plan identifies evaluations to be performed evaluations to be performed audits and reviews to be performed audits and reviews to be performed standards that are applicable to the project standards that are applicable to the project procedures for error reporting and tracking procedures for error reporting and tracking documents to be produced by the SQA group documents to be produced by the SQA group amount of feedback provided to the software project team amount of feedback provided to the software project team Participates in the development of the project’s software process description. Participates in the development of the project’s software process description. The SQA group reviews the process description for compliance with organizational policy, internal software standards, externally imposed standards (e.g., ISO-9001), and other parts of the software project plan. The SQA group reviews the process description for compliance with organizational policy, internal software standards, externally imposed standards (e.g., ISO-9001), and other parts of the software project plan.

31. 31 Role of the SQA Group-II Reviews software engineering activities to verify compliance with the defined software process. Reviews software engineering activities to verify compliance with the defined software process. identifies, documents, and tracks deviations from the process and verifies that corrections have been made. identifies, documents, and tracks deviations from the process and verifies that corrections have been made. Audits designated software work products to verify compliance with those defined as part of the software process. Audits designated software work products to verify compliance with those defined as part of the software process. reviews selected work products; identifies, documents, and tracks deviations; verifies that corrections have been made reviews selected work products; identifies, documents, and tracks deviations; verifies that corrections have been made periodically reports the results of its work to the project manager. periodically reports the results of its work to the project manager. Ensures that deviations in software work and work products are documented and handled according to a documented procedure. Ensures that deviations in software work and work products are documented and handled according to a documented procedure. Records any noncompliance and reports to senior management. Records any noncompliance and reports to senior management. Noncompliance items are tracked until they are resolved. Noncompliance items are tracked until they are resolved.

32. 32 Why SQA Activities Pay Off? cost to find and fix a defect 100 10 log scale 1 Req. Design code test system test field use 1 3-6 10 15-40 30-70 40-100

33. 33 Formal Technical Reviews (FTR)... there is no particular reason why your friend and colleague cannot also be your sternest critic. Jerry Weinberg Reviews & Inspections

34. 34 What Are Reviews? a meeting conducted by technical people for technical people a meeting conducted by technical people for technical people a technical assessment of a work product created during the software engineering process a technical assessment of a work product created during the software engineering process a software quality assurance mechanism a software quality assurance mechanism a training ground a training ground

35. 35 What Reviews Are Not A project summary or progress assessment A project summary or progress assessment A meeting intended solely to impart information A meeting intended solely to impart information A mechanism for political or personal reprisal! A mechanism for political or personal reprisal!

36. 36 The Players review leader producer recorder reviewer standards bearer (SQA) maintenanceoracle user rep

37. 37 Conducting the Review be prepared—evaluate product before the review review the product, not the producer keep your tone mild, ask questions instead of making accusations stick to the review agenda raise issues, don't resolve them avoid discussions of style—stick to technical correctness schedule reviews as project tasks record and report all review results 1. 2. 3. 4. 5. 6. 7. 8.

38. 38 Review Options Matrix trained leader agenda established reviewers prepare in advance producer presents product “reader” presents product recorder takes notes checklists used to find errors errors categorized as found issues list created team must sign-off on result IPR—informal peer review WT—Walkthrough IN—Inspection RRR—round robin review IPRWTIN RRR nomaybemaybemaybenomaybenonononoyesyesyesyesnoyesnonoyesyes yesyesyesnoyesyesyesyesyesyesyesyesyesnonoyesnonoyesmaybe * *

39. 39 Sample-Driven Reviews (SDRs) SDRs attempt to quantify those work products that are primary targets for full FTR (Formal Technical Review)s. SDRs attempt to quantify those work products that are primary targets for full FTR (Formal Technical Review)s. To accomplish this … Inspect a fraction a i of each software work product, i. Record the number of faults, f i found within a i. Inspect a fraction a i of each software work product, i. Record the number of faults, f i found within a i. Develop a gross estimate of the number of faults within work product i by multiplying f i by 1/a i. Develop a gross estimate of the number of faults within work product i by multiplying f i by 1/a i. Sort the work products in descending order according to the gross estimate of the number of faults in each. Sort the work products in descending order according to the gross estimate of the number of faults in each. Focus available review resources on those work products that have the highest estimated number of faults. Focus available review resources on those work products that have the highest estimated number of faults.

40. 40 Metrics Derived from Reviews inspection time per page of documentation inspection time per KLOC or FP errors uncovered per reviewer hour errors uncovered per preparation hour errors uncovered per SE task (e.g., design) number of minor errors (e.g., typos) number of errors found during preparation number of major errors (e.g., nonconformance to req.) (e.g., nonconformance to req.) inspection effort per KLOC or FP

41. 41 Statistical SQA Product & Process measurement... an understanding of how to improve quality... Collect information on all defects Find the causes of the defects Move to provide fixes for the process Page 728-729

42. 42 Six-Sigma for Software Engineering The term “six sigma” is derived from six standard deviations—3.4 instances (defects) per million occurrences—implying an extremely high quality standard. The term “six sigma” is derived from six standard deviations—3.4 instances (defects) per million occurrences—implying an extremely high quality standard. The Six Sigma methodology defines three core steps: The Six Sigma methodology defines three core steps: Define customer requirements and deliverables and project goals via well- defined methods of customer communication Define customer requirements and deliverables and project goals via well- defined methods of customer communication Measure the existing process and its output to determine current quality performance (collect defect metrics) Measure the existing process and its output to determine current quality performance (collect defect metrics) Analyze defect metrics and determine the vital few causes. Analyze defect metrics and determine the vital few causes. Two additional: Two additional: Improve the process by eliminating the root causes of defects. Improve the process by eliminating the root causes of defects. Control the process to ensure that future work does not reintroduce the causes of defects. http://mathworld.wolfram.com/StandardDeviation.html

43. 43 Software Reliability A simple measure of reliability is mean-time-between- failure (MTBF), where A simple measure of reliability is mean-time-between- failure (MTBF), where MTBF = MTTF + MTTR The acronyms MTTF and MTTR are mean-time-to-failure and mean-time-to-repair, respectively. Software availability is the probability that a program is operating according to requirements at a given point in time and is defined as Software availability is the probability that a program is operating according to requirements at a given point in time and is defined as Availability = [MTTF/(MTTF + MTTR)] x 100%

44. 44 Software Safety Software safety is a software quality assurance activity that focuses on the identification and assessment of potential hazards that may affect software negatively and cause an entire system to fail. Software safety is a software quality assurance activity that focuses on the identification and assessment of potential hazards that may affect software negatively and cause an entire system to fail. If hazards can be identified early in the software process, software design features can be specified that will either eliminate or control potential hazards. If hazards can be identified early in the software process, software design features can be specified that will either eliminate or control potential hazards.

45. 45 ISO 9001:2000 Standard ISO 9001:2000 is the quality assurance standard that applies to software engineering. ISO 9001:2000 is the quality assurance standard that applies to software engineering. The standard contains 20 requirements that must be present for an effective quality assurance system. The standard contains 20 requirements that must be present for an effective quality assurance system. The requirements delineated by ISO 9001:2000 address topics such as The requirements delineated by ISO 9001:2000 address topics such as management responsibility, quality system, contract review, design control, document and data control, product identification and traceability, process control, inspection and testing, corrective and preventive action, control of quality records, internal quality audits, training, servicing, and statistical techniques. management responsibility, quality system, contract review, design control, document and data control, product identification and traceability, process control, inspection and testing, corrective and preventive action, control of quality records, internal quality audits, training, servicing, and statistical techniques.

46. 46 Summary Reactive/proactive Risk Strategies Reactive/proactive Risk Strategies Risk Identification Risk Identification Risk Projection Risk Projection Risk Mitigation, Monitoring and Management Risk Mitigation, Monitoring and Management Quality Concepts Quality Concepts SQA SQA Software reviews Software reviews Statistical SQA Statistical SQA Software reliability and safety Software reliability and safety ISO 9000 Quality Standards ISO 9000 Quality Standards