1. University of Southern California Center for Systems and Software Engineering Risk Management CS 577a, Fall 2010 November 24, 2010 11/24/2010©USC-CSSE1

2. University of Southern California Center for Systems and Software Engineering Risk vs Issue A Risk is an uncertain event that could impact your chosen path should it be realised. Risks are events that are not currently affecting you – they haven’t happened yet. Once a risk is realised, it has the potential to become an Issue 11/24/2010©USC-CSSE2 Source: http://agile101.net/2009/07/26/agile-risk-management-the-difference-between-risks-and-issues/http://agile101.net/2009/07/26/agile-risk-management-the-difference-between-risks-and-issues/

3. University of Southern California Center for Systems and Software Engineering Reactively Managing a Software Development Problem System Integration Time: We just started integrating the various software components to be used in our project and we found out that COTS* products A and B can’t talk to one another This is a problem, caused by a previously unrecognized risk, that materialized: The risk that two COTS products might not be able to talk to one another; specifically that A and B might not be able to talk to one another) We’ve got too much tied up in A and B to change Our best solution is to build wrappers around A and B to get them to talk via CORBA** This will result in: –a 3 month schedule overrun = $100K contract penalty –a $300K cost overrun *COTS: Commercial off-the-shelf **CORBA: Common Object Request Broker Architecture 11/24/2010©USC-CSSE3

4. University of Southern California Center for Systems and Software Engineering Proactively Managing a Risk (assessment) System Design Time: –A and B are our strongest COTS choices –But there is some chance that they can’t talk to one another Probability that A and B can’t talk to one another = probability of loss: P(L) From previous experience, with COTS like A and B, we assess P(L) at 50% –If we commit to using A and B, and we find out at integration time that they can’t talk to one another Size of loss S(L) = $300K + $100K = $400K We have a risk exposure of RE = P(L) * S(L) = (.5) * ($400K) = $200K 11/24/2010©USC-CSSE4

5. University of Southern California Center for Systems and Software Engineering Risk Management Strategy 1: Buying Information System Design Time: –Let’s spend $30K and 2 weeks prototyping the integration of A and B –This will buy information on the magnitudes of P(L) and S(L) –If RE = P(L) * S(L) is small, we’ll accept and monitor the risk –If RE is large, we’ll use one/some of the other strategies 11/24/2010©USC-CSSE5

6. University of Southern California Center for Systems and Software Engineering Other Risk Management Strategies Risk Avoidance –COTS product C is almost as good as B, and we know, from having used A and C, that C can talk to A –Delivering on time is worth more to the customer than the small performance loss Risk Transfer –If the customer insists on using A and B, have them establish a risk reserve, to be used in case A and B can’t talk to each other Risk Reduction –If we build the wrappers and the CORBA connections right now, we add cost but minimize the schedule delay Risk Acceptance –If we can solve the A and B interoperability problem, we’ll have a big competitive edge on future procurements –Let’s do this on our own money, and patent the solution 11/24/2010©USC-CSSE6

7. University of Southern California Center for Systems and Software Engineering Software Risk Management 11/24/2010©USC-CSSE7 Checklists Decision driver analysis Assumption analysis Decomposition Performance models Cost models Network analysis Decision analysis Quality factor analysis Risk exposure Risk leverage Compound risk reduction Buying information Risk avoidance Risk transfer Risk reduction Risk element planning Risk plan integration Prototypes Simulations Benchmarks Analyses Staffing Milestone tracking Top-10 tracking Risk reassessment Corrective action Risk Resolution Risk Assessment Risk Control Risk Identification Risk Analysis Risk Prioritization Risk mgmt Planning Risk Management Risk Monitoring

8. University of Southern California Center for Systems and Software Engineering COTS and External Component Risks COTS risks: immaturity; inexperience; COTS incompatibility with application, platform, other COTS; controllability Non-commercial off-the shelf components: reuse libraries, government, universities, etc. –Qualification testing; benchmarking; inspections; reference checking; compatibility analysis 11/24/2010©USC-CSSE8

9. University of Southern California Center for Systems and Software Engineering Risk Exposure Factors (Satellite Experiment Software) 11/24/2010©USC-CSSE9 Unsatisfactory Outcome (UO) Prob (UO) Loss (UO) Risk Exposure A. S/ W error kills experiment B. S/ W error loses key data C. Fault tolerance features cause unacceptable performance D. Monitoring software reports unsafe condition as safe E. Monitoring software reports safe condition as unsafe F. Hardware delay causes schedule overrun G. Data reduction software errors cause extra work H. Poor user interface causes inefficient operation I. Processor memory insufficient J. DBMS software loses derived data 3 - 5 4 - 8 5 6 8 6 1 2 10 8 7 9 3 4 1 5 7 2 30 - 50 24 - 40 28 - 56 45 15 24 8 30 7 4

10. University of Southern California Center for Systems and Software Engineering Risk Management Plans 11/24/2010©USC-CSSE10 For Each Risk Item, Answer the Following Questions: 1. Why? Risk Item Importance, Relation to Project Objectives 2. What, When? Risk Resolution Deliverables, Milestones, Activity Nets 3. Who, Where? Responsibilities, Organization 4. How? Approach (Prototypes, Surveys, Models, …) 5. How Much? Resources (Budget, Schedule, Key Personnel)

11. University of Southern California Center for Systems and Software Engineering Risk Management Plan: Fault Tolerance Prototyping 1. Objectives (The “Why”) –Determine, reduce level of risk of the software fault tolerance features causing unacceptable performance –Create a description of and a development plan for a set of low-risk fault tolerance features 2. Deliverables and Milestones (The “What” and “When”) –By week 3 1. Evaluation of fault tolerance option 2. Assessment of reusable components 3. Draft workload characterization 4. Evaluation plan for prototype exercise 5. Description of prototype –By week 7 6. Operational prototype with key fault tolerance features 7. Workload simulation 8. Instrumentation and data reduction capabilities 9. Draft Description, plan for fault tolerance features –By week 10 10. Evaluation and iteration of prototype 11. Revised description, plan for fault tolerance features 11/24/2010©USC-CSSE11

12. University of Southern California Center for Systems and Software Engineering Risk Management Plan: Fault Tolerance Prototyping ( concluded ) Responsibilities (The “Who” and “Where”) –System Engineer: G. Smith Tasks 1, 3, 4, 9, 11, support of tasks 5, 10 –Lead Programmer: C. Lee Tasks 5, 6, 7, 10 support of tasks 1, 3 –Programmer: J. Wilson Tasks 2, 8, support of tasks 5, 6, 7, 10 Approach (The “How”) –Design-to-Schedule prototyping effort –Driven by hypotheses about fault tolerance-performance effects –Use real-time OS, add prototype fault tolerance features –Evaluate performance with respect to representative workload –Refine Prototype based on results observed Resources (The “How Much”) $60K - Full-time system engineer, lead programmer, programmer (10 weeks)*(3 staff)*($2K/staff-week) $0K - 3 Dedicated workstations (from project pool) $0K - 2 Target processors (from project pool) $0K - 1 Test co-processor (from project pool) $10K - Contingencies $70K - Total 11/24/2010©USC-CSSE12

13. University of Southern California Center for Systems and Software Engineering Risk Monitoring Milestone Tracking –Monitoring of risk Management Plan Milestones Top-10 Risk Item Tracking –Identify Top-10 risk items –Highlight these in monthly project reviews –Focus on new entries, slow-progress items Focus review on manger-priority items Risk Reassessment Corrective Action 11/24/2010©USC-CSSE13

14. University of Southern California Center for Systems and Software Engineering Project Top 10 Risk Item List: Satellite Experiment Software 11/24/2010©USC-CSSE14 Risk Item Mo. Ranking This Last #Mo. Risk Resolution Progress Replacing Sensor-Control Software 1 4 2 Top Replacement Candidate Unavailable Developer Target Hardware Delivery Delays 2 5 2 Procurement Procedural Delays Sensor Data Formats Undefined 3 3 3 Action Items to Software, Sensor Teams; Due Next Month Staffing of Design V&V Team 4 2 3 Key Reviewers Committed; Need Fault- Tolerance Reviewer Software Fault-Tolerance May 5 1 3 Fault Tolerance Prototype Successful Compromise Performance Accommodate Changes in Data 6 - 1 Meeting Scheduled With Data Bus Bus Design Designers Testbed Interface Definitions 7 8 3 Some Delays in Action Items; Review Meeting Scheduled User Interface Uncertainties 8 6 3 User Interface Prototype Successful TBDs In Experiment Operational - 7 3 TBDs Resolved Concept Uncertainties In Reusable - 9 3 Required Design Changes Small, Monitoring Software Successfully Made

15. University of Southern California Center for Systems and Software Engineering Early Risk Management in 577 Project TasksRisk Management Skills; Skill-building activities Select projects; form teamsProject risk identification Staffing risk assessment and resolution - Readings, lectures, homework, case study, guidelines Plan early phasesSchedule/budget risk assessment, planning Risk–driven processes (ICSM) - Readings, lectures, homework, guidelines, planning and estimating tools Formulate, validate concept of operation Risk-driven level of detail - Readings, lecture, guidelines, project Manage to plansRisk monitoring and control - Readings, lecture, guidelines, project Develop, validate FC packageRisk assessment and prioritization - Readings, lecture, guidelines, project FC Architecture ReviewRisk-driven review process Review of top-N project risksReadings, lecture, case studies, review 11/24/2010©USC-CSSE15

16. University of Southern California Center for Systems and Software Engineering Software Risk Management Techniques Source of RiskRisk Management Techniques 1. Personnel shortfallsStaffing with top talent; key personnel agreements; team- building; training; tailoring process to skill mix; walkthroughs 2. Schedules, budgets, Process Detailed, multi-source cost and schedule estimation; design to cost; incremental development; software reuse; requirements descoping; adding more budget and schedule; outside reviews 3. COTS, external components Benchmarking; inspections; reference checking; compatibility prototyping and analysis 4. Requirements mismatch Requirements scrubbing; prototyping; cost-benefit analysis; design to cost; user surveys 5. User interface mismatch Prototyping; scenarios; user characterization (functionality; style, workload); identifying the real users 11/24/2010©USC-CSSE16

17. University of Southern California Center for Systems and Software Engineering Software Risk Management Techniques Source of RiskRisk Management Techniques 6. Architecture, performance, quality Simulation; benchmarking; modeling; prototyping; instrumentation; tuning 7. Requirements changesHigh change threshold: information hiding; incremental development (defer changes to later increments) 8. Legacy softwareReengineering; code analysis; interviewing; wrappers; incremental deconstruction 9. COTS, Externally- performed tasks Pre-award audits, award-fee contracts, competitive design or Prototyping 10. Straining computer science Technical analysis; cost-benefit analysis; prototyping; reference checking 11/24/2010©USC-CSSE17

18. University of Southern California Center for Systems and Software Engineering Different Risks/Opportunities Yield Different Processes 10/21/201018 Architected Agile E.g. Business data processing Use Single NDI E.g. Accounting System NDI-Intensive E.g. Supply Chain Management Services-Intensive E.g. Community Services With addressable risk(s), the project moves on the next phase With provided architecture and functionalities from NDI, the team could spend close to no effort in Valuation and Foundations phase With provided architecture and functionalities from NDI, the team could spend close to no effort in Valuation and Foundations phase The team spends more effort in assessing NDI(s) and their interoperability, enter Operation phase sooner Although the patterns look similar, NDI and services have different risks Although the patterns look similar, NDI and services have different risks

19. University of Southern California Center for Systems and Software Engineering Validation Results on Process Adoption Incidents of Process Selection and Direction Changes 10/21/201019 #of teamsResults on Project Process Selection 8/14Selected the right process pattern from the beginning 3/14Unclear project scope ; re-select right at the end of the Exploration phase 1/14Minor changes on project scope ; right at the end of the Valuation phase 1/14Major change in Foundations phase 1/14Infeasible project scope

20. University of Southern California Center for Systems and Software Engineering Top 10 Risk Categories: 1989 and 1995 11/24/2010©USC-CSSE20 1989 1. Personnel shortfalls 2. Schedules and budgets 3. Wrong software functions 4. Wrong user interface 5. Gold plating 6. Requirements changes 7. Externally-furnished components 8. Externally-performed tasks 9. Real-time performance 10. Straining computer science 1995 1. Personnel shortfalls 2. Schedules, budgets, process 3. COTS, external components 4. Requirements mismatch 5. User interface mismatch 6. Architecture, performance, quality 7. Requirements changes 8. Legacy software 9. Externally-performed tasks 10. Straining computer science

21. University of Southern California Center for Systems and Software Engineering Top 10 Risk Categories: 1995 and 2010 19952010 1. Personnel shortfalls1. Customer-developer-user team cohesion 2. Schedules, budgets, process2. Personnel shortfalls 3. COTS, external components3. Architecture complexity; quality tradeoffs 4. Requirements mismatch4. Budget and schedule constraints 5. User interface mismatch5. COTS and other independently evolving systems 6. Architecture, performance, quality6. Lack of domain knowledge 7. Requirements changes7. Process Quality Assurance 8. Legacy software8. Requirements volatility; rapid change 9. Externally-performed tasks9. User interface mismatch 10. Straining computer science10. Requirements mismatch 11/24/2010©USC-CSSE21

22. University of Southern California Center for Systems and Software Engineering Primary CS577 Risk Categories (all on 1995 list) and Examples Personnel shortfalls: commitment (This is team member’s last course; only needs C to graduate); compatibility; communication problems; skill deficiencies (management, Web design, Java, Perl, CGI, data compression, …) Schedule: project scope too large for 24 weeks; IOC content; critical-path items (COTS, platforms, reviews, …) COTS: see next slide re multi-COTS Rqts, UI: mismatch to user needs (recall overdue book notices) Performance: #bits; #bits/sec; overhead sources Externally-performed tasks: Client/Operator preparation; commitment for transition effort 11/24/2010©USC-CSSE22

23. University of Southern California Center for Systems and Software Engineering Top 11 - Risk distribution in CSCI577 11/24/2010©USC-CSSE23

24. University of Southern California Center for Systems and Software Engineering Comparing between risks in Fall and Spring 11/24/2010©USC-CSSE24

25. University of Southern California Center for Systems and Software Engineering Conclusions Risk management starts on Day One –Delay and denial are serious career risks –Data provided to support early investment Win Win spiral model provides process framework for early risk resolution –Stakeholder identification and win condition reconciliation –Anchor point milestones Risk analysis helps determine “how much is enough” –Testing, planning, specifying, prototyping,… –Buying information to reduce risk 11/24/2010©USC-CSSE25