1. بسم الله الرحمن الرحيم
الحمد لله ، والصلاة والسلام على رسول الله
Introduction to Risk Management And Software Architecture Risk Assessment
Hany H. Ammar
LANE Department of Computer Science and Electrical Engineering West Virginia University, Morgantown, West Virginia, USA, and
Faculty of Computers and Information, Cairo University, Cairo, Egypt

2. OUTLINE Risk Management Software Architecture Risk Assessment
Maintainability-based risk
Conclusions
Next Steps
SW Architecture Risk Assessment Keynote Presentation MySEC’06

3. Risk Management
SW Architecture Risk Assessment Keynote Presentation MySEC’06

4. SW Architecture Risk Assessment Keynote Presentation MySEC’06

5. For NASA Programs
RISK MANAGEMENT: An organized, systematic decision-making process that efficiently identifies risks, assesses or analyzes risks, and effectively reduces or eliminates risks to achieving program goals.
RISK: A Program “Risk” is any circumstance or situation that poses a threat to: crew or vehicle safety, Program controlled cost; Program controlled schedule; or major mission objectives, and for which an acceptable resolution is deemed unlikely without a focused management effort
SW Architecture Risk Assessment Keynote Presentation MySEC’06

6. Risk Management Cycle
Identify: Identify that a risk exits and give it a meaningful name.
Analyze: Determine the severity of the risk according to the risk matrix. If the risk is negligible (low to medium severity, low likelihood of occurrence), stop here. However, if the risk could cause damage to the system or the system's users, continue.
Plan: Decide how to combat the risk based on the risk's severity and likelihood of occurrence.
Mitigate: Follow the plan formulated in the previous phase as closely as possible to combat the risk. If this approach does not work, return to the previous phase and make a new plan. If the plan does work, continue analyzing the risk to determine whether it has been reduced to an acceptable severity level.
Track: Once the risk has been mitigated to an acceptable severity level, the risk should be tracked to ensure the continued control of the risk. If at any time the risk seems to resurface, the risk management cycle should begin again, starting with the analysis phase.
SW Architecture Risk Assessment Keynote Presentation MySEC’06

7. SW Architecture Risk Assessment Keynote Presentation MySEC’06

8. Risk Definition
According to NASA Software Safety Technical Standard, risk is defined as:
“exposure to the chance of injury or loss. It is a function of the possible frequency of occurrence of the undesired event, of the potential severity of resulting consequences, and of the uncertainties associated with the frequency and severity”.
For software intensive systems, a risk is a combination of a likelihood of occurrence of an abnormal event or failure and the potential consequences or severity of that event or failure to a system's operators, users, or environment
SW Architecture Risk Assessment Keynote Presentation MySEC’06

9. Likelihood of Occurrence
Risk Matrix
Severity
Likelihood of Occurrence
Probable
Occasional
Remote
Improbable
Catastrophic
High
High-Medium
Medium
Critical
Medium-Low
Marginal
Low
Negligible
SW Architecture Risk Assessment Keynote Presentation MySEC’06

10. SW Architecture Risk Assessment Keynote Presentation MySEC’06

11. NASA IV&V Facility
NPD C for Software IV&V Policy states: "Task the IV&V Facility in Fairmont, West Virginia to manage the performance of all IV&V for software identified per the established criteria, and for any other safety critical software (as defined in NASA-STD )"
SW Architecture Risk Assessment Keynote Presentation MySEC’06

12. IV&V Function
Software Independent Verification & Validation (IV&V) is a systems engineering process employing rigorous methodologies for evaluating the correctness and quality of the software product throughout the software life cycle.
Software IV&V is adapted to the characteristics of the project. Different projects require different level of IV&V
SW Architecture Risk Assessment Keynote Presentation MySEC’06

13. IV&V Lifecycle Activities
System Requirements Review
Preliminary Design
Review
Critical
Design
Review
System Test
Mission Readiness Review
System
Retirement
S/W
FQT
Launch
Initial
IVVP
Signed
Baseline
IVVP
Signed
- IV&V provides support and reports for Project milestones
Technical Analysis Reports document major phases
IVVP is updated to match changes in Project
IV&V
Provides
CoFR
IV&V Final Report
Concept
Phase
2.0
Requirements
Phase
3.0
Design
Phase
4.0
Implementation
Phase
5.0
Test
Phase
6.0
Operations &
Maintenance Phase
7.0
IV&V Phase Independent Support
1.0
Note: numbers correspond to IV&V WBS
Life-cycle IV&V is designed to mesh with the Project schedule and provide timely inputs to mitigate risk
Dialog between the IV&V Facility and the Project must begin before SRR
For most Projects, IV&V ends (and the Final Report is delivered) on or about MRR. Some Projects have extended S/W development post-launch or major upgrades/maintenance (e.g. Shuttle, MER)
SW Architecture Risk Assessment Keynote Presentation MySEC’06

14. Software Project Resolution
Project Resolution is commonly categorized into three resolution types:
Successful Projects
Completed and operational, and:
On Schedule
On Cost
With all originally specified features and functions
Challenged Projects
Completed and operational, but:
Behind Schedule  Project Risk
Over Cost  Project Risk
With fewer features and functions than originally specified
 Product Risk
Failed Projects:
Cancelled before completion or never implemented
SW Architecture Risk Assessment Keynote Presentation MySEC’06

15. Software CHAOS
The Standish Group has examined 30,000 Software Projects in the US since This "CHAOS" research has revealed a decided improvement in IT project management with the implementation of standards and practices such as IV&V. This improvement correlates with the rise in project success depicted in the chart below:
Project Resolution History ( )
The Standish Group International, Inc.: Extreme CHAOS (2001) - The 2001 update to the CHAOS report.
SW Architecture Risk Assessment Keynote Presentation MySEC’06

16. Error Detection/Correction
Early error detection and correction are vital. The cost to correct software errors multiplies during the software development lifecycle. Early error detection and correction reduce costs and save time.
Direct Return on Investment of Software Independent Verification and Validation:
Methodology and Initial Case Studies, James B. Dabney and Gary Barber, Assurance Technology
Symposium, 5 June 2003.
SW Architecture Risk Assessment Keynote Presentation MySEC’06

17. IV&V Characteristics
Includes Risk Identification and Mitigation Techniques
Provides Independent Evaluation / Assessment of:
Are we building the product right? = Verification
Are we building the right product? = Validation
Requires Technical, Managerial and Financial Independence
Makes a value added contribution, everyone shares the same mission success objective
For NASA Management - Provides Mission Assurance
For Project Management - Provides Unbiased Source of Help
Helps deliver
Risk Identification and Mitigation
Increased Quality and Safety
Improved Timeliness and Reliability
Reduced Rework Cost
NPD : Requires NASA programs and projects that contain mission or safety critical software to document decisions concerning the use of IV&V.
SW Architecture Risk Assessment Keynote Presentation MySEC’06

18. OUTLINE Risk Management Software Architecture Risk Assessment
Reliability-based risk
Performance-based risk
Maintainability-based risk
Component Ranking
Conclusions
Next Steps
SW Architecture Risk Assessment Keynote Presentation MySEC’06

19. Software Architecture Risk Assessment
This work is funded in part by grants to West Virginia University Research Corp. from the NSF (ITR) Program, and from the NASA Office of Safety and Mission Assurance (OSMA) Software Assurance Research Program (SARP) managed through the NASA Independent Verification and Validation (IV&V) Facility, Fairmont
SW Architecture Risk Assessment Keynote Presentation MySEC’06

20. What keeps satellites working 24/7 ?
Project Overview
Risk Assessment of software architecture components, usage scenarios, and requirements
Risk definition is based on
* Frequency of abnormal events
* Severity or consequences of events
Reliability-based risk,
Performance-based risk
Requirement–based risk
Severity Analysis
Maintainability-based risk
What keeps satellites working 24/7 ?
SW Architecture Risk Assessment Keynote Presentation MySEC’06

21. Software Architecture Risk Assessment
An architecture-based approach for risk assessment
Components\connectors, requirements, and scenario risk,
Define several types of risk factors
Reliability-based Risk [IEEE Trans. on Rel 2001, on SE, 2002, 2003]
Probability of failure
* Severity or Consequences of this failure
Maintainability-based Risk [RAMS 06, ICSM 05, ICSM 04]
Probability of performing maintenance task
* Cost of performing this task
The losses caused by low system maintainability can be:
High cost of maintenance effort
Loss of the system by aging
Performance-based Risk [IEEE Trans. SE, Jan. 2005]
Probability of missing timing or performance requirements
* Severity or Consequences
SW Architecture Risk Assessment Keynote Presentation MySEC’06

22. Importance / Benefits
Components
Risk
Factor
Components Risk
Identify the high risk components of the system in terms of Reliability/Maintainability/Performance
SW Architecture Risk Assessment Keynote Presentation MySEC’06

23. Software Architecture Risk Assessment Methodology
Requirements Model
System Architecture Model
Maintainability-based
Risk Analysis
Performance-based
Reliability-based
Software Architecture Risk Assessment
Components
Ranking
Components Risk Factors
SW Architecture Risk Assessment Keynote Presentation MySEC’06

24. OUTLINE Risk Management Software Architectures
Software Architecture Risk Assessment
Maintainability-based risk
Conclusions
Next Steps
SW Architecture Risk Assessment Keynote Presentation MySEC’06

25. Maintainability-based Risk
[ICSM 05] AbdelMoez, Goseva, Ammar, Mili, Fuhrman, “Architectural level Maintainability Based Risk Assessment”
[RAMS 06] AbdelMoez, Goseva, Ammar,” Methodology for Maintainability Based Risk Assessment”, Jan 2006.
SW Architecture Risk Assessment Keynote Presentation MySEC’06

26. Importance / Benefits Maintainability-based Risk
According to Pigoski, 60%-80% of the system budget is spent on maintenance
Enhancements (perfective/ adaptive maintenance) account for 78%-83% of the maintainer effort
SW Architecture Risk Assessment Keynote Presentation MySEC’06

27. Importance / Benefits Maintainability-based Risk
Unisys holds the NASA contract to maintain and support 14 million lines of ground software for the space shuttle
There were 3,800 requirement changes made to the software after the loss of Challenger. These changes resulted in 900 software releases, of which 30 applied to the mission-control center with 3 of these being major upgrades
Reference:
IEEE Software, Vol.6,  No.1,  pp
SW Architecture Risk Assessment Keynote Presentation MySEC’06

28. Importance / Benefits Trade off Analysis for Perfective Maintenance
Risk
Factor
Components Risk
Components
Patterns
Maintainability risk for perfective maintenance (open source case study Borg)
SW Architecture Risk Assessment Keynote Presentation MySEC’06

29. Requirments maturity Index / change / error reports
Software Architecture Risk Assessment Methodology: Maintainability-based Risk
SW Architecture
Requirments maturity Index
/ change / error reports
(1) Estimate components
Initial Change Probability (ICP)
(2) Estimate Change Propagation (CP) probabilities
(3) Estimate Size of Change (SC)
ICP=[icpi]
SC=[sci/j]
CP=[cpi/j]
(4) Estimate component risk factor
SW Architecture Risk Assessment Keynote Presentation MySEC’06

30. Maintenance change propagation
Outgoing maintenance
Incoming maintenance
SW Architecture Risk Assessment Keynote Presentation MySEC’06

31. Estimating Change PropagationV1 C1 C2
= 1
V11
V12
Change in Provided
Service
V13
Change .
V13
Required
Services
= 0
Change Propagation Probabilities matrix CP=[cpij ]
cpij is the probability that a change in Ci due to corrective/ perfective maintenance requires a change in Cj while maintaining the overall function of a system S
cpij = P([Cj]  [Cj'] | [Ci]  [Ci'] ^ [S] = [S'] )
cpij is estimated by
cpij =
SW Architecture Risk Assessment Keynote Presentation MySEC’06

32. Estimating Size of Change
V11 V1
V12
V13
Change
Change in Provided
Service M1 M2 M3 … M7
Receiving
Component
methods
Size of change SC=[scij ]
scij is defined as the ratio between the number of affected methods of the receiving component caused by the changes in the interface elements of the providing components and the total number of methods in the receiving component
scij is estimated by
scij =
SW Architecture Risk Assessment Keynote Presentation MySEC’06

33. CM1 Maintainability-Based Risk in Adaptive Maintenance Context
SW Architecture Risk Assessment Keynote Presentation MySEC’06

34. Case Study: NASA CM1 UML Model Structure Diagram
The UML-RT Model of CM1 was
Developed by WVU students (Nathan, Tom and Rajesh, Summer 2004) based on the CM1 software design specification
SW Architecture Risk Assessment Keynote Presentation MySEC’06

35. Change Propagation Probabilities for CM1
The Change Propagation probabilities CP is estimated using the CM1 UML model
The Change Propagation probabilities CP can be automatically estimated from UML-RT models, or java source
SW Architecture Risk Assessment Keynote Presentation MySEC’06

36. Size of Change for CM1
The Size of Change metrics SC is estimated using the CM1 UML model
The Size of Change metrics SC Probabilities CP can be automatically estimated from UML-RT models, or Java source
SW Architecture Risk Assessment Keynote Presentation MySEC’06

37. Requirments maturity Index / change / error reports
Software Architecture Risk Assessment Methodology: Maintainability-based Risk
SW Architecture
Requirments maturity Index
/ change / error reports
(1) Estimate components
Initial Change Probability (ICP)
(2) Estimate Change Propagation (CP) probabilities
(3) Estimate Size of Change (SC)
ICP=[icpi]
SC=[sci/j]
CP=[cpi/j]
(4) Estimate component risk factor
SW Architecture Risk Assessment Keynote Presentation MySEC’06

38. Maintainability-based Risk For corrective maintenance (case study CM1)
ICP is estimated
using error reports data
SW Architecture Risk Assessment Keynote Presentation MySEC’06

39. Prioritizing Corrective Maintenance Tasks for CM1
The CM1 case study has 98 error reports of components bugs.
Assuming that these errors have not been yet fixed, we calculate the frequency of errors occurrences in the components of the system.
Then, we estimate the initial change probability ICP of the components by normalizing the frequency of error occurrences by the total number of error reports.
For maintenance tasks of components with high severity-levels, they should be fixed regardless of their corresponding maintainability-based risk because of the consequences of such potential failures on the system.
On the other hand for maintenance tasks of components that have low severity-levels, we should examine the components maintainability-based risk. So, maintenance tasks of low severity-level and high maintainability-based risk should be avoided or delayed in the maintenance plan
Critical
Major
Cat
Cat.
Minor
Severity Level
TMALI
TIS
SSI
SCUI
1553
ICUI
EDAC
DPA
DCX
DCI
CCM
BIT
Components
SW Architecture Risk Assessment Keynote Presentation MySEC’06

40. using change reports data
Maintainability-based Risk Maintainability risk for Adaptive maintenance (case study CM1)
ICP is estimated
using change reports data
SW Architecture Risk Assessment Keynote Presentation MySEC’06

41. Tool Support
SW Architecture Risk Assessment Keynote Presentation MySEC’06

42. Technology Readiness Level The Software Architecture Risk Assessment Tool Support
SW Architecture Risk Assessment Keynote Presentation MySEC’06

43. The tool will be developed as a web application
SW Architecture Risk Assessment Keynote Presentation MySEC’06

44. Conclusions
Risk Management is vital to the success of projects and products
A risk Assessment process is needed
Software Architecture is a major determinant of software quality
Software Architecture can be used to manage project and product risks
Development of a methodology and a process for software architecture risk assessment
Continued development of a software architecture risk assessment tool to support the methodology
SW Architecture Risk Assessment Keynote Presentation MySEC’06

45. Papers Published
Vittorio Cortellessa, Katerina Goseva-Popstojanova, Kalaivani Appukkutty, Ajith R. Guedem, Ahmed Hassan, Rania Elnaggar, Walid Abdelmoez, Hany H. Ammar, “Model-Based Performance Risk Analysis, IEEE Transactions on Software Engineering, January 2005, (Vol. 31, No. 1), pp.3-20.
Katerina Goseva-Popstojanova, Ahmed Hassan, Ajith Guedem, Walid Abdelmoez, Diaa Eldin M. Nassar, Hany Ammar, Ali Mili, "Architectural-Level Risk Analysis Using UML", IEEE Transactions on Software Engineering, October 2003 (Vol. 29, No. 10), pp
S. Yacoub, H. Ammar, “A Methodology for Architectural-Level Reliability Risk Analysis,” IEEE Transactions on Software Engineering, Vol. 28, No. 6, June 2002.
W. AbdelMoez, K. Goseva-Popstojanova, H.H. Ammar,” Methodology for Maintainability-Based Risk Assessment”, Proc. of the 52nd Annual Reliability & Maintainability Symposium (RAMS 2006), Newport Beach, Ca., January 23-26,  
Israr P. Shaik , W. Abdelmoez, R. Gunnalan, M. Shereshevsky, A. Zeid, H.H. Ammar, A. Mili, C. Fuhrman, “Change Propagation for Assessing Design Quality of Software Architectures”, Proc. of 5th IEEE/IFIP Working Conference on Software Architecture (WICSA), Pittsburgh, Pa., USA, November 6-9, 2005.
AbdelMoez, W., I. Shaik, R. Gunnalan, M. Shereshevsky, K. Goseva-Popstojanova, H.H. Ammar, A. Mili, C. Fuhrman, “Architectural level Maintainability Based Risk Assessment”, Proc. of poster papers in IEEE International Conference on Software Maintenance (ICSM 2005), Budapest, Hungary, September 25-30,2005.
W. Abdelmoez, D. M. Nassar, M. Shereshevsky, N. Gradetsky, R. Gunnalanm and H. H. Ammar, Bo Yu, and Ali Mili "Error Propagation in Software Architectures". In Proceedings of the 10th International Symposium on Software Metrics (METRICS'04), September , 2004 , IEEE Comp. Soc., pp
Abdelmoez, W., M. Shereshevsky, R. Gunnalan, H. H. Ammar, Bo Yu, S. Bogazzi, M. Korkmaz, A. Mili, "Software Architectures Change Propagation Tool (SACPT),” 20th IEEE International Conference on Software Maintenance (ICSM'04) September , 2004 , Chicago, Illinois, IEEE Comp. Soc., pp 517
A. Hassan, K. Goseva-Popstojanova, and H. Ammar, “UML Based Severity Analysis Methodology”, Proceedings of the 2005 Annual Reliability and Maintainability Symposium (RAMS 2005), Alexandria, VA, January 2005.
SW Architecture Risk Assessment Keynote Presentation MySEC’06