Center for Reliability Engineering Integrating Software into PRA B. Li, M. Li, A. Sinha, Y. Wei, C. Smidts Presented by Bin Li Center for Reliability Engineering University of Maryland, College Park July 20, 2004

Center for Reliability Engineering Integrating Software into PRA Research Objectives The objective of our research is to extend the current PRA (Probabilistic Risk Assessment) methodology to integrate software in the risk assessment process. Such extension requires modeling the software, the computer platform on which it resides and the interactions it has with other systems.

Center for Reliability Engineering Framework

Center for Reliability Engineering Software related failure mode taxonomy

Center for Reliability Engineering Software related failure mode taxonomy

Center for Reliability Engineering Validation of the Failure Mode Taxonomy Validation Criteria: –Completeness –Consistency –Repeatability –Applicability Validation Process

Center for Reliability Engineering Completeness and Applicability Failure Modes Added By JSC

Center for Reliability Engineering Repeatability and Consistency The conflicts in two rounds

Center for Reliability Engineering Repeatability The measurement of repeatability (R) is the repeatability coefficient (Cohen’s Kappa), Kappa values less than 0.45 indicate inadequate repeatability, values above 0.62 indicate good repeatability, and values above 0.78 indicate excellent repeatability R = 0.46

Center for Reliability Engineering Results of the Validation of the Taxonomy The UMD and the JSC teams reached the following consensus: completeapplied –The taxonomy is complete and can be applied to aerospace systems of various natures; –The taxonomy includes failure modes applicable to autonomous real time systems and mission critical systems; –The taxonomy considers all the failure modes in software; –There is sufficient data available for the validation and enough flexibility to use alternative data. RepeatabilityConsistencyRepeatability and Consistency are adequate.

Center for Reliability Engineering Test-Based Approach - Procedure Identify events/components controlled by software in the MLD Identify events/components controlled by software in accident scenarios Specify the functions involved Modeling of the Software Component in ESDs/ETs and Fault Trees Quantification

Center for Reliability Engineering Identify Software Controlled Events/Components in the MLD

Center for Reliability Engineering Identify Software Controlled Events/Components in Accident Scenarios

Center for Reliability Engineering Identify software behavior from ESD/ET –Identify stimuli and results Identify software component from requirements specifications –Identify inputs and outputs Match stimuli/inputs and results/outputs Specify the Functions Involved

Center for Reliability Engineering Modeling Software Component in ESDs/ETs and FTs

Center for Reliability Engineering Utilizing testing to obtain the probability that the software leads to an unsafe state The process is as follows: –Define the test cases. These test cases cover both the normal input and the abnormal input. The testing strategy includes the identification of normal input space and abnormal input space. Test cases are randomly sampled from these spaces. –Build a Finite State Machine model of the software component to represent its behavior (the oracle). The operational profile derived from the input tree is also embedded into this FSM model. –Automate the testing using the test scripts generated from the FSM model. –Define and identify the software component’s safe and unsafe conditions within the context of each ESD sequence. Quantification

Center for Reliability Engineering Scalability The test based approach can be used for large scale systems because large finite state machines have been built and large systems can be tested by WinRunner. Scalability, describes the relationship between the effort needed to use this method for large systems and the effort needed for the smaller systems which are part of the investigation. Contributors to the effort are: The modeling effort (time to build the finite state machine), The test case generation time (time to generate the test cases in TestMaster) The test execution time (time to execute test cases in WinRunner).

Center for Reliability Engineering Modeling Time COCOMO II is used to calculate the time to construct the finite state machine model. PM = A *(Size) E *27%*25% A=2.94 Emin=0.91 Emax=1.226 Size1=70FP PMmax= 1 and PMmin=0.63 Size2=700FP PMMax=16.5 and PMmin=5.4

Center for Reliability Engineering Test Generation time Test generation time in full coverage is a function of the size of the model. Empirical relations of the following forms can be found: where Empirical study shows :

Center for Reliability Engineering Calculation of FSM Model Size Size of the model is a function of Function Points and the Operation profile. Procedure for calculating the size –Determine the basic size from Function Point calculations for the system. –Determine the reliability requirement for the testing process. –Calculate the number of iterations required for the target reliability. –Calculate the size of the largest iterating sub-model. –Calculate the modified size.

Center for Reliability Engineering Test Execution Time Test Execution time( t exec ) is a linear function of the number of the input/output ( n i/o ), numbers of check points ( m ) and the waiting time for responses( Ts ) Empirical study shows:

Center for Reliability Engineering Summary of Scalability Study The results of the scalability show that: Modeling time can be calculated by using COCOMO II; Test generation time in full cover is a function of the size of the model; Test execution time is a linear function of the number of the input/output, numbers of check points and the waiting time for responses.

Center for Reliability Engineering Ongoing and Future Research Continue the application of a large scale system –The application we chose is CM1 from the Metrics Data Program Finalize the scalability study Continue the support failure modes study Continue the output failure modes study Conduct the fault propagation study