1. OSMA2003 Center for Reliability Engineering 1 Integrating Software into PRA Presented by C. Smidts Center for Reliability Engineering University of Maryland

2. Center for Reliability Engineering 2 Integrating Software into PRA Probabilistic Risk Assessment (PRA) is a technique to assess the probability of failure or success of a mission. Current PRA neglects the contributions of software to the risk of the mission. The objective of our research is to extend current PRA methodology to integrate software in the risk assessment process.

3. Center for Reliability Engineering 3 What We Have Done to Date Built a Software Failure Mode Taxonomy Failure Modes’ Quantification: generic high-level data -Public Literature -Expert Opinion Collaborated with JSC through Ms. Alice Lee -Validate Our Methodology -Collect Data Developed a Test-Based Methodology for Integrating Software Into PRA

4. Center for Reliability Engineering 4 What We Are Planning to Do in the Future Investigate Scalability Issues of the Test-based Approach Continue the validation of our methodology with JSC Apply the approach to JSC system Revise the methodology based on NASA system Develop an Analytical Approach Apply the Analytical Approach to JSC system Revise the Analytical Approach based on NASA system

5. OSMA2003 Center for Reliability Engineering 5 Integrating Software into PRA: A Test-based Approach Presented by C. Smidts C. Smidts, B. Li, M. Li Center for Reliability Engineering University of Maryland

6. Center for Reliability Engineering 6 Integrating Software into PRA - Approach We are working on an approach to integrate software into PRA. -Step 1: Identify events/components controlled/supported by software in MLD, accident scenarios, fault trees. -Step 2: Specify the functions involved -Step 3: Model software functions in ESDs/ETs and Fault Trees -Step 4: Construct the input tree -Step 5: Quantify the input tree -Step 6: Develop and perform software safety tests

7. Center for Reliability Engineering 7 Example System An exit system in a building is used as the example in this case study. The exit system includes an emergency exit system and the PACS system. The Emergency exit system includes an emergency exit door and a marked egress router. It provides an escape route for personnel located inside the building during emergency situations. The PACS system is a simplified version of an automated personal entry/exit access system used to provide privileged physical access to rooms /buildings, etc. Personal ID and PIN are needed to access this system.

8. Center for Reliability Engineering 8 Integrating software into PRA - Approach Step 1: Identify events/components Identify events/components controlled/supported by software in MLD, accident scenarios, fault trees. For all such events, create/expand contributors to account for software. Verify that no neglected “events” may now have become possible due to software.

9. Center for Reliability Engineering 9 MLD

10. Center for Reliability Engineering 10 MLD

11. Center for Reliability Engineering 11 Accident Description Fire is the initiating event Response systems: Emergency system and PACS system End State: Loss of life

12. Center for Reliability Engineering 12

13. Center for Reliability Engineering 13 Integrating software into PRA - Approach Step 2: Specify the functions involved Not all software functions are involved in accident scenarios, i.e, not all software functions are involved in particular scenarios/fault trees or even in the entire realm of possible scenarios/fault trees. To identify the specific functions involved in a scenario, determine the specific input to/output from the software – this will describe one function. A list of possible functions can be found in the requirements. Match the input/output combinations of these functions to the risk model

14. Center for Reliability Engineering 14 Integrating software into PRA – Approach PACS Functional Decomposition

15. Center for Reliability Engineering 15 Integrating software into PRA - Approach Actions and their inputs and outputs

16. Center for Reliability Engineering 16 Integrating software into PRA - Approach Step 3: Modeling software function in ESDs/ETs and Fault trees In the ESDs/ETs, the function of interest is modeled as

17. Center for Reliability Engineering 17 Integrating software into PRA - Approach Step 3: Modeling software function in ESDs/ETs and Fault trees In the fault tree, the function of interest is modeled as

18. Center for Reliability Engineering 18 Integrating software into PRA - ESD

19. Center for Reliability Engineering 19 Integrating software into PRA - Approach Step 4: Input Tree Build the input tree for the particular function involved The input tree is a decomposition of the space of possibilities The input tree is mostly generic for a function. But may VARY due to context.(i.e. probabilities of basic events may vary, certain events may conflict with the rest of the scenario conditions.)

20. Center for Reliability Engineering 20 Integrating software into PRA - Approach Step 4: Input Tree

21. Center for Reliability Engineering 21 Input Fault Tree Input Fault Tree for SW1

22. Center for Reliability Engineering 22 Input Fault Tree

23. Center for Reliability Engineering 23 Integrating software into PRA - Approach Step 5: Quantify the input tree

24. Center for Reliability Engineering 24 Integrating software into PRA - Approach Step 6: Develop and perform software safety tests These tests’ unique objective is to answer the questions contained in the model, i.e. in the MLD, accident scenarios and fault tree. The test is completely automated using Test Generation/test execution tools (TestMaster/WinRunner). The process is as follows: -Build a finite State Machine model of the software by following the software functional decomposition derived from the risk model and the software requirements. -Derive the test profile and output conditions to be quantified from the risk model -Define and run the test cases according to the following test strategy -Analysis consists in computing the probabilities of the different outcomes based on the test data.

25. Center for Reliability Engineering 25 TestMaster Model

26. Center for Reliability Engineering 26 Test Script Example win_activate ("mmount-76.umd.edu - CRT"); start_time1= get_time(); type ("1 "); Check_Message(Message_b,1); type ("0 "); Check_Message(Message_c,1); type ("155721495 "); Check_Message(Message_b,1); type ("0 "); Check_Message(Message_c,1); type("GayyardLupieN "); Check_Message(Message_b,1); type("1 "); end_time1=get_time(); report_msg("Cardtime is "&(end_time1-start_time1)"Seconds"); start_time2= get_time(); Check_Message(Message_d,1); wait( 9); type("4"); Check_Message(Message_e, 1); wait(3); type("5"); Check_Message(Message_f, 1); wait(1); type("1"); Check_Message(Message_g, 1); wait(3); type("9"); end_time2=get_time(); report_msg("PINtime is "&(end_time2-start_time2)"Seconds"); Case_Judge(Message_a,1);

27. Center for Reliability Engineering 27 Test Profile Test Profile for PACS

28. Center for Reliability Engineering 28 Failure modes application Test Case Selection 1.Sample from the profile/input tree to see whether we have a “Normal” or an “Abnormal Input”. 2.If it is a normal input, select randomly from the “Normal Input” domain. 3.If it is an abnormal input, randomly select the failure mode according to the profile/input tree. 4.Then randomly select the “base”value from the “Normal Input” domain and mutate this “base” value using the rules given below:

29. Center for Reliability Engineering 29 Testing Results 200 cases have been tested for SW1 and SW2. 19 cases failed. SW1 fails in only one case (58). Therefore, the point-estimate probability of Card failure is 1/200=0.005. 18 cases failed for SW2. Therefore, the unsafe probability (gate closed) is 18/199 =0.09. Failed cases classification

30. Center for Reliability Engineering 30 Testing Results Card time and Probability

31. Center for Reliability Engineering 31 Testing Results PIN time and Probability

32. Center for Reliability Engineering 32 Test Cases Coverage Input Failure Modes Coverage (SW1) Input Failure Modes Coverage (SW2)

33. Center for Reliability Engineering 33 ESD

34. Center for Reliability Engineering 34 Future Work Represent hardware-related input failure modes in test model Quantification of input fault tree based on field data Output failure modes/Support failure modes Sensitivity Analysis Scalability -Test case generation -Test case execution -Number of test cases for each software component