1 Software Reliability Assurance for Real-time Systems Joel Henry, Ph.D. University of Montana NASA Software Assurance Symposium September 4, 2002

2 Overview System development Testing problems Solution approach Results and conclusions Practical application

3 Controls devices that control: –Wind generator –Model support –Tunnel atmosphere Emphasizes reliability and safety Utilizes multiple development strategies Based on simple structure System Development Example: Wind Tunnel Software

4 System Development Simple Structure Controlling Computers External Devices ALGORITHM Sample Inputs Run Software Update Outputs

5 Engineer builds graphical model in MATLAB Models enter simulate-debug-simulate- debug phase Engineer auto-generates source code Source code is compiled, linked, and then deployed Hardware/software integration begins System Development

6 Recall simple model –Input variables – sampled over time –Outputs variables – produced over time –Sample time – variable or set frequency Consider test requirements –Input file/matrix –Output file/matrix –Analysis tools Testing Problems Test Size

7 Consider an example –100 input variables –50 output variables –100 millisecond sample time Assume you want to test a one hour operation period –100 inputs*10 per second*3600seconds = 3,600,000 values –50 outputs*10 per second*3600seconds = 1,800,000 values Ignore issues of useful inputs and defect detection Testing Problems Test Size

8 Domain determinants –Input variable – minimum, maximum, and accuracy –Output variable – minimum, maximum, and accuracy Consider test requirements –Input file/matrix with all possible values for input –Output file/matrix much more complex problem Testing Problems Domain Coverage

9 Consider an example for input variable –Input variable for pressure in a tank Min – 0 Max – Accuracy – 3 (decimal places) –1000*1000 = 1,000,000 possible values Ignore issues of legal sequencing and combinations Testing Problems Domain Coverage

10 Automation to: –Generate large input matrices/files –Perform simulation and/or test autogenerated code –Analyze output matrices/files Methods to: –Evaluate domain coverage –Aid debugging –Evaluate results Solution Approach Overview

11 Command and Control Algorithm Generate Tests Simulate Model Test Auto-code Detect Faults Evaluate Results MATLAB/Simulink Environment Source Code Executable Code Verification and Validation Methodology Solution Approach Suite of testing tools

12 Generate Tests Simulate Model Test Auto-code Detect Faults Evaluate Results Suite of testing tools Verification and Validation Methodology Command and Control Algorithm MATLAB/Simulink Environment Model Information Test Data Test Results Executable Code Test Data Test Results Solution Approach

13 Solution Approach Generate Tests Simulate Model Test Auto-code Detect Faults Evaluate Results Suite of testing tools Verification and Validation Methodology Data Graphs Raw Value Files Completeness, MTTF, Reliability File

14 Execute multiple tests –Evaluate testing effectiveness –Track trends in model reliability Automate and evaluate –Specify effectiveness and reliability goals –Evaluate on a per test case basis –Track through testing phase over all tests Results and Conclusions

15 Results and Conclusions Example – Multiple Tests

16 Results and Conclusions Example – Multiple Tests Bucket Coverage (%)

17 Results and Conclusions Example – Multiple Tests

18 Results and Conclusions Example – Multiple Tests

19 Can do domain testing supported with automation Can set quantitative goals Can evaluate progress toward goals Can measure MTTF, domain coverage, confidence percentages, and reliability Can create an organizational history Practical Application What?

20 Invest in automation Integrate domain coverage with application specific testing Establish goals and collect data Calculate MTTF, domain coverage, and reliability Use common sense with quantitative data Practical Application How?

21 Questions and Contact Info Joel Henry –MATT and RATT – links/index.htm MATLAB users -