Software Fault Injection Kalynnda Berens Science Applications International Corporation NASA Glenn Research Center

Software Fault Injection NASA Glenn Research Center2 What is Software Fault Injection? A testing technique that aids in understanding how software behaves when stressed in unusual ways. A product-based assurance technique. Variations in the technique allow it to be applied to many types of software and for different purposes.

Software Fault Injection NASA Glenn Research Center3 How does SFI work? Legal permutations or faults are input at interfaces (external and/or internal). Outputs show whether the injected fault propagates through the software. Requires instrumentation (software code) to observe the propagation process.

Software Fault Injection NASA Glenn Research Center4 Uses for Software Fault Injection Finding defects in software Robustness Testing COTS Validation/Determining failure modes Safety Verification Security Assessment Software Testability Analysis

Software Fault Injection NASA Glenn Research Center5 SFI Examples Operating System Validation Ballista (CM) – Linux and VxWorks robustness WindowsNT Network Security NCSA httpd server Safety Advanced Automatic Train Control system Magneto Stereoaxis System SFI can be used with or without source code

Software Fault Injection NASA Glenn Research Center6 SFI without Source Code Create software wrapper for COTS functions and other interfaces “Trick” OS to call wrapper functions first Software under test usually run in debug mode Wrapper can be used Pass through for baselining response Call alternative function Call original function but change result

Software Fault Injection NASA Glenn Research Center7 SFI wrapper operations Application Wrapper OS or Hardware Pass through wrapper: Call alternative function: Call original function but change result: Application Wrapper OS or Hardware New Function

Software Fault Injection NASA Glenn Research Center8 Center Initiative on SFI Can SFI be used by an Independent V&V engineer? Is SFI a useful and cost-effective technique for NASA? Are the errors and problems found of sufficient severity or abundance? Are the costs of applying the technique reasonable for the number/severity of errors found? Is SFI a good tool for safer software?

Software Fault Injection NASA Glenn Research Center9 Methodology Determine scope Select projects Metrics Perform SFI on projects Create Test Plan (prototype due 1st quarter, FY02) Lessons Learned

Software Fault Injection NASA Glenn Research Center10 Determine Scope Why narrow the scope? SFI is a collection of related techniques Comparison across projects requires using one technique for all Why no source/interfaces technique chosen IV&V perspective (cost effective) “Outside” events or system limitations trigger many errors Interfaces selected COTS software Hardware User input Communications medium

Software Fault Injection NASA Glenn Research Center11 Project Selection Potential Projects CM-2 Tempest Web Server (VxWorks and Java) MDCA, FPP, SAMS, others Selection Criteria Selection difficulties Project support not free Contracted software not accessible Final Choice

Software Fault Injection NASA Glenn Research Center12 Metrics Time spent per task Familiarization, researching errors, instrumenting software, testing Subjective “effort” scale per task Software project metrics SLOC, #classes/modules, complexity, interface information Fault Injection metrics #faults, #failures, #faults no effect/correctly handled

Software Fault Injection NASA Glenn Research Center13 SFI Process Obtain Tempest software (completed) Obtain access to VxWorks (completed) ***Overcome compatibility problems Determine all interfaces to test Select errors to inject Create necessary wrappers for SFI Record test procedure and results

Software Fault Injection NASA Glenn Research Center14 Tempest Interfaces VxWorks OS Task creation and control functions C/C++ language functions File system functions Networking functions Outside world Requests from external sources Standard HTML, built-in functions Tempest (VxWorks version) can execute OS functions

Software Fault Injection NASA Glenn Research Center15 Example Injection Errors OS errors Memory allocation failures File errors (corrupted, not found) Single task abort, hang External World errors Invalid request Too many requests Requests too frequent

Software Fault Injection NASA Glenn Research Center16 Test Plan How to perform software fault injection on “generic” software Steps prior to actual testing Method of determining errors to inject Procedure for performing the test Appendices of lessons learned, example faults, other guidance

Software Fault Injection NASA Glenn Research Center17 Difficulties Encountered Tempest documentation limited VxWorks simulator does not support networking Cost of hardware and full VxWorks not within the budget Attempt to “fake” networking unsuccessful

Software Fault Injection NASA Glenn Research Center18 Status and Future Work VxWorks incompatibilities not easily overcome Shift to Java version of Tempest for now Test VxWorks version of Tempest on actual hardware (if possible) or alternate operating system (Linux, uClinux, eCos) If funding continues, test on actual flight experiment (CM-2).