1. IV&V T ESTING S TRATEGIES FOR I NDEPENDENT V ERIFICATION OF NASA M ISSION S OFTWARE I MPLEMENTATION 3 rd Annual Workshop on Independent Validation and Verification September 17, 2009

2. O BJECTIVES Vision for an IV&V Test Environment Value of an IV&V Test Environment IV&V Test Environment Concept of Operations Characteristics of an IV&V test environment Software IV&V Testing Strategies Current status of the IV&V Test Environment 2

3. V ISION : P ROBLEM S TATEMENT The current method of analyzing source code in its static state doesn’t adequately evaluate the behavior of the code under operational and environmental conditions System interactions, dependability, quality attributes, among other properties are hard to evaluate with static analysis 3

4. V ISION : S COPE Develop, maintain and operate an adaptable test environment for the IV&V Program that enables the dynamic analysis of software behavior for multiple NASA missions Compliment the inspection of the development test design and subsequent test results as well as manual and automatic source code analysis in its static state These methods have been effective at uncovering evidence that the implementation answers the three questions 4

5. V ISION : V ALUE Uncover issues that would not be found by static code analysis, manual requirements based code inspection, or test results inspection Allow for the creation of goodness statements that will add more value to the project and provide a greater confidence in the assurance of the software 5

6. C ONCEPT OF O PERATIONS Intended to be common for for any NASA mission software IV&V project For a given NASA mission, develop test plan using the System Reference Model (SRM) For a given NASA mission flight software test article: Execute it Stimulate it according to IV&V test plan Observe its behavior 6

7. A PPROACHES TO E XECUTING SUT Mimic the intended target environment Acquire COTS emulators for the target processors Develop simulators Leverege simulators already created by the developer Use the development test environment Connect remotely Install our test environment at the development test site Don’t execute Analyze the developer’s test results 7

8. A PPROACHES TO S TIMULATING SUT Generate test scenarios from the system behavior models for a NASA mission found in theSRM Scope the type of scenarios based on the results from validating the development test design Install and use developers simulation models in our IV&V test environment Create our own simulation models to stimulate the software Don’t stimulate Developer incorporates IV&V test suite into their test plan and IV&V analyzes the test results 8

9. A PPROACHES TO O BSERVING SUT Manually inspect the test results Monitor the behavior of the SUT using formal mechanisms such as state chart assertions created by the SRM and validation teams On-line during test Off-line after test, based on test results 9

10. T ESTING S TRATEGIES A strategy is a plan of action designed to achieve a particular goal 1 The expected goal of an analyst using the system: To determine whether or not the implemented system behaves as we expect it to, doesn’t behave like we don’t what it to, and to ensure it behaves as it should under adverse conditions. Possible strategies Improvement to the development test design Analyze the development test results Supplement the development test with our test suite Test independent of the developer 10 1: From the Oxford English Dictionary

11. 11 Develop Software Test Suite Analyze Development Test Results Configure IV&V Test Environment Integrate IV&V Test Environment with Development Facility Execute Software Test Suite at Development Test Facility Execute Software Test Suite at IV&V Facility Analyze IV&V Test Results Limitations do not exist in developer’s test designLimitations exist in developer’s test design Development test facility is not available for IV&VDevelopment test facility is available for IV&V

12. T EST E NVIRONMENT C HARACTERISTICS Usable – common user interface across all IV&V projects Adaptable – allow for the reconfiguration of testing resources for a individual IV&V projects Extensible – allow for the future integration of new simulation tools, simulation models, and development test articles Accessible Interface with test monitors (e.g assertion state chart repositories) Interface with development laboratories Portable – for the possibility of testing at development site 12

13. S TATUS Vision and Concept of Operations documents are baselined Took inventory of available hardware and software resources at the facility Created first draft of cost estimate and acquiring vendor quotes Working on a proof of concept to explore the process of dynamically testing flight software as a form of software implementation verification. The prototype will be used to improve our Concept of Operations and validate our Vision. Using uncompiled mission code that was already provided by developer 13

14. C HALLENGES Collecting all the information and materials needed to develop the test suite and configure the test environment in a timely manner. This may take multiple iterations Configuring the test environment with the necessary simulators and emulators to successfully run and stimulate the software under test 14