1. Tom Gullion SRMV Toolsmith tgullion@msisinc.com (608) 315-2780 Executing UML Models as Verification Oracles

2. Overview Charter –Execute UML models as oracles of behavior exhibited by software under test Challenge –Testing facilities for flight control software and related domains are expensive in terms of cost and resources and access –Verification teams often find it difficult to get adequate access to those testing facilities Goal –Reanimating test logs from the test labs against System Reference Models –That is, verify the SRM against test logs. And vice versa!

3. The Challenge Testing facilities for flight control software and related domains are expensive –in terms of cost and resources and access Verification teams often find it difficult to get adequate access to those testing facilities Access to logs from test runs is common Access to software simulations may happen Access to live test rigs could possibly happen… How can we leverage this test data in Verification?

4. Test Data Reanimation Data from a test log is used as a stream of input events SRM is implemented/extended as a test oracle Behavior of the model, under that stimulus stream, is compared with the behavior logged for the software under test

5. Basic Reanimation Approach test definition comand/event stack Reanimate SRM data converted to stream of events Behavior of the model, under that stimulus stream, is compared with the behavior logged for the software under test SRM as test oracle exec

6. Test log expectations Logs from testing / simulation facilities will have timeframes, command names, parameter data and other metadata We won’t be interested in all of it (at least, at first) We may even prefer to extract relevant data from multiple sources for our own scenarios (e.g., testing across component boundaries)

7. Case study: Ejection Seat Behavior

8. Test scenario Example: the testing facility ran a simulation of a flight where the crew ejected

9. Reanimate Ejection Seat Events Focus only on ejection seat commands in test logs Configure model runner environment Execute commands against SRM to reanimate the events

10. Reanimation details COMMENT START SCRIPT Flight with Eject 3330 EJECTION_SEAT arm 6322 EJECTION_SEAT enable 6323 EJECTION_SEAT eject 6326 EJECTION_SEAT separate 6330 EJECTION_SEAT releaseMain COMMENT END SCRIPT FRAME: 3330 EJECTION SEAT PHASE CHANGE TO Armed FRAME: 6322 EJECTION SEAT PHASE CHANGE TO Ready FRAME: 6323 EJECTION SEAT PHASE CHANGE TO Ejected FRAME: 6326 EJECTION SEAT PHASE CHANGE TO Separated FRAME: 6330 EJECTION SEAT PHASE CHANGE TO MainChute Scenario complete: SUCCESS Input events (Frame, Command, Parameter(s)) Results playback against SRM model element(s)

11. Ejection seat scenario results Use reanimation results to verify logged events match SRM modeled behavior But this scenario only verifies a single slice of the SRM. Let’s expand the reanimation to include flight modes…

12. Reanimating multiple statemachines 3320 FLIGHT_MODE preflight 3330 EJECTION_SEAT arm 4532 FLIGHT_MODE taxi 5832 FLIGHT_MODE takeoff 6322 EJECTION_SEAT enable 6323 EJECTION_SEAT eject 6326 EJECTION_SEAT separate 6330 EJECTION_SEAT releaseMain COMMENT END SCRIPT Input events (Frame, Command, Parameter(s)) Results playback against SRM model element(s) FRAME: 3320 FLIGHT MODE CHANGE TO PreFlight FRAME: 3330 EJECTION SEAT PHASE CHANGE TO Armed FRAME: 4532 FLIGHT MODE CHANGE TO Taxi FRAME: 5832 FLIGHT MODE CHANGE TO Flight FRAME: 6322 EJECTION SEAT PHASE CHANGE TO Ready FRAME: 6323 EJECTION SEAT PHASE CHANGE TO Ejected FRAME: 6326 EJECTION SEAT PHASE CHANGE TO Separated FRAME: 6330 EJECTION SEAT PHASE CHANGE TO MainChute Scenario complete: SUCCESS

13. Flight with Ejection seat scenario results Now we’ve verified logged events match SRM modeled behavior in two statemachines This scenario only verifies proper event sequencing within its scope. But we’re not checking preconditions and other rules. Extend interfaces defined in SRM to implement rules…

14. Reanimating the SRM 3320 FLIGHT_MODE preflight 3330 EJECTION_SEAT arm 4532 FLIGHT_MODE taxi 5832 FLIGHT_MODE takeoff 6322 EJECTION_SEAT enable 6323 EJECTION_SEAT eject 6326 EJECTION_SEAT separate 6330 EJECTION_SEAT releaseMain COMMENT END SCRIPT Input events (Frame, Command, Parameter(s)) Results playback against SRM model element(s) FRAME: 3320 FLIGHT MODE CHANGE TO PreFlight FRAME: 3330 FlightController: Ejection seat armed EJECTION SEAT PHASE CHANGE TO Armed FRAME: 4532 FLIGHT MODE CHANGE TO Taxi FRAME: 5832 FLIGHT MODE CHANGE TO Flight FRAME: 6322 EJECTION SEAT PHASE CHANGE TO Ready FRAME: 6323 FlightController: Canopy released FlightController: Ejection seat positioned FlightController: Ejection seat ejected FlightController: main chute deployed EJECTION SEAT PHASE CHANGE TO Ejected FRAME: 6326 EJECTION SEAT PHASE CHANGE TO Separated FRAME: 6330 EJECTION SEAT PHASE CHANGE TO MainChute Scenario complete: SUCCESS

15. Complete scenario results Now we’ve verified logged events match SRM modeled behavior in two statemachines as well as a flight controller class From this framework, it is now possible to add in telemetry data and other command data. This structure is setup for reuse across other reanimations.

16. live data feed Live Test Rig Extensible Reanimation Approach comand/event stack Pluggable input data types; Pluggable exec environments Software Simulator exec Reanimate SRM adverse condition data test definition(s)

17. Extensible Reanimation Approach Pluggable input streams allow for future use of Other test logs (related tests, adverse conditions, etc.) "live" data feeds the use of accredited simulations as live or stale data sources Some or all these data sources can be used without changing the overall approach

18. Conclusions This approach is appropriate for complex collections of analysis cases –Simple analysis cases probably do not require reanimation We are building part of a larger construct –Analyze adverse conditions against specific tests logs or accredited simulations –Enables use/reuse of SRM without extensive modification –Enables traceability from reanimation result back through SRM to originating requirements –Allow distributed machines to play different roles (data source vs data analysis) –Flexibility for machines (real and virtual) to interface with existing infrastructure (IV&V not required to manage all machines) This approach embraces the existing overall framework and is designed to scale

19. Applying the 3 Questions What is the system supposed to do? –Verified by success and failure scenarios What is the system not supposed to do? –Verified by success and failure scenarios What does the system do under adverse conditions? –Verified by adverse conditions test data –Can predict possible failure scenarios that don’t seem adequately handled, propose test scenarios for dev team

20. Acknowledgements Thanks to NASA IV&V for the opportunity to present these concepts Special thanks to Frank Huy, Bill Stanton, Todd Gauer, Karl Frank, Steve Driskell, Dan Sivertson, Tom Hempler and Jeff Micke for technical contributions and assistance

21. Questions? / Comments? Thank you! Tom Gullion tgullion@msisinc.com (608) 315-2780