1. Presenter: Shant Mandossian EFFECTIVE TESTING OF HEALTHCARE SIMULATION SOFTWARE

2. SIMULATION SOFTWARE Modeled after real world systems. Allows users to observe an operation without actually performing it. Used in initial system designs in order to optimize component selection.

3. SIMULATION SOFTWARE Two Different Types Discrete Event Simulation Continuous Simulation

4. SIMULATION SOFTWARE CONTINUOUS SIMULATION

6. SIMULATION SOFTWARE DISCRETE EVENT SIMULATION

8. Input Agent Resources Non-Agent Resources Artifacts Oracle Resource Manager Output Significant Components

9. SIMULATION SOFTWARE TESTING COMMON METHODS Research in simulation software focuses on validation Insufficient investigation into testing the software itself.

10. SIMULATION SOFTWARE TESTING WHY IS IT DIFFICULT? Simulation software exists to find the answers in the first place. No test oracle exists because exact simulation results are simply not known. Subtle errors in implementation can greatly affect the accuracy of the simulation.

11. SIMULATION SOFTWARE TESTING WHY IS IT DIFFICULT? Simulations consist of many interacting components Each step or event can take a random amount of time to complete.

12. SIMULATION SOFTWARE TESTING N-VERSION PROGRAMMING Early approach to testing software without a test oracle Independent programming teams develop an application from the same specifications. Identical sets of input are fed into each version of the program. Results are compared in an effort to detect defects.

13. SIMULATION SOFTWARE TESTING METAMORPHIC TESTING

14. The Bad Like most tests, this method only shows us if defects exist. Not that the software is defect-free. The Good Provides a way of revealing potential defects without the help of a test oracle

15. SIMULATION SOFTWARE HEALTHCARE Healthcare professionals rely on simulation software

16. SIMULATION SOFTWARE HEALTHCARE Software is used to simulate elements of the human body for purposes of training and decision making.

17. SIMULATION SOFTWARE HEALTHCARE Simulations of a specific organ like the heart. Simulations of an entire physiological system.

18. SIMULATION SOFTWARE HEALTHCARE Emergency Department

19. SIMULATION SOFTWARE TESTING EMERGENCY DEPARTMENT SIMULATOR Process Patient arrives Patient is seen by a triage-nurse Patient registers with the clerk Clerk collects information puts in the patient record If a bed is available, patient goes to treatment area If all beds are occupied, patients waits in the waiting room until a bed is cleared Patient is taken to a free bed Patient is seen by a nurse Doctor takes assessment of the patient Assessment may result in tests Once tests are done, doctor admits or discharges the patient Nurse fills out paperwork.

20. SIMULATION SOFTWARE TESTING EMERGENCY DEPARTMENT SIMULATOR Agent Resources Clerk Triage Nurse Nurse Doctor Non-Agent Resources Beds

21. SIMULATION SOFTWARE TESTING EMERGENCY DEPARTMENT SIMULATOR Finding Metamorphic Properties All resources used in simulation Doctors, Nurses, beds, etc… Input given to the simulator Simulation instructors Program source code

22. SIMULATION SOFTWARE TESTING EMERGENCY DEPARTMENT SIMULATOR Potential Bottleneck Patients wait until a bed is available. Run multiple simulations with varying numbers of beds, but keep all other resources constant. Plot average Length of Stay (LOS) for each run-through of the simulator to see if results are as expected.

23. SIMULATION SOFTWARE TESTING EMERGENCY DEPARTMENT SIMULATOR Results show that LOS improves with more beds, but the amount of improvement decays.

24. SIMULATION SOFTWARE TESTING EMERGENCY DEPARTMENT SIMULATOR Plotting the utilization of doctors over the number of beds shows that on average, utilization increases but also with a decay

25. SIMULATION SOFTWARE TESTING EMERGENCY DEPARTMENT SIMULATOR Both Length of Stay (LOS) of patients vs. number of beds and Doctor utilization vs number of beds, displayed expected results. New test case with unexpected results. Five patients arrive, 6 minutes apart. Resources Available One nurse, doctor, clerk, and triage nurse. Three beds. Results Average LOS for each patient = 217.4

26. SIMULATION SOFTWARE TESTING EMERGENCY DEPARTMENT SIMULATOR

29. With One Nurse Patient 4 was ready to be discharged by the doctor before patient 5 was ready to be seen by the same doctor. With Two Nurses Patient 5 was ready to be seen by the doctor before patient 4 was ready to be discharged. This is unexpected behavior.

30. SIMULATION SOFTWARE TESTING TESTING THE TEST METHOD Testing the effectiveness of Metamorphic Testing Mutation testing is used. Defects purposely injected into the source code. 25 out of 104 generated mutants were used in the test Many mutants were excluded for various reasons. Caused obvious errors and crashes Were caught by exception handling.

31. SIMULATION SOFTWARE TESTING TESTING THE TEST METHOD Metamorphic testing successfully caught all mutation defects.

32. SIMULATION SOFTWARE TESTING OTHER METHODS Other testing methods (instead of metamorphic) Assertion checking.

33. SIMULATION SOFTWARE TESTING EMERGENCY DEPARTMENT SIMULATOR Assertion checking Wasn’t used here in comparison Expected to perform less successfully. Example Off by 1 error for range [a, b-1] instead of [a, b] Will never produce an out-of-range result, causing the assertion test to miss the defect. Metamorphic testing can take on the statistical mean and variance over many run-throughs and find an inconsistency.

34. SIMULATION SOFTWARE TESTING CONCLUSION AND SUMMARY Simulation Software Discrete Event Simulation Testing Simulation Software Relation to healthcare software Difficult to test Metamorphic Testing Mutation testing on Metamorphic testing Assertion Checking