09/18/06 1 Software Security Vulnerability Testing in Hostile Environment Herbert H. Thompson James A. Whittaker Florence E. Mottay

09/18/06 2 Introduction  Functionality and Security Are Often in Contention as Development Goals  Increased Functionality Leads to Decreased Security  Increase Security requires less Software Interaction with It Environment  Security Issues are left under-exercised or not traversed at all due to the difficulty in simulating a hostile environment  Network Failures during aremote transaction  Disk Write errors, Memory Failures  Such Failures Need to be Integrated into the Test Cases

09/18/06 3 Placing a System under Stress  A Security defect, in a general sense, is any attribute of the software that violates policies regarding access to resources  Sensitive information is written out to an unencrypted file  Allows attackers to Deny access to a web server by authorized users  Some Security vulnerabilities can be discovered using conventional testing techniques  Many others security defects only surface when the application is placed under stressed environments  Sources of Stress: load, memory or resource deprivation, etc  When stress is present error handling routines are executed

09/18/06 4 Continuation  Three Basic Approaches to Introduce Environmental failure  Code-Based Fault Injection  External Stress Simulation  Runtime Fault Injection

09/18/06 5 Symptoms of Security Defects  Security failures derived from the interaction with the environment  Reading/Writing Files  Writing to the Registry  Sending information across the network

09/18/06 6 Monitoring for Symptoms  An application interaction with its environment is its most critical behaviour  External Activities can be observed by monitoring the System Calls the Application makes  Application can be monitored by getting in between the application and the environment when a system call is performed  To determine if there exist a security risk a special tool can be used to provide system-level design detail  Holodeck an application tool intercepts systems calls and allows to view the interaction between the application and its environment  Holodeck allows the tester to carefully examine the interaction to find potential points of security break

09/18/06 7 Different types of approaches for security testing  Code-Based Fault Injection  Forces error conditions  Error handling is white box in nature  Involves modifying the source code and hard coding return values  Forces the application to traverse a particular path  Although efficient it has some problems Tester do not always have access to all the parts of source code Even with source code testers might lack expertise of knowledge of the code’s design and structure to effectible implement the branch testing It is time consuming implementing those situations at code level; one instance per situation

09/18/06 8 Continuation Example of Code-based Fault Injection hModule= LoadLibraryEx(TEXT(“msrating.dll”),NULL, LOAD_LIBRARY-AS-DATAFILE); hModule= NULL; // Hard-Coded failure of //LoadLibraryEx

09/18/06 9 Different types of approaches for security testing (cont)  External Stress Simulation  Involves simulating a high volume of activity on the system  Accomplished by using an external application that does not interact directly with the application under test  Or by limiting disk or memory resources Large files Background processes, etc

09/18/06 10 Different types of approaches for security testing (cont)  Runtime Fault Injection  Black box fault injection  System calls are monitored and values returned by these calls are controlled  Faults are simulated without changing the code of the application  The application is placed in a realistically hostile environment by inserting a runtime fault  A single system call can be isolated to manipulate its response during stress situation

09/18/06 11 Conclusion  Error handling routines are subjected to much less testing than functional code  Leaving any portion of the code untested is dangerous for security vulnerabilities  Error handlers are there usually to patched a failure and are thus not constructed with security in mind  Executing test cases in faulty environment should be an essential part of any sound security testing