A Binary Technology for COTS Software Integrity Anant Agarwal Richard Schooler InCert Software

Active-X, Java, and increasing reliance on “commercial off-the-shelf” technology help infiltrators make unknowing accomplices of legitimate users CACM, July ‘99 Durst, Champion, Witten, Miller, Spagnuolo, AFRL, on mission critical computer systems

The Mission Critical Environment Output Input Operating System SAP COTS Binary The development environment The deployment environment COTS Binary

“Hostility” in The Mission Critical Environment COTS Binary Input Output Operating System SAP Virus attacks Bad data (remember Y2K) Invalid/Null arguments Untrusted third party data C2 security requirements Untrusted third party software

Objective COTS Binary Input Output Operating System SAP To improve the integrity of the deployment environment for COTS software in the presence of "hostilities"

Some Current Approaches Applied at source level during development e.g., type based safety; work of Lee et al. Applied at link time with special object formats e.g., software fault isolation; work of Pandey et al. Applied through interpretor e.g., safe Java interpreters Applied during program execution e.g., middleware This approach works with COTS packages, PC, Mainframe, etc. -- hence it is a widely adopted commercial approach Modify OS, like middleware integrated into OS e.g., wrap OS layer to intercept calls for services

COTS Integrity Approach through Binary Augmentation The development environment The deployment environment COTS New Missing source Legacy COTS Binary COTS New Missing source Legacy BAS Recovery logs Access constraints Logging requirements Argument ranges Rare code execution (defaults for fault tol., policy specs for security) COTS New Missing source Legacy User Specified

The Current Commercial Solution: Middleware COTS Binary Input Output Operating System SAP Middleware Slow Maintenance nightmare Cannot handle untrusted software Cannot deal with Viruses Cannot improve fault tolerance of COTS package itself

Why other solutions (source, link, interpreter) do not often apply to COTS software Source Out- sourced devlp. Missing source COTS bin obj pkgs Packaged COTS executable Legacy objects COTS New Missing source Legacy Consider this Vendor’s development environment

Why other solutions (source, link, interpreter) do not often apply to COTS software Vendor wants to supply generic COTS (try convincing m to customize word for you) User wants to customize security policy Impossible to take a security approach involving “writing-all-code-afresh” Near impossible for user to arm-twist vendor into adding security features (note the difficulty being faced by ARM like apps mgmt standards)

Needed: An Approach to Integrity that Works with COTS binaries, even legacy codes Allows a user to establish desired security levels and to some extent modify policy on the fly Works completely at the user’s deployment site Our’s is a systems level approach that attempts to satisfy the above goals

COTS Integrity Approach through Binary Augmentation The development environment The deployment environment COTS New Missing source Legacy COTS Binary COTS New Missing source Legacy BAS Recovery logs Access constraints Logging requirements Argument ranges Rare code execution (defaults for fault tol., policy specs for security) COTS New Missing source Legacy User Specified

Three Major Components in the Prototype, Three Major Tasks Core technology for customizable agent insertion into PC/NT, PC/Linux Anomaly detection Rapid recovery technology

Three Major Components in the Prototype Core technology for customizable agent insertion Develop basic instrumentation technology for NT Hard-to-find relocations -- use incremental control and dataflow analysis to create control-flow graph Dynamic methods from binary translation to augment static analysis Evaluate on-the-fly binary rewriting versus table driven approaches for augmenting agent function Optimize performance of on-the-fly instrumentation, and that of the instrumented COTS binary during its production run

Three Major Components in the Prototype Anomaly detection Several defaults -- open to other ideas Rare code exec (application path signatures, and test path signatures if available) address ranges, null ptrs, historical value ranges etc. User specified -- need help here We want to leverage an existing spec Training phase to relate user function to code Develop training instrumentation agents e.g., fire bad transaction, agents record code path, arguments, etc, and cause alert in production run

Three Major Components in the Prototype Rapid recovery technology One form of agents that records where program has been, so trace can enable fast recovery from crash or alert following an actual or suspected attack Record program path in circular trace buffers Maintain time stamp info at entry and exit points into program to enable stitching together multithreaded traces Record values upon user request (as a side effect, can create logs for various security requirements, e.g., C2) Cause alarm into theater wide console (e.g. unicenter TNG) upon alert/crash, and write buffers to file Implement mechanisms for trace compression and execution speed

User Model Import COTS software package User specifies policies and constraints, but fault tolerance mechanisms, logging rapid recovery and other useful defaults exist Our binary augmentation compiler will stitch in appropriate code (agents) into binary; hard to analyze code to be augmented with on-the-fly code rewriting Run COTS software. Stitched in code or agents perform logging, checks, and apply constraints On a crash or user-specified alert the dynamic sequencing of instructions and data values can be retrieved from trace logs, and will enable rapid recovery

Measures of Success We will build a prototype system, work with real users, and measure Core technology for agent insertion into binary: Can we handle all binaries, even dusty decks? Performance degradation to be under 1 percent Anomaly detection What fraction of injected problems can we detect - automatically - with user spec Rapid recovery technology Performance degradation to be under 1 percent Can we cut recovery time significantly? We will measure recovery time with and without As a bonus, can we catch problems before system| goes down?

Some Challenges Core technology for agent insertion into binary Creation of a general instruction framework for multiple ISAs -- how much work is it to go to another platform? How to deal with unknown relocations, e.g., for dusty decks - an integrated static and dynamic method? Anomaly detection How to relate user function to binary checks - learning phase to obtain execution path signatures? How to minimize runtime overhead - use compiler optimization technologies for agents? (e.g., steal registers, inline code, sampling, multilevel checks) Rapid recovery technology Runtime overhead issue - especially for data values use dataflow analysis and offline simulation to obtain intermediate data values? Logfile size issue - use logfile compression methods?

What do we Need? A partner from a defense site with PC/NTs and ES390’s, so we can get help on perceived sources of attack, test our planned system in a realistic environment and obtain accurate measures of performance, generality, and ease of use. Suggestions for a theater wide “console,” so we can integrate our alerts within that standard API. e.g., any defense site using Unicenter TNG? Suggestions on a spec for user-specified security Is there a “attack scenarios” benchmark set? If not, we must create one.

Technology Transition Build prototype and make available to a defense installation Integrate our system with its alerts into a commercial off the shelf theater wide console Our COTS based approach will make our technology interesting to both military and commercial IT organizations with mission critical enterprise software systems (remember, eTrade’s disastrous shutdown!), so we will look move experimental system to a commercial product

Summary A systems approach to COTS Integrity User specifies policies and constraints, but useful defaults exist Approach based on binary agent insertion Integrity technology will work even with legacy binaries, requires no new formats, or language modifications