Software Security Monitors: Theory & Practice David Walker Princeton University (joint work with Lujo Bauer and Jay Ligatti)

July 2003Software Security MonitorsDavid Walker General-purpose Security Monitors A security monitor (program monitor) is a process that runs in parallel with an untrusted application –monitors examine application actions decide to allow/disallow application actions may terminate an application, log application actions, etc. –monitors detect, prevent, and recover from erroneous or malicious behavior at run time –monitors generalize specific enforcement mechanisms such as access control lists, etc.

July 2003Software Security MonitorsDavid Walker What is a security monitor? Monitors analyze & transform untrusted application actions: Application generates actions to be input into monitor Monitor Machine executes actions output by monitor a3a1a2 Input Stream a4a2 Output Stream a1 ……

July 2003Software Security MonitorsDavid Walker Possible Monitor Actions Accept the action Halt the application Suppress (skip) the operation Insert some computation Also: replace results; raise exceptions

July 2003Software Security MonitorsDavid Walker Formalizing security monitors Security monitors => formal automata that transform a stream of program actions Given: a set of possible program actions A Monitors are deterministic state machines: (Q, q 0, T) where Q= state set q 0 = start state T = transition function

July 2003Software Security MonitorsDavid Walker Operational Semantics  Single step (determined by T):  (S in, q)  (S in ’, q’)  Multi-step (reflexive, transitive closure of T):  (S in, q)  (S in’, q)  Output sequence is observable Input sequences are not observable SoSo SoSo

July 2003Software Security MonitorsDavid Walker A Hierarchy of Security Monitors Insert Suppress OK Halt Truncation Suppression Insertion Edit We classify monitors based on their transformational abilities (ie: based on T).

July 2003Software Security MonitorsDavid Walker An Example: E-Banana.com Set of application actions: A = { take(n), // take n bananas pay(n), // pay for n bananas browse,// browse for bananas receipt// commit } Edit Automaton: take(n)pay(n) pay(n);take(n);receipt pay(n)take(n) receipt tntn pnpn tp n init browse start

July 2003Software Security MonitorsDavid Walker Edit Automata  Definition: (Q,q 0,T) –where T = (t,e,i) –State transition function t t : action x state  state –Emission function e e : action x state  {+,-} –Insertion function i i : action x state  action sequence x state

July 2003Software Security MonitorsDavid Walker Edit Automata Operational Semantics –(S, q)  (S’, q’) if S=a;S’ and t(a,q)=q’ and e(a,q)= + –(S, q)  (S’, q’) if S=a;S’ and t(a,q)=q’ and e(a,q)= - –(S, q)  (S, q’) if S=a;S’ and i(a,q)=(S ins, q’) –(S, q)  (empty, q) otherwise a S ins (E-Accept) (E-Suppress) (E-Insert) (E-Halt)

July 2003Software Security MonitorsDavid Walker Security Policies A program execution is a sequence of actions A Security Property is a predicate over executions. Example Properties: –P(S) iff bananas taken equal bananas paid for in S –Access control, resource bounds policies are policies Non-properties: –Relations between different executions of a program –Information-flow policies

July 2003Software Security MonitorsDavid Walker What does it mean to enforce a policy? Principle of Soundness All observable outputs obey the policy  sequences S in.  state q’.  sequence S o 1. (S in, q 0 )  (empty, q’) 2. P(S o ) Principle of Transparency Semantics of executions that already obey policy must be preserved 3. P(S in )  (S in  S o ) SoSo

July 2003Software Security MonitorsDavid Walker Some Useful Equivalences  Remove/Insert unnecessary actions –fclose(f);fclose(f)  fclose(f) Replace a sequence with equivalent actions –socket(S);send(S,m)  socketSend(S,m) Permute independent actions –fopen(f);fopen(g)  fopen(g);fopen(f) Necessary properties: –reflexive, symmetic & transitive –S  S’  P(S)  P(S’)

July 2003Software Security MonitorsDavid Walker E-Banana.com Equivalence Rules: 1) (browse; S)  S 2) (S1; take(n); pay(n); S2)  (S1; pay(n); take(n); S2)

July 2003Software Security MonitorsDavid Walker Conservative Enforcement  Enforcer satisfies Soundness but not necessarily Transparency   properties P. (  sequence S. P(S))  P can be conservatively enforced Conservative

July 2003Software Security MonitorsDavid Walker Effective Enforcement  Enforcer satisfies Soundness and Transparency  provides some flexibility for the enforcer to edit the execution sequence  guarantees the final results of running the application with the monitor are semantically equivalent to running the application without the monitor Conservative Effective

July 2003Software Security MonitorsDavid Walker Precise Enforcement  Definition  Enforcer satisfies Soundness and Transparency  Enforcer must output actions in lock-step with application  Motivation  In some scenarios, operations cannot be delayed without disrupting application semantics Conservative Precise Effective

July 2003Software Security MonitorsDavid Walker What properties can be enforced? The enforceable properties depend upon –the definition of enforcement (conservative, effective, precise) –the class of automaton (truncation, suppression, insertion, edit) –the space of possible input programs if the monitor can assume certain “bad” executions do not occur, it can enforce more properties static program analysis (type systems; proof-carrying code) can constrain program execution in ways useful to run-time monitors

July 2003Software Security MonitorsDavid Walker Effective Enforcement An E-Banana.com policy: –Our edit automaton is an effective enforcer: It satisfies Soundness It satisfies Transparency Proofs are by induction over the possible inputs –Less powerful automata (truncation, suppression and insertion) cannot enforce the E-Banana property Proof by contradiction shows either Soundness or Transparency will be violated browse*; ((take(n);pay(n) | pay(n);take(n)) ; receipt)*

July 2003Software Security MonitorsDavid Walker A Simple Theorem Theorem: Any decideable predicate P on executions is a property that can be effectively enforced by some edit automaton –Proof: construct a transactional edit automaton that suppresses and logs program actions when ¬P(S) and commits (outputs) when P(S), for every initial sequence of actions S in a program execution

July 2003Software Security MonitorsDavid Walker Effectively Enforceable Properties Editing Properties Insertion Properties Suppression Properties Trunc. Prop.

July 2003Software Security MonitorsDavid Walker Summary of theoretical results We have developed the following rigorous methodology for reasoning about run-time security: 1.Define the computational framework using formal operational semantics 2.Define what it means to enforce a policy 3.Prove results about enforceable policies & mechanisms from definitions 1 & 2

July 2003Software Security MonitorsDavid Walker Future Work/Research Ideas Proper definitions of enforcement for infinite execution sequences –Understanding edit automata on infinite sequences Understand transactional policies & develop “transaction automata” –what can they enforce? Incorporate more practical elements into the model –security environment; cryptographic secrets –replacement of results, exceptions and program state

July 2003Software Security MonitorsDavid Walker Polymer, the Language Polymer –A domain-specific language for programming security monitors (ie: edit automata) –Java + a couple of simple extensions: atomic policy definitions encapsulating –a set of security-relevant actions –security state –decision procedure that produces security “suggestions” (halt, suppress action, insert action, etc) compositional policy definitions involving –higher-order policy combinators

July 2003Software Security MonitorsDavid Walker Securing Untrusted Applications Java application policy interface instrumented application describes security- relevant program points contains hooks to call monitor untrusted code separately compiled from policy

July 2003Software Security MonitorsDavid Walker Securing Untrusted Applications Java application policy interface policy implementation instrumented application secure application implements dynamic security policy combines application and policy

July 2003Software Security MonitorsDavid Walker class limitFiles extends Policy { private int openFiles = 0; private int maxOpen = 0; limitFiles(int max) { maxOpen = max; }.... } Atomic Polymer Policy private policy state policy constructor new policy definition extends policy class

July 2003Software Security MonitorsDavid Walker class limitFiles extends Policy { private int openFiles =... private int maxOpen =... public ActionPattern[] actions = new ActionPattern[] {, };.... } Atomic Polymer Policy Continued set of policy- relevant methods

July 2003Software Security MonitorsDavid Walker class limitFiles extends Policy { private int openFiles =... private int maxOpen =... public ActionPattern[] actions =... Suggestion before(Action a) { aswitch (a) { case fileOpen(String s) : if (++openFiles <= maxOpen) return Suggestion.OK(); else return Suggestion.Halt(); case fileClose(File f) :... Atomic Polymer Policy Continued policy behavior

July 2003Software Security MonitorsDavid Walker class limitFiles extends Policy { private int openFiles =... private int maxOpen =... public ActionPattern[] actions =... Suggestion before(Action a) { aswitch (a) { case fileOpen(String s) : if (++openFiles <= maxOpen) return Suggestion.OK(); else return Suggestion.Halt(); case fileClose(File f) :... Atomic Polymer Policy Continued

July 2003Software Security MonitorsDavid Walker class limitFiles extends Policy { public ActionPattern[] actions =... private int openFiles =... private int maxOpen =... Suggestion before(Action a) { aswitch (a) { case fileOpen(String s) : if (++openFiles <= maxOpen) return Suggestion.OK(); else return Suggestion.Halt(); case fileClose(File f) :... Atomic Polymer Policy Continued

July 2003Software Security MonitorsDavid Walker Complex Monitors Combine atomic policies defined over a variety of different resources –eg: sample applet policy file system access control number of files opened restricted network access –no network access after local file is read –communication with applet source only

July 2003Software Security MonitorsDavid Walker Policy Combinators Programmers may write parameterized policy combinators: –And, Or, Forall, Exists, Chinese wall,... s1 s2 AndPolicy: P1P2  s

July 2003Software Security MonitorsDavid Walker Policy Combinators class AndPolicy extends Policy { private Policy p1; private Policy p2; AndPolicy(Policy pol1, Policy pol2) { p1 = pol1; p2 = pol2;... } } first-class policies

July 2003Software Security MonitorsDavid Walker Policy Combinators class AndPolicy extends Policy {... Suggestion before(Action a) { Suggestion s1 = p1.before(a); Suggestion s2 = p2.before(a); if (s1.isOK() && s2.isOK()) return Suggestion.OK(); else... } using suggestions In reality, writing combinators is very tricky

July 2003Software Security MonitorsDavid Walker Summary of Language Design Polymer facilitates the implementation of program monitors by 1.encapsulating all elements (relevant actions, state, decision procedure) of atomic policies in a single place 2.providing mechanisms to compose policies in a well-defined manner 3.coming equipped with a formal semantics –we’re working on it

July 2003Software Security MonitorsDavid Walker Conclusions Technology for securing extensible systems is in high demand –Software security monitors are one part of the solution For more information, see –Edit Automata: Enforcement Mechanisms for Run-time Security Policies. IJIS –Types and effects for non-interfering program monitors. ISSS 2002 & LNCS –More Enforceable Security Policies. FCS –

July 2003Software Security MonitorsDavid Walker End

July 2003Software Security MonitorsDavid Walker Realistic Monitors Protect complex system interfaces –interfaces replicate functionality in many different places –method parameters communicate information in different forms –eg: Java file system interface 9 different methods to open files 4 different methods to close files filename strings, file objects, self used to identify files

July 2003Software Security MonitorsDavid Walker Abstract Action Definitions java.lang.io FileReader(String fileName); FileReader(File file); RandomAccessFile(...);... FileReader.close(); RandomAccessFile.close();... fileOpen(String n); fileClose();

July 2003Software Security MonitorsDavid Walker Abstract Action Definitions class fileOpen extends ActionSig { boolean canMatch(Action a) { aswitch (a) { case FileReader(_) : return true; case RandomAccessFile () : return true;... } String parameter1(Action a) {.... }

July 2003Software Security MonitorsDavid Walker class limitFiles extends Policy {... Suggestion step(Action a) { aswitch (a) { case fileOpen(String s) :... case fileClose() :... } Abstract Action Pattern Matching fileOpen.canMatch(a) fileOpen.parameter1(a)

July 2003Software Security MonitorsDavid Walker Taxonomy of Precisely Enforceable Properties

July 2003Software Security MonitorsDavid Walker Secure Application Java core Polymer language extensions Host System (Java) Program Monitor Definition Untrusted application

July 2003Software Security MonitorsDavid Walker Policy Architecture: Simple Policies Java core Polymer language extensions Host System (Java) Simple Policy Def. system interface

July 2003Software Security MonitorsDavid Walker Policy Architecture: Abstract Actions Java core Polymer language extensions Host System (Java) Abstract Action Def. concrete system interface abstract system interface Simple Policy Def.

July 2003Software Security MonitorsDavid Walker Policy Architecture: Complex Policies Java core Polymer language extensions Host System (Java) Abstract Action Def. Simple Policy Def. Policy Comb. Def. Complex, System-specific Policy concrete system interface abstract system interface

July 2003Software Security MonitorsDavid Walker Securing Extensible Systems Many questions: –Our application requires property X. Can we enforce it precisely or will we have to get by with an approximation? –How do we write down our policy succinctly and unambiguously? –What specific mechanism will we need to enforce our policy? –How do we implement the mechanism?

July 2003Software Security MonitorsDavid Walker Summary A general framework for formal reasoning about security monitors –defined a hierarchy of security monitors –gave meaning to the word “enforceable” –developed rigorous proofs concerning enforceable properties Polymer: A programming language for composing security monitors –techniques for modular monitor design & composition –formal semantics as an extension of FeatherWeight Java