Anomaly Detection Using Call Stack Information Security Reading Group July 2, 2004 Henry Feng, Oleg Kolesnikov, Prahlad Fogla, Wenke Lee, Weibo Gong Presenter: Jonathan McCune

2 Overview Introduction Related work VtPath Experiments Exploits Comparisons Conclusion

3 Introduction Runtime, training-phase based, anomaly detection system Uses call stack and program counter in addition to system call hooks Novel features based on virtual paths Compared experimentally and analytically with many other approaches

4 Related work Static analysis –Wagner, et al. (abstract stack) –Callgraph Run-time, training based: –This –Sekar, et al. (FSA) (last week’s srg) –N-gram, Var-gram

5 “Intrusion detection via static analysis.” Wagner, et al. Static analysis of program source code NDFA from control-flow graph –Cannot predict branches statically Impossible path problem Abstract stack model –Pushdown automaton Large automata can be resource intensive Claimed zero false positives

6 “Detecting Manipulated Remote Call Streams.” Giffin, et al. Static analysis of binary executables Tied to platform, not programming language Insertion of “null calls” helps impossible path problem Unusual performance analysis

7 “A fast automaton-based method for detecting anomalous program behavior.” Sekar, et al. Compact deterministic FSA from system call analysis of running programs Suffers from false positives, but not as a result of non-determinism Impossible paths not addressed DLLs not adequately addressed

8 Virtual Path (VtPath) Run-time analysis Learns correct behavior via training executions Utilizes return address information Generates abstract path and compares with learned behavior Similar to abstract stack of Wagner, et al., but avoid pushdown automaton

9 Virtual Path Example main() foo1() foo() foo2() foo3() main() fooA() foo() fooB() fooC()

10 VtPath Training Phase Two hash tables: –RA (return address) table –VP (virtual path) table RAs and VPs gradually added during normal program execution Use NULL entries at beginning and end of paths

11 VtPath Online Detection Phase Stack anomaly – if virtual stack list unavailable (common during buffer overflow attacks) Return address anomaly – if virtual stack list {a 0, a 1, … a n } contains a i not in RA table System call anomaly – if a n does not have the correct system call Virtual path anomaly – if virtual path at current system call is not in VP table

12 VtPath – Impossible Path Problem Return addresses are part of virtual path information Modifying return address saved on the stack will cause the program to return to a different location The next system call is likely to trigger a virtual path anomaly

13 VtPath - Implementation Issues Non-standard control flows –Signals (sigreturn system call) Treat each one like a separate program invocation –setjmp()/longjmp() and function pointers Hard to handle statically; handled at runtime if trained properly Dynamically linked libraries –Relative loading positions can change, invalidating PC values from training runs –Use a “block” model during training to capture file name and block length, ignoring start address Block anomaly – block lookup fails because attacker is trying to load a malicious DLL

14 Experiments – VtPath vs FSA Convergence times similar, but –FSA generates more transitions than VtPath (less efficient, less precise) –In practice, multiple levels of DLL functions are called quite frequently – VtPath more effective False Positives nearly identical VtPath executes faster, uses more memory Common exploits detected by both

15 Exploits Authors developed two masked mimicry attacks detectable only by VtPath Impossible Path Execution (IPE) Attack 1 –Exploits parallel structure of privileged / unprivileged code within same function IPE Attack 2: –F() called twice from within same function with different arguments –Overflow local variable to change option passed in

16 Attack 1

17 Attack 2

18 Comparison of SysCall-based Anomaly Detection Schemes 1.State-based / Information captured Data/heap values least useful (transient) Code segment of some value Syscalls and callstack most useful More information = runtime overhead 2.False positives How well is normal behavior captured? Only relevant for dynamic systems Proportional to resolution of program analysis 3.Detection capability More granularity = better detection capability SysCalls from invalid points vs. statistical regularity of training data vs. attacks’ deviation from perceived normal

19 Comparison of SysCall-based Anomaly Detection Schemes 4.Space requirement System call sequences Number of NDFA transitions (Wagner) Transitions in automaton 5.Convergence (training) time Cover most possible states / transitions VtPath requires more data than FSA Static techniques have big advantage here 6.Runtime overheads SysCall interception Hash lookup for valid state / return address / virtual path

20 Conclusions Using call stack (VtPath) has value There is no magic bullet –Everything is “complementary” Impossible path execution (IPE) is an important class of attack Use of FSA to analyze more than two consecutive SysCalls could improve VtPath

21 Discussion!