1. SMT Solvers for Software Security
George Nosenko,
Security researcher at Digital Security

2. Member of DSecRG. System Developer Reverse Engineer
SMT Solvers for Software Security
#whoami
Member of DSecRG.
System Developer
Reverse Engineer
Security Researcher
© 2002—2013, Digital Security

3. Just like the first time using a SMT constraint solver
SMT Solvers in very simple terms
What is a SMT Solver?
Just like the first time using a SMT constraint solver
© 2002—2013, Digital Security

4. “What is the meaning of life?” Solver tries to answer “42”
SMT Solvers in very simple terms
What is a SMT Solver?
Solver is a program
You ask a question
“What is the meaning of life?”
Solver tries to answer
“42”
© 2002—2013, Digital Security

5. Question is a logical formula
SMT Solvers in very simple terms
How can I ask a question?
Question is a logical formula
b+2 = c, f(read(write(a,b,3), c-2)) ≠ f(c-b+1)
SMT-LIB: Language for expressing formulas
All solvers understand this language
© 2002—2013, Digital Security

6. There are many SMT-solvers (over 20)
SMT Solvers in very simple terms
What solver should I choose?
There are many SMT-solvers (over 20)
CVC3, CVC4, STP, Alt-Ergo, Yices, Z3, etc
Z3 is my choice
Efficient SMT solver
Open Source Project:
Python, C/C++, .NET binding
Available online
Support Windows & Linux
© 2002—2013, Digital Security

7. structure of formula declaration precondition postcondition answer
SMT Solvers in very simple terms
SAT or not SAT? Ask a question.
structure of formula
declaration
precondition
postcondition
answer
sat, unsat, unknown
satisfiability
validity
model
(declare-const work Int)
(declare-const sleep Int)
(declare-const fun Int)
(assert (>= work 40))
(assert (>= sleep 42))
(assert (>= fun work))
(assert (= (+ work (+ sleep fun)) 168))
(check-sat)
(get-model)
sat (model (sleep: 42, fun: 63, work 63)
Taint Nobody Got Time for Crash
© 2002—2013, Digital Security

8. Mathematical precision Expressive power Data model
SMT Solvers in very simple terms
Properties of SMT solvers
Mathematical precision
Expressive power
Data model
Efficient implementation
Support Bit-vector & Array
© 2002—2013, Digital Security

9. SMT Solvers for Software Security
© 2002—2013, Digital Security

10. Fuzzing (whitebox or blackbox) Program Verification & Analysis
SMT Solvers for Software Security
Applications
Bug Hunting
Fuzzing (whitebox or blackbox)
Program Verification & Analysis
Exploit Generation
PoC, AEG, APEG
Automate generate payload
Protection Analysis
Obfuscation
Crypto Analysis
Malware Analysis
© 2002—2013, Digital Security

11. SMT Solvers for Software Security
What’s the point?
Idea: convert portions of code into logical formulas, and use SMT solver to prove properties about them
add eax, ebx
xor ebx, ebx
sub ecx, 0x123
setz bl
Is this snippet equivalent to “add eax, ebx”?
sub bl, bl
movzx ebx, bl
add ebx, 0xbbbbbbbb
add eax, ebx
What value must EAX have at the beginning of this snippet in order for EAX to be 0x after the snippet executes?
Taint Nobody Got Time for Crash
© 2002—2013, Digital Security

12. BV Operations in SMT-LIB 2.0
SMT Solvers for Software Security
BV Operations in SMT-LIB 2.0
Core
Bit-wise
Arithmetic
Comparison
Bit-propagation
=/bvcomp
distinct
ite
bvand
bvor
bvxor
bvnot
bvnand
bvnor
bvxnor
bvneg
concat
extract bvshl
bvlshr
bvashr
repeat
zero_extend
sign_extend
rotate_left
rotate_right
bvadd
bvmul
bvudiv
bvurem
bvsub
bvsdiv
bvsrem
bvsmod
bvshl
bvult
bvule
bvugt
bvuge
bvslt
bvsle
bvsgt
bvsge
Slides - SMT Workshop 2013
© 2002—2013, Digital Security

13. Array Operations in SMT-LIB 2.0: select-store axioms
SMT Solvers for Software Security
Array Operations in SMT-LIB 2.0: select-store axioms
Expression (select a i) returns the value stored at position i of the array a;
And (store a i v) returns a new array identical to a, but on position i it contains the value v.
(declare-const x Int)
(declare-const y Int)
(declare-const a1 (Array Int Int))
(assert (= (select a1 x) x))
(assert (= (store a1 x y) a1))
(check-sat)
© 2002—2013, Digital Security

14. BIL code for add %rax, %rbx
SMT Solvers for Software Security
Binary Analysis Platform:
BIL code for add %rax, %rbx
addr "add %rax,%rbx"
label pc_0x0
T_t1:u64 = R_RBX:u64
T_t2:u64 = R_RAX:u64
R_RBX:u64 = R_RBX:u64 + T_t2:u64
R_CF:bool = R_RBX:u64 < T_t1:u64
R_OF:bool = high:bool((T_t1:u64 ^ ~T_t2:u64) & (T_t1:u64 ^ R_RBX:u64))
R_AF:bool = 0x10:u64 == (0x10:u64 & (R_RBX:u64 ^ T_t1:u64^T_t2:u64))
R_PF:bool =
~low:bool(let T_acc:u64 := R_RBX:u64 >> 4:u64 ^ R_RBX:u64 in
let T_acc:u64 := T_acc:u64 >> 2:u64 ^ T_acc:u64 in
T_acc:u64 >> 1:u64 ^ T_acc:u64)
R_SF:bool = high:bool(R_RBX:u64) R_ZF:bool = 0:u64 == R_RBX:u64
© 2002—2013, Digital Security

15. Bug Hunting SMT Solvers for Software Security
© 2002—2013, Digital Security

16. CWE-190,191,192,194,196 May cause: Bypass sanity check Buffer Overflow
Bug Hunting
Vulnerability related with Integer
CWE-190,191,192,194,196
May cause:
Bypass sanity check
Buffer Overflow
Dangling Pointer
Use after free
Application specific
© 2002—2013, Digital Security

17. Integer Overflow in Linux Kernel. CVE-2013-2596
Bug Hunting
Integer Overflow in Linux Kernel. CVE
© 2002—2013, Digital Security

18. Integer Overflow in Linux Kernel. CVE-2013-2596
Bug Hunting
Integer Overflow in Linux Kernel. CVE
static int fb_mmap(struct file *file, struct vm_area_struct * vma){
if (!info) return -ENODEV;
...
off = vma->vm_pgoff << PAGE_SHIFT;
fb = info->fbops;
if (!fb)
return -ENODEV;
/* frame buffer memory */
start = info->fix.smem_start;
len = PAGE_ALIGN((start & ~PAGE_MASK) + info->fix.smem_len);
if (off >= len) {
/* memory mapped io */
off -= len;
start = info->fix.mmio_start;
len = PAGE_ALIGN((start & ~PAGE_MASK) + info->fix.mmio_len); }
mutex_unlock(&info->mm_lock);
start &= PAGE_MASK;
if ((vma->vm_end - vma->vm_start + off) > len)
return -EINVAL;
fb_pgprotect(file, vma, off);
if (io_remap_pfn_range(vma, vma->vm_start, off >> PAGE_SHIFT,
vma->vm_end - vma->vm_start, vma->vm_page_prot))
return -EAGAIN;
return 0;
© 2002—2013, Digital Security

19. How does Motochopper work?
Bug Hunting
How does Motochopper work?
1728 open("/dev/graphics/fb0", O_RDWR) = 6
...
1728 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_SHARED, 6, 0) = 0x400f2000
1728 munmap(0x4015b000, ) = 0
1728 mmap2(NULL, , PROT_READ|PROT_WRITE, MAP_SHARED, 6, 0) = 0x4015b000
1728 munmap(0x4015b000, ) = 0
1728 mmap2(NULL, , PROT_READ|PROT_WRITE, MAP_SHARED, 6, 0) = -1 EINVAL (Invalid
argument)
1728 mmap2(NULL, , PROT_READ|PROT_WRITE, MAP_SHARED, 6, 0x70900) = -1 ENOMEM
1728 mmap2(NULL, , PROT_READ|PROT_WRITE, MAP_SHARED, 6, 0x7b900) = -1 ENOMEM
(Out of memory)
1728 mmap2(NULL, , PROT_READ|PROT_WRITE, MAP_SHARED, 6, 0x7c900) = -1 ENOMEM
1728 mmap2(NULL, , PROT_READ|PROT_WRITE, MAP_SHARED, 6, 0x82900) = 0x4015b000
NAME     mmap2 - map files or devices into memory
#include <sys/mman.h>
void *mmap2(void *addr, size_t length, int prot,
int flags, int fd, off_t pgoffset);
© 2002—2013, Digital Security

20. Integer Overflow in Linux Kernel. CVE-2013-2596
Bug Hunting
Integer Overflow in Linux Kernel. CVE
© 2002—2013, Digital Security

21. Integer Overflow in OpenSSH. CVE-2002-0639
Bug Hunting
Integer Overflow in OpenSSH. CVE
© 2002—2013, Digital Security

22. Integer Overflow in OpenSSH. CVE-2002-0639
Bug Hunting
Integer Overflow in OpenSSH. CVE
input_userauth_info_response(){
...
u_int nresp;
nresp = packet_get_int();
if (nresp > 0) {
response = xmalloc(nresp * sizeof(char*));
for (i = 0; i < nresp; i++)
response[i] = packet_get_string(NULL); }
packet_check_eom();
© 2002—2013, Digital Security

23. Integer Overflow in OpenSSH. CVE-2002-0639
Bug Hunting
Integer Overflow in OpenSSH. CVE
(declare-const sizeof (_ BitVec 32))
(declare-const nresp (_ BitVec 32))
(declare-const mult (_ BitVec 32))
(assert ( = sizeof (_ bv4 32))) ; sizeof (char*) = 4
(assert ( = mult (bvmul nresp sizeof))) ; nresp*sizeof
(assert ( bvugt nresp (_ bv0 32) )) ; nresp > 0
(assert ( bvult mult nresp)) ; nresp*sizeof < nresp
(assert ( = mult (_ bv256 32))) ; nresp*sizeof = 256
(check-sat)
(get-model)
© 2002—2013, Digital Security

24. Verification & Static analyze with SMT
Bug Hunting
Verification & Static analyze with SMT
Single collaborative framework
It’s not heuristic bug-finding
It allows user to manipulate
Functional specification
Prove that source code satisfies specification
Expands with plug-ins
ACSL is a behavioral
specification language
© 2002—2013, Digital Security

25. Jessie is a plug-in for the Frama-C Functional Checking
Bug Hunting
Jessie: verification tools for C programs
Jessie is a plug-in for the Frama-C
Functional Checking
Safety Checking
Memory Safety
Integer Overflow
Checking Termination
© 2002—2013, Digital Security

26. Jessie: Integer Overflow Safety
Bug Hunting
Jessie: Integer Overflow Safety
#pragma JessieTerminationPolicy(user) 
int binary_search(long t[], int n, long v) { 
  int l = 0, u = n-1;
  while (l <= u) { 
    int m = l + (u - l) / 2; //int m = (l + u) / 2; 
    if (t[m] < v) 
      l = m + 1; 
    else if (t[m] > v) 
      u = m - 1; 
    else return m;  
  } 
  return -1;  } 
> frama-c -jessie binary-search.c
© 2002—2013, Digital Security

27. Immunity Debugger & SMT: Infrastructure
Bug Hunting
Immunity Debugger & SMT: Infrastructure
SequenceAnalyzer – Models x86 as operations over a set of SMT primitives.
Solver – Ctypes interface to the CVC3 SMT solver API. Supports a variety of theories including quantifier free, bit-vector arithmetic, linear arithmetic etc.
CodeGraph/PathGenerator – Purely static CFG building and path generation.
PathWalker – SMT based path traversal. Each conditional jump is checked for feasibility and the path discarded if not SAT.
BugChecker – Subclasses provide the check_ins method which will be passed the SMT context representing the current path.
© 2002—2013, Digital Security

28. Immunity Debugger & SMT: !find_int_overwlow.py
Bug Hunting
Immunity Debugger & SMT: !find_int_overwlow.py
© 2002—2013, Digital Security

29. PROTECTION ANALYSIS SMT in protection analysis
© 2002—2013, Digital Security

30. Using SMT to defeat simple hashing algorithms
SMT in protection analysis
Using SMT to defeat simple hashing algorithms
def round_hash(a, b, c, d):
out = [ ]
for i, n in enumerate((a, b, c, d)):
nn = 0
for j in range(32):
nn |= (rotl(n, SCRAMBLE_TABLE[(i << 2)+j]) & 1) << j
nn ^= XOR_TABLE[i]
out.append(nn)
out[0] = rotl(out[0], ROT_TABLE[0])
out[1] = rotl(out[1], ROT_TABLE[1])
out[2] = rotl(out[2], ROT_TABLE[2])
out[3] = rotl(out[2], ROT_TABLE[3])
return out
a ^= c
b ^= d
for i in range(128):
a, b, c, d = round_hash(a, b, c, d)
© 2002—2013, Digital Security

31. Automated KeyGen Generation. Kao’s Toy Project
SMT in protection analysis
Automated KeyGen Generation. Kao’s Toy Project
© 2002—2013, Digital Security

32. Automated KeyGen Generation. Kao’s Toy Project
SMT in protection analysis
Automated KeyGen Generation. Kao’s Toy Project
Lift the checking algorithm to BIL
./toil -binrange ~/toyproject.exe 0x x o checkUnlockCode.il
Convert BIL to single static assignment form (SSA), unroll loop
./iltrans -il checkUnlockCode.il -to-ssa -simp-ssa -to-cfg -unroll 31 -rm-cycles \
-rm-indirect-ast -to-ast -normalize-mem -flatten-mem -pp-ast checkUnlockUnroll.il
egrep -v '^cjmp.*$' checkUnlockUnroll.il > checkUnlockUnrollOpt.il
Convert BIL to SMT-formula
./topredicate -il checkUnlockUnrollOpt.il -noopt -solver z3 -stp-out checkLoop.smt
line 18: assert --> define-fun alg () (Array (_ BitVec 32) (_ BitVec 8))
line 921: false --> ?mem_array_83_670
© 2002—2013, Digital Security

33. Create precondition and postcondition
SMT in protection analysis
Create precondition and postcondition
© 2002—2013, Digital Security

34. Automatically craft an input that redirects control flow
AEG
Automatic Exploit Generation
Automatically craft an input that redirects control flow
Loosely defined as “Given a program and a vulnerability, automatically craft an input that redirects control flow to malicious code”
Automated Payload Creation
© 2002—2013, Digital Security

35. Get the trace to vulnerable code
AEG
Automatically craft an input that hijacks control flow
Get the trace to vulnerable code
Convert the trace into set of constraints
Freach
Generate the set of conditions that make code exploitable
Fexploit = Cval U Caddr
Solve (Freach U Fexploit)
SMT-solver defines required input
© 2002—2013, Digital Security

36. Automatically craft an input that hijacks control flow
AEG
Automatically craft an input that hijacks control flow
Freach =
{ t0= eax + ebx, zf ==1 }
Cval =
{ eax = 0xdeadbeef }
Caddr =
{ t1= ebp + 4, t1 = ebp +ecx }
Fexploit = Cval U Caddr
© 2002—2013, Digital Security

37. Automatically craft an input that hijacks control flow
AEG
Automatically craft an input that hijacks control flow
Freach =
{t0 = eax + ebx, zf == 1}
Cval =
{ eax = 0xdeadbeef }
Caddr =
{t1 = ebp + 4, t1 = ebp + ecx}
Fexploit = Cval U Caddr
Input ={eax = 0xdeadbeef, ebx = 0x , ecx = 4}
(declare-const t0 (_ BitVec 32))
(declare-const t1 (_ BitVec 32))
(declare-const eax (_ BitVec 32))
(declare-const ebx (_ BitVec 32))
(declare-const ecx (_ BitVec 32))
(declare-const ebp (_ BitVec 32))
; Freach = {zf = 1, t0 = eax + ebx}
(assert (= t0 (bvadd eax ebx))) ; t0 = eax + ebx
(assert (= t0 #x )) ; zf = 1
; Cval = { eax = 0xDEADBEEF }
(assert (= eax #xdeadbeef)) ; eax = 0xDEADBEEF
; Caddr = { t1 = ebp + 4, t1 = ebp + ecx}
(assert
(and
(= t1 (bvadd ebp #x )) ; t1 = ebp + 4
(= t1 (bvadd ebp ecx))) ; t2 = ebp + ecx )
sat (model
(define-fun ecx () (_ BitVec 32) #x )
(define-fun eax () (_ BitVec 32) #xdeadbeef)
(define-fun ebx () (_ BitVec 32) #x )
© 2002—2013, Digital Security

38. Automated Payload Creation
Automate Generation Payload
Automated Payload Creation
Data Execution Prevention (DEP)
Windows 8 ROP mitigation enforces policies on who/where can call VirtualAlloc() or VirtualProtect() to enable memory executable at run-time
IOS already totally forbid code injection: Writable pages have NX permission & only signed pages are executable
Return Oriented Programming
fun at first time, then hurt
hundreds and thousands of ROP-gadgets
“bad characters"
find a suitable gadget can be difficult
research efforts aimed at solving the problem of automatic generation ROP-chains
© 2002—2013, Digital Security

39. An interesting example from 0verckl0ck
Automate Generation Payload
An interesting example from 0verckl0ck
Given:
we can write into eax, but only ASCII printable char
we have ROP-gadgets like these:
add eax, 0xc9f4458b; add eax, 0xdeadbeef;
add eax, 0x0fcf; add eax, 0x13b2;
add eax, 0x1337; add eax, 0x42;
Goal:
make eax = 0xb00bdead
determine the initial value eax
find the minimum sequence of calls gadgets
© 2002—2013, Digital Security

40. source: http://rise4fun.com/Z3Py/OrzP
Automate Generation Payload
An interesting example from 0verckl0ck
source:
assert( init_eax + g1*0xc9f4458b + g2*0xdeadbeef + g3*0x0fcf + g4*0x13b2 + g5*0x g6*0x42 = 0xb00bdead)
assert(ascii_printable( init_eax ) )
sum (g1,g2,g3,g4,g5,g6) --> min
answer:
0x522e707c + 3*0xc9f4458b + 8*0x13b2 = 0xb00bdead
© 2002—2013, Digital Security

41. Finding gadgets with specific samntics
Automate Generation Payload
Finding gadgets with specific samntics
© 2002—2013, Digital Security

42. Automate Generation Payload
Immunity Debugger: !find_gadget
This script looks for a sequence that satisfies the constraints we specify
© 2002—2013, Digital Security

43. Automate Generation Payload
OptiROP
© 2002—2013, Digital Security

44. Automate Generation Payload
OptiROP
© 2002—2013, Digital Security

45. ROPC: https://github.com/pakt/ropc
Automate Generation Payload
ROPC:
© 2002—2013, Digital Security

46. ROPC : Type of gadgets that ROPC find &use
Automate Generation Payload
ROPC : Type of gadgets that ROPC find &use
Name
Input
Parameters
Semantic Definition
NopG _
nop
LoadConstG
OutReg, Value
OutReg  Value
MoveRegG
InReg, OutReg
OutReg  InReg
ArithmeticG
InReg1, InReg2, OutReg op
OutReg <- InReg1 op InReg2
StoreMemG
AddrReg, InReg
# Bytes, Offset
M[AddrReg+Offset]<-InReg
LoadMemReg
AddrReg, OutReg
OutRegM[AddrReg+Offset]
ArithmeticStoreG
InReg, AddrReg
# Bytes, Offset, op
M[AddrReg+Offset] op  InReg
ArithmeticLoadG
OutReg, AddrReg
OutReg op  M[AddrReg+Offset
© 2002—2013, Digital Security

47. ROPC-LLVM: https://github.com/programa-stic/ropc-llvm
Automate Generation Payload
ROPC-LLVM:
© 2002—2013, Digital Security

48. Questions ? SMT Solvers for Software Security
© 2002—2013, Digital Security