Reverse Engineering Ian Kayne For School of Computer Science, University of Birmingham 2 nd February 2009

02/09/092 RCE  Reverse (Code) Engineering – “reversing”  What is it?  Why is it done?  Malware research & defence  System interoperability requirements  Review and audit of software/security system  Why is it useful to security specialists?  “Learn the principles”

02/09/093 RCE  Required knowledge/skills (x86)  Platform knowledge – stack, registers etc  “Some” assembly language  C/C++ & as many other languages as possible  Operating system mechanisms  win32api  Toolset (debugger, disasm, hex editor…)  Mindset (patterns, logic)

02/09/094 Complex math RCE  Imagine a strong protection mechanism Asymmetric Shareware-style crippled features Symmetric Checksums

02/09/095 RCE  Reversing demonstration

02/09/096 RCE  After the demonstration, recap: 1.Analysed executable 2.Set breakpoints on likely API calls 3.Traced up the call stack 4.Analysed the code 5.Found the good boy/bad boy “switch” 6.Patched the jump “live” to test 7.Converted RVA to file offset, patched file  1 byte patch

02/09/097 Protection  Imagine a strong protection mechanism again  License key system  CRC  Anti-debugging techniques  Encryption

02/09/098 Protection  Encryption for protection  Data must be decrypted before use  Code must be decrypted before execution  UPX (packer), Armadillo, Themida…  Can be made very hard, but not impossible  Remember the jump loop – EB FE  Generics – break one, break all  Homebrew is risky – “learn the principles”

02/09/099 Protection  Some obfuscation techniques:  Encode obvious “beacon” strings  Avoid win32api/library functions:  bpx MessageboxA  Use alternative functions/mechanisms  E.g.: SetWindowPos instead of ShowWindow  Roll your own api/GUI functions  Can’t break on GetWindowText if you don’t use it!  Hide code within the executable  Self modifying code, PE sections etc

02/09/0910 Protection  Some anti-debugger techniques  Deliberate exceptions (code in SEH)  Self-debugging (can’t “stack” debuggers)  Timers and counters  Alter DR0 – DR7 hardware debug registers  IsDebuggerPresent()  Check for/attack known debugger processes, windows, services, drivers… (Starforce) 

02/09/0911 RCE  Why are these low-level technical techniques important?  “Learn the principles”  Your first job: consultant to betting company about to release online gambling game  The basics:  Internet security  Server security  Data security  But… what about the end-user software?

02/09/0912 Review  Thank you!  Questions  Comments  Items to review  Further study