1. October 20-23rd, 2015 FEEBO: A Framework for Empirical Evaluation of Malware Detection Resilience Against Behavior Obfuscation Sebastian Banescu Tobias Wüchner Aleieldin Salem Marius Guggenmos Martín Ochoa Alexander Pretschner Technische Universität München Chair XXII: Software Engineering

2. October 20-23rd, 2015 Background Obfuscation: Transform program, make it harder to understand, preserve functionality  Syntactic obfuscation: change binary code layout (e.g. instruction order)  Behavior obfuscation: change observable behavior (e.g. syscall trace) 2 Original Program Obfuscation Tool Obfuscated Program

3. October 20-23rd, 2015 Motivation & Problem Statement Cat & mouse game between malware creators & malware detection community: 1.Signature-based detection uses static binary 2.Malware developers generally bypass signature-based detection via syntactic obfuscation (many tools available) 3.Behavior-based detection uses system call trace and counters syntactic obfuscation 4.Hypothesis: Malware developers bypass behavior-based detection via behavior obfuscation Problem: There is no automated tool that performs behavior obfuscation 3

4. October 20-23rd, 2015 Objectives 1.Build FEEBO, a behavior obfuscation tool for malware binaries 2.Build behavior-based detection approach according to state-of-the-art 3.Empirically assess impact of behavior obfuscation on detection approach 4

5. October 20-23rd, 2015 Design and Implementation of Obfuscation Tool Obfuscation generally requires access to source code  Not available for real-world binaries + machine code packed Our approach: obfuscation via dynamic binary instrumentation 1.Intercept system calls 2.Apply obfuscation transformation 3.Continue execution 5

6. October 20-23rd, 2015 FEEBO tool for Intel PIN:  Applicable to any x86 binary (even obfuscated)  No debugging symbols needed for binary  No static modification of binaries FEEBO supported obfuscation transformations:  System call insertion  System call reordering  System call substitution URL: https://www22.in.tum.de/feebo/ FEEBO FEEBO – Framework for Empirical Evaluation of Behavior Obfuscation 6

7. October 20-23rd, 2015 FEEBO Supported Obfuscation Transformations Let [R, C, L, W, W, L] be the orginal log trace consisting of system calls 1.System call insertion: breaks n-gram patterns by adding new syscalls  [R, L, C, W, L, W, C, R, W, L] 2.System call reordering: breaks n-gram patterns by re-arranging the syscalls  [C, R, W, W, L, L] 3.System call substitution: breaks n-gram patterns by replacing syscalls with equivalent syscalls  [R, P, L, M, W, N] 7

8. October 20-23rd, 2015 Evaluation 1.Feasibility Evaluation:  Which FEEBO transformations produce running malware?  Which FEEBO transformations make the executables crash? 2.Effectiveness Evaluation:  Can malware transformed with FEEBO bypass behavior-based detection?  How does accuracy of behavior-based detection look after obfuscation? 8

9. October 20-23rd, 2015 Feasibility Evaluation Public malware dataset (Malicia dataset) comprises 11,688 samples Randomly sampled 100 samples spread across 16 malware families Generated 100 x 300 = 30.000 sample logs 9 FEEBO Original Malware Sample Malware Sample Obfuscated with Syscall insertion Malware Sample Obfuscated with Syscall reordering Malware Sample Obfuscated with Syscall substitution x 100 3% crashed 54% crashed 14% crashed

10. October 20-23rd, 2015 Measuring Actual Degree of Obfuscation Input parameters of FEEBO include:  Probability that one or more system calls are inserted/substituted  The lower & upper limits of inserted system calls Obfuscated malware samples output by FEEBO with the same input parameters may have different degrees of obfuscation Measure actual degree of obfuscation via Levenshtein distance of logs: 10

11. October 20-23rd, 2015 Effectiveness Evaluation: Classifier for Malware Detection 11 SVM-Raw SVM-Alpha

12. October 20-23rd, 2015 Effectiveness Evaluation: Generating Obfuscated Malware Logs Public malware dataset (Malicia dataset) comprises 11,688 samples Generated 200 x 11,688 = 2,337,600 sample logs Goodware dataset 730 samples (Windows utilities and cnet.com top rated) Problems:  Datasets for training classifiers are highly unbalanced  Training the SVM classifier for 11,688 samples takes ~3,5 hours 12 FEEBO Simulator Original Malware Sample Malware Sample Obfuscated with Syscall insertion Malware Sample Obfuscated with Syscall substitution x 100 Sample 1000 logs

13. October 20-23rd, 2015 Training only with Logs from Original Malware Samples 13

14. October 20-23rd, 2015 Effectiveness Evaluation: Training only with Original Malware Trained SVM-alpha with:  Malware set: 1000 original samples  Goodware set: 730 samples Tested with 200 x 1000 samples obfuscated with syscall insertion 14

15. October 20-23rd, 2015 Effectiveness Evaluation: Training only with Original Malware Trained SVM-raw with:  Malware set: 1000 original samples  Goodware set: 730 samples Tested with 200 x 1000 samples obfuscated with syscall insertion 15

16. October 20-23rd, 2015 Effectiveness Evaluation: Training only with Original Malware Trained SVM-alpha with:  Malware set: 1000 original samples  Goodware set: 730 samples Tested with 200 x 1000 samples obfuscated with syscall substitution 16

17. October 20-23rd, 2015 Effectiveness Evaluation: Training only with Original Malware Trained SVM-raw with:  Malware set: 1000 original samples  Goodware set: 730 samples Tested with 200 x 1000 samples obfuscated with syscall substitution 17

18. October 20-23rd, 2015 Training with Logs from Both Original & Obfuscated Malware Samples 18

19. October 20-23rd, 2015 Effectiveness Evaluation: Training with Original & Obfuscated Malware Trained SVM-alpha with:  Malware set: 1000 original samples + 1000 obfuscated samples  Goodware set: 730 samples Tested with 200 x 1000 samples obfuscated with syscall insertion 19

20. October 20-23rd, 2015 Effectiveness Evaluation: Training with Original & Obfuscated Malware Trained SVM-raw with:  Malware set: 1000 original samples + 1000 obfuscated samples  Goodware set: 730 samples Tested with 200 x 1000 samples obfuscated with syscall insertion 20

21. October 20-23rd, 2015 Effectiveness Evaluation: Training with Original & Obfuscated Malware Trained SVM-alpha with:  Malware set: 1000 original samples + 1000 obfuscated samples  Goodware set: 730 samples Tested with 200 x 1000 samples obfuscated with syscall substitution 21

22. October 20-23rd, 2015 Effectiveness Evaluation: Training with Original & Obfuscated Malware Trained SVM-raw with:  Malware set: 1000 original samples + 1000 obfuscated samples  Goodware set: 730 samples Tested with 200 x 1000 samples obfuscated with syscall substitution 22

23. October 20-23rd, 2015 Conclusions Built FEEBO a behavior obfuscation tool which offers system call insertion, reordering and substitution FEEBO can obfuscate real-world malware samples with impact on accuracy of state-of-the-art behavior-based detection approaches FEEBO is a fair way to compare different behavior-detection approaches Future work:  Improve the reordering transformation  Extend semantic substitution transformation 23

24. October 20-23rd, 2015 Thank you for your attention! Contact: banescu@in.tum.de 24

25. October 20-23rd, 2015 Backup slides 25

26. October 20-23rd, 2015 26