1. Yet Another Heapspray Detector Danny Kovach Raytheon SI

2. Introduction Our main purpose is to detect malware.

3. Introduction Currently we monitor an application in a VM for such behavior as: –Loading drivers –Creating executable files –Network activity Heap sprays are very hard to detect.

4. What is a heapspray? Technique used to put executable code onto the heap. Consists of –NOP sled –Shellcode Goal: direct execution flow to the NOP sled; shellcode.

5. How to detect a heapspray? Nozzle [1] BuBBle [3] Entropy

6. Idea! Treat byte values on the heap as a random variable and do math!

7. Assumptions Bytes on a normal heap should be randomly distributed (white noise) Fourier transform of white noise has constant magnitude.

8. Visualizing the Heap (normal program operation)

11. Fourier Transforms of the Heap (normal program operation)

14. Visualizing the Heap (heap spray)

17. Fourier Transforms of the Heap (heap spray)

20. Problem:

21. Low hanging fruit?

22. More Analysis Used the open source tool RapidMiner Started by making a decision tree

23. Results

24. 100% accurate for all our test cases. Rushed into production (without further testing). FAIL!

25. Next attempt: Statistics Assume that distribution of bytes is Gaussian

26. Statistics for normal heap About 40 counts > 1 standard deviation Actual measurement: 20 – 30

27. Statistics for Heapspray NOP slide altered distribution Typically 2 – 8 > 1 standard deviation

28. Advantages of a Statistical Approach Easy to code Friendly to system resources More general than hard coded approach Theoretically sound

29. Results Out of over 500 files tested, we had 100% success. 0 false positives 0 false negatives

30. How to defeat Write shellcode so as to minimally alter normal distribution. Most likely will leave some signature. Invites cat and mouse game.

31. References 1.http://research.microsoft.com/en-us/projects/nozzle/http://research.microsoft.com/en-us/projects/nozzle/ 2.http://en.wikipedia.org/wiki/Heap_sprayinghttp://en.wikipedia.org/wiki/Heap_spraying 3.https://lirias.kuleuven.be/bitstream/123456789/265421/ 1/fulltext.pdfhttps://lirias.kuleuven.be/bitstream/123456789/265421/ 1/fulltext.pdf 4.http://www.mathnstuff.com/math/spoken/here/2class/9 0/normal.htmhttp://www.mathnstuff.com/math/spoken/here/2class/9 0/normal.htm