1. Executive Director and Endowed Chair
CS 5323
Malware Detection
Prof. Ravi Sandhu
Executive Director and Endowed Chair
Lecture 14
© Ravi Sandhu
World-Leading Research with Real-World Impact!

2. Highlights Virus detection is undecidable
Cohen dissertation (1985), paper (1987)
Anti-virus (more generally anti-malware) is a great business model
Need regular updates
Infinite supply of new malware
Malware can be stealthy
Malware can be really stealthy
© Ravi Sandhu
World-Leading Research with Real-World Impact! 2

3. Malware Detection Techniques
Nwokedi Idika and Aditya Mathur, A Survey of Malware Detection Techniques, Purdue University, Feb 2007.
© Ravi Sandhu
World-Leading Research with Real-World Impact! 3

4. Malware Detection Techniques
Misuse
Detection
Behavior-Based
Detection
Nwokedi Idika and Aditya Mathur, A Survey of Malware Detection Techniques, Purdue University, Feb 2007.
© Ravi Sandhu
World-Leading Research with Real-World Impact! 4

5. Signature Limitations
S needs
regular
updates
Nwokedi Idika and Aditya Mathur, A Survey of Malware Detection Techniques, Purdue University, Feb 2007.
© Ravi Sandhu
World-Leading Research with Real-World Impact! 5

6. Anomaly Based Training Phase Detection Phase Infer patterns Infer
specifications
© Ravi Sandhu
World-Leading Research with Real-World Impact! 6

7. Anomaly Based Limitations
Blue area is false positives
If white area extends outside blue area we have false negatives
Nwokedi Idika and Aditya Mathur, A Survey of Malware Detection Techniques, Purdue University, Feb 2007.
© Ravi Sandhu
World-Leading Research with Real-World Impact! 7

8. Stealthy Malware Defeat signature-based detection Encrypted malware
Polymorphic malware
Metamorphic malware
Rootkit can misrepresent the existence or content of executable files
You, I., and Yim, K. Malware obfuscation techniques: A brief survey. IEEE International Conference on Broadband, Wireless Computing, Communication and Applications, Nov 2010, pp
© Ravi Sandhu
World-Leading Research with Real-World Impact! 8

9. Encrypted Malware execute malware Encrypted Main Body Key Decryptor
Cleartext Main Body
propagate
malware
Encrypted Main Body
Key’
Decryptor
© Ravi Sandhu
World-Leading Research with Real-World Impact! 9

10. Encrypted Malware execute malware Encrypted Main Body Key Decryptor
Cleartext Main Body
propagate
malware
reveals
signature
Encrypted Main Body
Key’
Decryptor
© Ravi Sandhu
World-Leading Research with Real-World Impact! 10

11. Polymorphic Malware execute malware Encrypted Main Body Key Decryptor
Cleartext Main Body
propagate
malware
Encrypted Main Body
Key’
Obfuscated Decryptor
© Ravi Sandhu
World-Leading Research with Real-World Impact! 11

12. Polymorphic Malware execute malware Encrypted Main Body Key Decryptor
Cleartext Main Body
propagate
malware no
signature
Encrypted Main Body
Key’
Obfuscated Decryptor
© Ravi Sandhu
World-Leading Research with Real-World Impact! 12

13. Execute in a sandbox and detect the signature after decryption
Polymorphic Malware
execute
malware
Encrypted Main Body
Key
Decryptor
Cleartext Main Body
propagate
malware no
signature
Encrypted Main Body
Key’
Obfuscated Decryptor
Execute in a sandbox and detect the signature after decryption
© Ravi Sandhu
World-Leading Research with Real-World Impact! 13

14. Polymorphic Malware execute malware Encrypted Main Body Key Decryptor
Cleartext Main Body
propagate
malware no
signature
Encrypted Main Body
Key’
Obfuscated Decryptor
Execute in a sandbox and detect the signature after decryption
Mutation Engines automate this construction
© Ravi Sandhu
World-Leading Research with Real-World Impact! 14

15. Metamorphic Malware execute malware Original Main Body
propagate
malware
execute
malware
Obfuscated Main Body
Obfuscated Main Body
propagate
malware
execute
malware
Obfuscated Main Body
Obfuscated Main Body no
signature
© Ravi Sandhu
World-Leading Research with Real-World Impact! 15

16. Obfuscation Techniques
Dead-Code Insertion
Register Reassignment
Subroutine Reordering
Instruction substitution
Code transposition
Code Integration
© Ravi Sandhu
World-Leading Research with Real-World Impact! 16

17. Really Stealthy Malware
Not visible in source code
Reappears in binary code due to malware infected compiler
In theory could reappear in binary code due to other components in binary execution workflow
Loader
Linker OS
BIOS
Ken Thompson. Reflections on trusting trust. Commun. ACM 27, 8 (August 1984),
© Ravi Sandhu
World-Leading Research with Real-World Impact! 17

18. Malicious Compiler Inserts a Backdoor
OS Login module
Malicious Compiler
Binary
Infected Login Binary
© Ravi Sandhu
World-Leading Research with Real-World Impact! 18

19. Malicious Compiler Inserts a Backdoor
OS Login module
Assumption: Malicious behavior cannot be detected in binary, but may be detectable in compiler source
Malicious Compiler
Binary
Infected Login Binary
© Ravi Sandhu
World-Leading Research with Real-World Impact! 19

20. Self-Compiler Compiler source for language L
Compiler binary for language L
Compiler binary for language L
© Ravi Sandhu
World-Leading Research with Real-World Impact! 20

21. Malicious Self-Compiler in Binary and Source
Malicious Compiler source for language L
Compiler binary for language L
Malicious Compiler binary for language L
© Ravi Sandhu
World-Leading Research with Real-World Impact! 21

22. Malicious Self-Compiler in Binary and Source
Source code analysis will reveal malicious behavior
Malicious Compiler source for language L
Compiler binary for language L
Malicious Compiler binary for language L
© Ravi Sandhu
World-Leading Research with Real-World Impact! 22

23. Doubly Malicious Self-Compiler in Binary and Source
Source code analysis will reveal doubly malicious behavior
Doubly Malicious Compiler source for language L
Compiler binary for language L
Doubly Malicious Compiler binary for language L
© Ravi Sandhu
World-Leading Research with Real-World Impact! 23

24. Doubly Malicious Complier Binary Behavior
Compiler source for language L
OS Login module
Doubly Malicious Compiler binary for language L
Doubly Malicious Compiler binary for language L
Infected Login Binary
Doubly Malicious Compiler binary for language L
© Ravi Sandhu
World-Leading Research with Real-World Impact! 24

25. Doubly Malicious Complier Binary Behavior
Compiler source for language L
OS Login module
Doubly Malicious Compiler binary for language L
Doubly Malicious Compiler binary for language L
Infected Login Binary
Doubly Malicious Compiler binary for language L
No trace of malicious behavior in source code
© Ravi Sandhu
World-Leading Research with Real-World Impact! 25

26. Malicious Self-Compiler in Binary but not in Source
Compiler source for language L
Malicious Compiler binary for language L
Malicious Compiler binary for language L
partial
countermeasure
Wheeler, D.A., Countering trusting trust through diverse double-compiling, 21st Annual Computer Security Applications Conference, pp.13-48, 5-9 Dec
© Ravi Sandhu
World-Leading Research with Real-World Impact! 26