Forensic Analysis of Toolkit-Generated Malicious Programs Yasmine Kandissounon TSYS School of Computer Science Columbus State University 2009 ACM Mid-Southeast Conference Gatlinburg, Tennessee November 12-13, 2009

State of the Threat (Jan – Jun 2009) Microsoft Security Intelligence Report : – 115,854,807 infections in first half 2009 – 94,985,967 infections in second half 2008  An increase of about 22% (2008) AVTest Labs – 15,000 to 20,000 new specimens analyzed each day. (4 times as many as in 2006, 15 times as many as in 2005) (ESET ) Talented teams of programmers Automated Malware Creation: – W32.Evol, W32.Simile, W32.NGVCK, W32.VCL, etc.

What Does the AV Industry Need? Automation – (Szor 2005) The need for analysis by humans is a major bottleneck! Ability to quickly and accurately detect new malware. – (Team Cymru, 2008) 1000 new samples submitted, only 37% detected by commercial AV products! Badly needs “good” Generic Signatures – (Kaspersky Lab 2008) Windows Explorer was flagged as malicious – AVIEN’s HARLEY (On average, current detection(using generic signatures) rates are no better than 70%-80%)

Our Problem: Engine Generated Malware ENGINE VIRUS SAMPLE Variant1 Variant2 Variant nVariant3 Network Too many signatures challenge the detector Malware detector Signature Database (Virus Definitions) In Out

Solution: Use Engine Signature ENGINE VIRUS SAMPLE Variant1 Variant2 Variant nVariant3 Internet Use one small piece of info about the engine to detect all of the variants. Malware detector Engine Signature In Out Network

MALWARE GENERATION AS A HIDDEN MARKOV MODEL NOP * CALL JMP MOV Transition Matrix = Engine Signature (Choice of relevant instructions = 5 most frequent instructions) NOP MOV PUSH CALL JMP * NOP MOV PUSH CALL JMP * PUSH MOV JNZ MOV PUSH MOV NOP MOV NOP ADD JMP MOV NOP PUSH JZ PUSH MOV CALL MOV CALL SUB MOV PUSH MOV CALL POP MOV Transition matrix is n+1 by n+1 and represents the engine  Problem: Find smallest n that will induce best accuracy MOV * MOV PUSH MOV NOP MOV NOP * JMP MOV NOP PUSH * PUSH MOV CALL MOV CALL * MOV PUSH MOV CALL POP MOV Take only the n most frequent instructions, for some n

Subjects and Preparation 100 malware samples of W32.Evol and W32.Simile (Metamorphic viruses) 100 malware samples generated by NGVCK 100 malware samples generated by VCL – Source: benign samples – Source: sourceforge.net, download.com, installation of Windows Vista.

Classification Method For each sample – Identify a training subset of size 30 – Compute the transition matrix for each trainer – Take the average of these. – This average is the engine signature for the sample. For each instance not used for training – Compute the transition matrix of the instance – Compute the Euclidian Distance between the instance and each of the engine signatures generated in the above stage – The signature that is found to be closest to this instance’s transition matrix is declared to be the instances’ family. If there are ties, choose one at random.

Average Matrix Classifier (1 st Order Markov Chain) Results: RELEVANT INSTRUCTIONSMISCLASSIFICATIONS % % 108% 1511%

K-Nearest Neighbor Classification Concept Results Limitations: – Time – Space

Discussion Average Matrix vs Knn – Time and space efficiency – Accuracy – Behavioral characteristics not taken into account RIs: ideal RI in the vicinity of 20

Conclusion and Further Work Conclusion – Good Accuracy (8% misclassifications) – Small Signature (11 by 11 matrix) – Fast Detection (12 min for 150 tests) Further Work – 2 nd order – Work with more samples – Work with other families of malware – Different ways of choosing the relevant instructions – Try a different distance measure

References dyn/content/article/2008/03/19/AR html J euralNets.html. Last retrieved April 12, 2009 M.R. Chouchane. “Approximate Detection of Machine- morphed Malicious Programs”. Ph.D. Dissertation. (2008) Using Engine Signature to Detect Metamorphic Malware. Chouchane and Lakhotia, WORM 2006.

References Ivan Krsul and Eugene H. Spafford, Authorship Analysis: Identifying the Author of a Program. Computers & Security (1997) Peter Szor, The Art of Computer Virus Research and Defense. (Chapter 7) 2005 Wing Wong and Mark Stamp, Hunting for Metamorphic Engines. J Comput Virol (2006) last retrieved April 12, 2009