Fuzzing with complexities Vishwas Sharma

Introduction We all have been a witness to major threats in the past years and I guess no one could forget names like ‘Conficker’ (1), ‘Stuxnet’ (2) and ‘Aurora Project’ (3). All these malware had a unique delivery system which was based on exploiting the host operating system and further talking control of the OS. These threats are always there and only thing we expect to achieve is that, we find vulnerability before a bad guy do and do something about it. Software companies spend a lot of their time and money in making their product more stable, more reliable and more secure. Vista Microsoft has made sure that functions like strcpy, sprintf etc. are eliminated at the Software development lifecycle (SDL)

Introduction In fact all major vendors have realized the importance of having a secure SDL and importance of testing in their product. Google and Firefox have a policy of rewarding any researcher who comes up with a bug or a resulting exploit. Figure 1: Microsoft Simplified SDL (4)

Software Testing Software testing is any activity aimed at evaluating an attribute or capability of a program or system and determining that it meets its required results. Unlike most physical systems, most of the defects in software are design errors, not manufacturing defects

Code Coverage Code coverage is one of the most important metrics used to decide on the completeness of the test cases. This metrics gives us the relationship between the test conducted and instructions executed with the application.

Code Coverage Of course this metrics can be further broken down into more detailed metrics Function coverage - Has each function (or subroutine) in the program been called? Statement coverage - Has each node in the program been executed? Decision coverage - Has every edge in the program been executed? For instance, have the requirements of each branch of each control structure (such as in IF and CASE statements) been met as well as not met? Condition coverage - Has each Boolean sub-expression evaluated both to true and false? Condition coverage - Both decision and condition coverage should be satisfied.

Code Coverage An example of Code coverage

Code Coverage Test needed to find bugs Tests needed for coverage Shows that even on a good coverage some bugs would still be left alone

BlackBox Testing No knowledge of the inner working of the software, neither of the protocol or kind of input expected, this situation is rightly named as Black-box Testing Black box testing Expected InputBoundary valuesIllegal values

Whitebox Testing Information on internal data structure and algorithms is completely shared between the product development team and the tester’s team Information can be used to test API’s, Code Coverage, fault injection, Mutation of testing and many more. White box testing Statement TestingDecision TestingConditional Testing

Fuzzing The first person credit of working and formulating this technique is Barton Miller and his students from University of Wisconsin-Madison in 1989 In simple words it is the technique in which repeated invalid or mutated or malformed input is supplied to application with only intention to find bugs the application It is observed that fuzzing is most effective against application developed in C/C++, these languages make the programmer responsible for memory management whereas managed code i.e. developed in C#, Java etc. would yield bugs of a very different class

Fuzzing TermDefinition Dumb fuzzing Corruption of data packets randomly without awareness of data structure. Smart fuzzing Corruption of data packets with awareness of the data structure, such as encodings (for example, base-64 encoding) and relations (checksums, bits indicating the presence of some fields, fields indicating offsets or lengths of other fields). Black-box fuzzing Sending of malformed data without actual verification of which code paths were hit and which were not. White-box fuzzing Sending of malformed data with verification that all target code paths were hit—modifying software configuration and the fuzzed data to traverse all data validations in the tested code. Generation Generation of fuzzed data automatically—not basing on any previous input. Mutation Corruption of valid data according to defect patterns, to produce fuzzed data. Mutation template Well-formed buffer that represents an equivalence class of the input. The fuzzer takes the mutation template as an input—producing a fuzzed buffer to be sent to the tested software. Code coverageTechnology (such as that which is bundled in Microsoft Visual Studio 2005) that allows inspection of which code paths were executed during testing. This is useful for verification of test effectiveness and improvement of test coverage.

Fuzzing Important distinction between Fuzzing and other testing activity. This distinction is the intent. A testing team knows a lot about the program and basically test that whether a program is behaving as it is supposed to behave where as a security researcher only care that his fuzzer crashes your tested application.

Fuzzer I would like to make note of two python based fuzzing framework available in the open source community that I use most extensively. PeachFuzzer - Peach is a SmartFuzzer that is capable of performing both generation and mutation based fuzzing (10). Sulley - Sulley is a fuzzer development and fuzz testing framework consisting of multiple extensible components. Sulley (IMHO) exceeds the capabilities of most previously published fuzzing technologies, commercial and public domain

Fuzzer I would like to make note of two python based fuzzing framework available in the open source community that I use most extensively. PeachFuzzer - Peach is a SmartFuzzer that is capable of performing both generation and mutation based fuzzing. Sulley - Sulley is a fuzzer development and fuzz testing framework consisting of multiple extensible components. Sulley exceeds the capabilities of most previously published fuzzing technologies, commercial and public domain Peach Fuzzing Platform

Fuzzer Peach is been improved day in and day out and it is the only other open source fuzzer that is maintained apart from Metasploit fuzzer. Peach is written as primary data fuzzer, but as it open source it can be extended to secondary and even nth-class fuzzer. Peach fuzzer is also used by adobe in its testing of Adobe reader Sulley is not maintained but is as good as you can get when it comes to generation based fuzzing Collection of fuzzers

Complexity “Software bugs will almost always exist in any software module with moderate size: not because programmers are careless or irresponsible, but because the complexity of software is generally intractable -- and humans have only limited ability to manage complexity. It is also true that for any complex systems, design defects can never be completely ruled out” - Jiantao Pan, Carnegie Mellon University In many of the fuzzers it is observed that test cases produced fails to achieve the basic packet sanitation test of the target application if the fuzzer is has improper understanding of the input type and structure

Complexity A study done by Microsoft on a 450 lines of code and then testing it with various fuzz combinations to see the effective results that was produced is shown below : TechniqueEffortCode coverageDefects found Black box + Dumb10 min50%25% White box + Dumb30 min80%50% Black box + Smart2 hr80%50% White box + Smart2.5 hr99%100% Analysis based on Effort in producing fuzzer and defects found correlated with kind of fuzzer

Packets An example of ASCII based packet (irc) There are few other examples quite popularly known eg. – HTML – CSS – FTP – And many more WHO #UnrealIRCd JOIN #Support PONG irc.fyremoon.net irc2.dynam.ac STATS u

Binary based Packets ZIP local file header OffsetBytesDescription [5] [5] 04Local file header signature = 0x04034b50 42Version needed to extract (minimum) 62General purpose bit flag 82Compression method 102File last modification time 122File last modification date 144CRC Compressed size 224Uncompressed size 262File name length (n) 282Extra field length (m) 30nFile name 30+nmExtra field But what happens when the formats no longer sticks to one data format? What happens when our data switches from one set of data format like ASCII to binary and then binary to ASCII again and to add a cherry on top sections are encoded differently even the ascii portion can be encoded and even imported from other binary or ASCII based formats

Example of one such format %PDF-1.5 %µµµµ 1 0 obj >>> endobj 2 0 obj > endobj 3 0 obj >/ProcSet[/PDF/Text/ImageB/ImageC/ImageI] >>/MediaBox[ ] /Contents 4 0 R/Group >/Tabs/S/StructParents 0>> endobj 4 0 obj > stream xœ\[Û¸’~ÿ Gû ­ˆÝ€AI&9ÈbgÏì&»ó0³jµÜÖÄq÷Hêtz~ýÖ…”HY”ÅÁœ´e²X$ëúUÉÑë_£Ÿ~zýË»?GÉõuôöçwÑÛÏ/_¼þ "™Å‰Ž>ï^¾QÿQ.ãDê(×i,Šèó×—/’8)%±øÿ*K£înññýóå‹ßJ’¤¼ÎáŸ2¿¾*ñßwþ¿ÅÛë+ú>¹¾*ðsÉŸeÆæ¹—ñ¿ÅþÞŽ+Òë+‘à Example of one such complex formats ie. PDF We see these being used in every day applications like office documents, Adobe PDF, SMB protocols and more. One cannot try to randomly fuzz these files as they have pretty good input validation modules which prevent any dumb attempt to fuzz them

What we know so far What we have gathered until here is summarized here as we move ahead you will find answers to these problems Complications Code coverage is cannot be used as a metrics while testing test type of input streams Data format consistency i.e. Either ASCII or Binary is not maintained Multiple encoded levels exists which can be different within the same file Multiple data format either binary or ASCII are embedded within these input stream

Some answers Code Coverage fails for these applications Protocol awareness can be used as once we have all the information of a protocol that we could have, we can intuitively say that the packet which contains the most number of tags or objects would require more code to be covered with that module. Now this could be said that we cannot guarantee the code coverage still because if we do not find a packet that contains all the tags or object Testing all cases in one go was never the idea but multiple tests covering every tag is what will be fruitful. Data format inconsistency One can easy write a fuzzer of either and ASCII based packet or for binary based packet. But when these formats get together in a packet, it becomes unnaturally difficult to write one. The solution lays in visualizing and breaking problem in parts which we most comfortable in. We can use the separate out the data generation capability from both ASCII and Binary format. Remember here I have trying to separate out these capabilities not necessarily for fuzzing.

Some answers Multiple Files Embedded in a single packets With separating of types we can further separate to a secondary level data production module ie. A different level of generating data. What this means is that if a PDF file if we have a font and image embedded inside the file we can actually write a different fuzzer for font and for an image and combines each of these result with the PDF files in the manner similar to multiple encoding level problem. Multiple Encoding levels As we have separated ASCII with Binary in the same format one can further add custom encoding in each packet as one like. They will all fall back together when we combine them later. See the case study for more clarification. For example in a PDF file if we have a multiple font embedded inside the file we can make use of different encoders for each such font as each is generated separately

Strategy Now is the right time to talk about the strategy that I have used when fuzzing one such format, PDF. You will find different definition of these terms, but this is what I understand out of them. This process is typically described in the terms of system under test and called for directed area with the system, where as in my study I have taken it out of box and placed these conditions on Data packet itself.

Attack point selection Not to fuzzTo Fuzz Not to Fuzz To Fuzz Attack Point Selection The attach point selection is a simple process in which I have tried to specify a specific point within the packet which needs to be tested. Now selection of these points depends a lot upon some gathered intelligence of the system, including pervious vulnerabilities. As this eliminates a few attack point as they have already been attacked before. For example if working on a simple PDF file which contain a U3D file which is known to previously cause a vulnerability in Adobe reader one can say this format is previously been tested primarily (after looking at the vulnerability) so a lot more efforts would be required in finding a vulnerability next time. One can focus his time and energy in finding other routes into the application which has still not been tested by security researchers.

Directed Fuzzing Not to fuzz Not to Fuzz To Fuzz Adobe Flash Player Multiple Tag JPEG Parsing Remote Code Execution Vulnerability -- Vulnerability Details: This vulnerability allows remote attackers to execute arbitrary code on vulnerable installations of Adobe Flash Player. User interaction is required in that a target must visit a malicious website. The specific flaw exists within the code for parsing embedded image data within SWF files. The DefineBits tag and several of its variations are prone to a parsing issue while handling JPEG data. Specifically, the vulnerability is due to decompression routines that do not validate image dimensions sufficiently before performing operations on heap memory. An attacker can exploit this vulnerability to execute arbitrary code under the context of the user running the browser. Figure 7: An example of Vulnerability disclosure Whenever a vulnerability is released it is released with a very few information. One such disclosure example would be.

Figure 7: An example of Vulnerability disclosure Demo CVE Integer overflow in CoolType.dll in Adobe Reader and 9.3.3, and Acrobat 9.3.3, allows remote attackers to execute arbitrary code via a TrueType font with a large maxCompositePoints value in a Maximum Profile (maxp) table.