Presentation is loading. Please wait.

Presentation is loading. Please wait.

TestIstanbul Conferences 2012 Parallel Execution of Fuzzing Test Suites Study of maximum throughput, resource consumption and bottlenecks for fast-speed.

Similar presentations

Presentation on theme: "TestIstanbul Conferences 2012 Parallel Execution of Fuzzing Test Suites Study of maximum throughput, resource consumption and bottlenecks for fast-speed."— Presentation transcript:

1 TestIstanbul Conferences 2012 Parallel Execution of Fuzzing Test Suites Study of maximum throughput, resource consumption and bottlenecks for fast-speed fuzzing

2 TestIstanbul Conferences 2012 About Ari Takanen The Past: Researcher and Lecturer – – University of Oulu – OUSPG/PROTOS research group – Software Quality related lectures The Present: Entrepreneur and Evangelist – 2001-today – CTO of Codenomicon – Evangelist: 10 conference talks every year – Author of two books: VoIP Security Fuzzing

3 TestIstanbul Conferences 2012 Agenda Intro: – Security Testing – Product Security Fuzzing – Overview – Techniques – Test Coverage – Fuzzing Scalability and Performance Performance case study – Multi-threaded execution – Test execution optimization – Command-line use – Results Conclusions

4 TestIstanbul Conferences 2012 What is Testing - Three Flavors Feature/Conformance tests validate positive requirements with use cases and test cases Performance/Load testing repeats one of the feature tests to validate performance limits Robustness/Fuzz testing uses misuse cases to verify software reliability with unexpected inputs

5 TestIstanbul Conferences 2012 Security Vulnerability = Just A Bug

6 TestIstanbul Conferences 2012 Codenomicon Labs Test Results Taken from:

7 TestIstanbul Conferences 2012 Fuzz Test Effectiveness against WiFi

8 TestIstanbul Conferences 2012 Some Helpful Definitions Vulnerability – a weakness in software, a bug Threat/Attack – exploit/worm/virus against a specific vulnerability Protocol Modeling – Technique for explaining interface message sequences and message structures Fuzzing – process and technique for security testing Anomaly – abnormal or unexpected input Failure – crash, busy-loop, memory corruption, or other indication of a bug in software

9 TestIstanbul Conferences 2012 Fuzzing In Short Fuzzing means crash-testing Also called: – Negative testing – Robustness testing – Grammar testing Based on sending systematically broken (rarely random) inputs to a software, in order to crash it We will ignore random mutator fuzzers for now Two techniques of smart model-based fuzzers: – Template-based – Specification-based

10 TestIstanbul Conferences 2012 Model Based Fuzzing Techniques Template Based Fuzzing (capture-replay, mutation) – Quality of tests is based on the used seed and modeling technique – Very quick to develop, but slow to run – Editing requires deep protocol know-how – Good for testing around known vulnerabilities Specification Based Fuzzing (generational, model-based) – Full test coverage – Always repeatable – Short test cycle, more optimized tests – Easy to edit and add tests

11 TestIstanbul Conferences 2012 Coverage Precision is about attack surface/protocol coverage All interfaces/protocols tested? All message sequences tested? All message structures tested? All data definitions tested? All tags (values) tested? Accuracy is about anomaly coverage Anomaly categories? SQL? Buffer overflow? All values: 0..65k, a..z, 0x00..0x255 ? Combinations of anomalies?

12 TestIstanbul Conferences 2012 Anomaly Coverage Selection

13 TestIstanbul Conferences 2012 Traffic Capture Fuzzing

14 TestIstanbul Conferences 2012 Traffic Capture Fuzzing Results Test against samba seems to find zero-day

15 TestIstanbul Conferences 2012 Key Questions in regards to Fuzzing Test Coverage: – Q: Which interfaces and protocols are tested? – A: Understanding of threat models and attack surface are the most critical starting point to fuzzing – Q: How well something is tested? – A1: Specification coverage explains how deep knowledge the fuzzer has on a tested interface, and is typically represented by protocol models or protocol templates – A2: Anomaly coverage explains how well each protocol element is tested, and is the hardest metric to measure

16 TestIstanbul Conferences 2012 Fuzzing Scalability Software-based Fuzzers scale for all testing needs – The throughput depends only on the available hardware – The entire Network Under Test (NUT) can be virtualized – Software fuzzers can attack every interface, and against all protocol layers XML HTTP TLS TCP/IP

17 TestIstanbul Conferences 2012 Why Fast Speed Fuzzing? One of the most important aspects of fuzzing is how fast you can execute test cases. The faster you can execute test cases, the more test cases you can run and the more vulnerabilities you will find. - Dr. Charlie Miller from Independent Security Evaluators. " Two means of speeding fuzz testing: – More processing power – More virtualization (e.g. in cloud)

18 TestIstanbul Conferences 2012 Generating Load with Defensics Full model-based message sequences Options for monitoring, instrumentation, fuzzing, … Amount of available CPUs and Logging Level impact performance Some speed records (test cases per second): – TLS: tc/s – IPv6: tc/s – HTTP: tc/s – DNS: tc/s (with capture replay)

19 TestIstanbul Conferences 2012 Motivation for our Performance Study Test our test generation efficiency for different type of protocol fuzzers using high-end servers Analyze benefits (test efficiency, ROI) of software-based fuzzers to appliances

20 TestIstanbul Conferences 2012 Raw Fuzzing Performance Metrics Packets or bytes per second does not really work as each fuzz test case can consist of several packets back and forth Test cases, or full protocol transactions, generated by each suite provides more insight Performance is generated by having fuzzers running in parallel These result in test cases per second or tc/s

21 TestIstanbul Conferences 2012 Parallel Fuzzing Benefits No hardware constraints (fuzzing tools are just software) No operating system constraints Parallel processes and threads

22 TestIstanbul Conferences 2012 Test Setup We used two different setups: In the first setup, both test target and the test generator are in the same host. In the second setup, test generator is in a separate machine from the SUT

23 TestIstanbul Conferences 2012 Performance Bottlenecks and Metrics Network bandwidth: – Can be measured with number of full protocol transactions, packets, and data volume CPU usage by different components of Defensics – Test generator(s) – Test driver(s) Memory usage of Defensics components Most interesting metric: – New type of bugs related to performance?

24 TestIstanbul Conferences 2012 Fuzz Test Can Generate 50x Load

25 TestIstanbul Conferences 2012 Generating Load with Dell R910

26 TestIstanbul Conferences 2012 Test Plans Using Command-Line With some fuzzers, you can run the tools faster on command-line With Defensics, the GUI is sometimes even faster E.g. with a laptop with dual-core Intel T7500 processor (with default tool settings): – x valid-case in loop using GUI: 30 seconds using command-line: 35 seconds

27 TestIstanbul Conferences 2012 Running Fuzzers in Parallel Defensics GUI has options to run several suites in parallel – 2 x x valid-case in loop using GUI: 22 seconds (about % faster)

28 TestIstanbul Conferences 2012 Using Multi-threaded Mode When CPU resources are available, multi-treaded mode is often best way to speed up testing Uses less memory Note that all parallel execution creates more false positives Lets look at that and other speed optimization hints now in live demo...

29 TestIstanbul Conferences 2012 Performance Value Users can now… – Increase test coverage through parallel tests – Speed up testing – Find performance bottle-necks – Integrate Defensics also in other functional tests Remember that Fuzzing ROI is measured in: – TCO vs. Value of Security Findings / Cost of Security Compromise – Reduced maintenance costs, failure reproduction costs – Better customer confidence, higher retention rate of customers – In performance: software-based tools can be significantly cheaper (and faster) than appliances

30 TestIstanbul Conferences 2012 Conclusions Test automation can be really fast, and automated Model-based testing is not necessarily manual work Scalability of test automation depends on the tool architecture, but also on available processing resources Cloud-based solution vs. Hardware-based solution Virtualization Off-line generation versus online test generation Oh and finally: Fuzzing finds lots of bugs, the more you test the more bugs you will find

31 TestIstanbul Conferences 2012 More News from Codenomicon Facebook: – Become fan of Codenomicon and Fuzzing Twitter: – CodenomiconLTD Codenomicon Website: – Newsletter every second month

32 TestIstanbul Conferences 2012 Our Book On Fuzzing! Takanen, DeMott and Miller: Fuzzing for Software Security Testing and Quality Assurance Aimed at the general public, you do not need to be a security specialist to read this book Purpose of the book is to teach next-gen testing approaches to: – Software practitioners – Security engineers – Academics

33 TestIstanbul Conferences 2012 PROACTIVE SECURITY AND ROBUSTNESS SOLUTIONS THANK YOU – QUESTIONS? Thrill to the excitement of the chase! Stalk bugs with care, methodology, and reason. Build traps for them..... Testers! Break that software (as you must) and drive it to the ultimate - but dont enjoy the programmers pain. [from Boris Beizer]

Download ppt "TestIstanbul Conferences 2012 Parallel Execution of Fuzzing Test Suites Study of maximum throughput, resource consumption and bottlenecks for fast-speed."

Similar presentations

Ads by Google