1. © 2011 Codenomicon. all rights reserved. Robustness Testing: Discover unknown vulnerabilities with Testing & QA Ari Takanen Codenomicon Ltd.

3. © 2011 Codenomicon. all rights reserved. Be Proactive with Security Modern security testing is about finding unknown zero-day vulnerabilities in devices and software before and after release Provides a quick technique for security assurance for any device or software www.codenomicon.com/unknown/

4. © 2011 Codenomicon. all rights reserved. Security Vulnerability = Just A Bug

5. © 2011 Codenomicon. all rights reserved. Same Applies to (Legacy) Mobile Phones

6. © 2011 Codenomicon. all rights reserved. The Challenge

7. © 2011 Codenomicon. all rights reserved. Internet of Things = Future market for security and testing 1875 1900 1925 1950 1975 2000 2025 50 B 5.0 B ~0.5 B PLACES PEOPLE THINGS Inflection points Global Connectivity Personal Mobile Digital Society Sustainable World Source: Ericsson

8. © 2011 Codenomicon. all rights reserved. Codenomicon Labs Test Results http://www.codenomicon.com/labs/results

9. © 2011 Codenomicon. all rights reserved. Smart phone – attack surface WIRELESS: Bluetooth: L2CAP, RFCOMM, SDP, OPP, A2DP, AVRCP, PBAP, DUN,... WIRELESS: 802.11: 802.11a/b/g/n, WPA, WPA2,.. WIRELESS: GPRS, EDGE/3G GSM, SMS, MMS, SMIL, OTA updates,... PHYSICAL CONNECTIVITY: USB, SERIAL, MEMORY CARD, SIM,.. IP CONNECTIVITY: IPv4 (ARP, ICMP, IGMP, IP, UDP, TCP), IPv6 (IP, ICMP, ND, RD, SEND, MLD, TCP, UDP), HTTP, TLS/SSL, OCSP, RTSP, SIP/IMS, RTP/RTCP, SigComp, DNS, MDNS, DHCP, NTP, SOAP, REST/JSON, SMTP, POP3, IMAP4, WAP/WMLC,.. [WEB] APPLICATIONS: XML, DRM, HTML5 (CSS, HTML, Javascript), AT commands, inter process APIs/RPCs, MEDIA: AUDIO (AAC, MP3, MP4, 3GP, WAV,...), IMAGES (JPG, GIF, PNG, TIFF,...), VIDEO (MPG1, MPG2, MP4/H.264, WEBM,... ), ARCHIVES (ZIP, JAR, CAB,...), DOCUMENTS (PDF, DOC, PPT,..), X509, EMAIL (MIME, calendar, vcards,...), DRM, Flash, Java classes, Application installers,...

10. © 2011 Codenomicon. all rights reserved. Approaches to testing, how does fuzzing fit in? Feature/conformance testing Performance/load testing Robustness testing – Fuzzing – Static Code Analysis

11. © 2011 Codenomicon. all rights reserved. Microsoft SDL & fuzzing & static code analysis

12. © 2011 Codenomicon. all rights reserved. Microsoft SDL: Fuzz Here? Many organizations choose to deploy fuzzing in other parts of the SDL as well. Many organizations choose to deploy fuzzing in other parts of the SDL as well.

13. © 2011 Codenomicon. all rights reserved. Definition of fuzzing Fuzzing is a technique for – intelligently and – automatically generating and passing into a target system – valid and – invalid message sequences to see if the system breaks, and if it does, what it is that makes it break.

14. © 2011 Codenomicon. all rights reserved. Product Security Terminology Vulnerability – a weakness in software, a bug. Threat/Attack – exploit against a specific vulnerability Protocol Modeling – functional behavior, interface message sequences and message structures Anomaly – abnormal or unexpected input Failure – crash, busy-loop, memory corruption, or other indication of a bug in software

15. © 2011 Codenomicon. all rights reserved. Types of fuzzing Random fuzzing – Apple 1980s – Barton P. Miller 1980s, 1990s Template based fuzzing – Capture traffic OR use sample files OR... create mutated test cases Specification based fuzzing – Model the specification, inject anomalies, transmit to target system

16. © 2011 Codenomicon. all rights reserved. Example Fuzzing Session

17. © 2011 Codenomicon. all rights reserved. What kinds of bugs does it find?

18. © 2011 Codenomicon. all rights reserved. Why We Must Fuzz? Update Frequency Designing systems for very long operational and legacy device support, security? Try to secure devices that get infrequent updates or those needing very high severity updates out of band Always-on applications or devices will have to deal with live updates, no down-time and still function in rugged/robust environments Mission critical devices will bring their own unique set of requirements – guaranteed up-time, high security and immunity from updates being an attack source

19. © 2011 Codenomicon. all rights reserved. Fuzzing vs. Common Criteria Calculation of attack potential for Fuzzing tools: FactorOpen Source Fuzzers ScoreCommercial Fuzzers Score Elapsed Time to Exploitation less than a week 1less than a day0 ExpertiseExpert6Layman0 Knowledge of TOE Public0 0 Window of Opportunity Easy1 1 EquipmentStandard0Specialized4

20. © 2011 Codenomicon. all rights reserved. Attack Potential for Fuzzing Tools Attack potential for fuzzing tools is 5-8… What does that mean: 0-9 = Basic = AVA_VAN.1-5 should not fail 10+ = Enhanced Basic required at EAL4 All Common criteria evaluated products should survive basic attacks such as fuzz-testing?

21. © 2011 Codenomicon Ltd. 21 Example: Traffic Capture Fuzzing

22. © 2011 Codenomicon. all rights reserved. Models and Rules

23. © 2011 Codenomicon. all rights reserved. Scaling Fuzz Tests Robustness is also about performance, and therefore model-based tools have to be fast in generating test cases

24. © 2011 Codenomicon. all rights reserved. Testing In The Cloud Test the interior and exterior of the cloud, including services, devices, applications, and hypervisor stability

25. © 2011 Codenomicon. all rights reserved. Model-based Fuzz-Testing Examples [FUZZING] tools are *amazing*. Using them is like being attacked by the most relentless adversary who uses every possible method to find flaws in your code We fixed subtle crash bugs in Samba that had been in the code for over ten years. We would *never* have found those bugs without the [FUZZING] tools. If you're serious about implementing protocols correctly, you need [FUZZING] tools. -- Jeremy Allison, Co Creator of Samba.

26. © 2011 Codenomicon. all rights reserved. Model-based Fuzz-Testing Examples

27. © 2011 Codenomicon. all rights reserved. Model-based Fuzz-Testing Examples

28. © 2011 Codenomicon. all rights reserved. Model-based Fuzz-Testing Examples

29. © 2011 Codenomicon. all rights reserved. Model-based Fuzz-Testing Examples

30. © 2011 Codenomicon. all rights reserved. Model-based Fuzz-Testing Examples

31. © 2011 Codenomicon. all rights reserved. Conclusions Why is fuzzing always an excellent choice for a testing solution... –... and sometimes the only feasible one? Easy to automate, systematic, top coverage, top efficiency Increasingly widely adopted, some contractors/customers require it Real life examples indicate: you will find security critical bugs by fuzzing

32. © 2011 Codenomicon, Ltd. 32 DEFEND. THEN DEPLOY. PROACTIVE SECURITY AND ROBUSTNESS SOLUTIONS THANK YOU – QUESTIONS? Thrill to the excitement of the chase! Stalk bugs with care, methodology, and reason. Build traps for them..... Testers! Break that software (as you must) and drive it to the ultimate - but dont enjoy the programmers pain. [from Boris Beizer]