1. 4/23/2003 1 Immunix & Defcon: Defending Vulnerable Code From Intense Attack Crispin Cowan, Ph.D Seth Arnold, Steve Beattie, Chris Wright WireX and John Viega, Secure Software

2. 4/23/2003 2 Talk Outline About WireX and Immunix –Secure Linux Systems The Defcon Challenge –Defend vulnerable code against massive attack Technology Transfer –Commercial products built on this technology

3. 4/23/2003 3 Software Security Software security is really simple: Make sure you only run perfect software Uh-oh :-) Intrusion Prevention: –Systems that detect attack attempts in real time, and reject them When bugs occur, they are not exploitable –Attacker cannot exploit the bug to gain unintended privileges

4. 4/23/2003 4 Immunix Security Technologies Shipping: StackGuard: stops buffer overflows FormatGuard: stops printf format bugs RaceGuard: stops temp file races SubDomain: contains vulnerable programs

5. 4/23/2003 5 StackGuard Stack Smashing Problem: –Weak bounds checking on inputs in C programs –Attacker overflows input buffer, corrupting adjacent state to gain control of the program –Most common target: function return address on the stack StackGuard: –C compiler enhancement –Ornaments call stack to detect corruption –Very low performance overhead –WireX has been shipping fully StackGuard’d system since 1999

6. 4/23/2003 6 FormatGuard Format String Problem: Sudden discovery in June 2000 –Vulnerability in WU-FTPD –Followed by hundreds of similar vulnerabilities Basis: arcane %n printf format string directive –Treat corresponding argument as an int * –Write back number of items formatted so far Problem: programs that pass un-filtered user input strings direct to printf FormatGuard: –Similar to StackGuard –compiled defense against printf format string vulnerabilities –CPP macro Counts arguments at the call site at compile time Compares that number to the format string presented at run time

7. 4/23/2003 7 RaceGuard Temporary File Race Problem: –Portable procedure for temporary file creation is non-atomic –If attacker gets in the middle, can redirect temporary file creation by privileged programs to corrupt the system RaceGuard: –Kernel enhancement to detect race attacks mid- way through –Abstract method: detect changes between stat() and open() accesses to the same file name

8. 4/23/2003 8 Containment If your software is vulnerable anyway, you need to contain it so that it runs with the least privilege necessary to perform designated function Chroot: basic isolation for vulnerable programs Immunix SubDomain: flexible confinement for vulnerable programs

9. 4/23/2003 9 Containment With Chroot “Change root”: makes some subdirectory appear to be the root (“/”) directory for the calling process and its children –Available as both a shell command and a system call Effect: chroot’d programs cannot affect anything outside the chroot “jail” –Limits impact of bugs in program, e.g. chroot BIND Benefits: Standard: Comes with most UNIX’s Compatible: several current programs have been modified to work within a chroot jail Fast: no performance degradation Limitations: Work: must move copies of everything a jailed program needs into the jail Isolation: jailed program cannot interact at all with the rest of the system

10. 4/23/2003 10 Containment With Immunix SubDomain Part of Immunix Kernel Extension: –Specify the list of files that a SubDomained program may access Effect: SubDomained programs cannot affect anything they don’t explicitly need access to –Limits impact of bugs in program, e.g. SubDomain CGI scripts Benefits: Flexible: SubDomained programs can have controlled interaction with the rest of the system Compatible: SubDomain can confine binary programs without modifications Fast: 1% or less performance overhead Limitations: Work: must specify “shape” of SubDomain

11. 4/23/2003 11 Containing PHF PHF: infamous vulnerable CGI script –legitimate function: database lookup of user information –sloppy parsing of CGI input –can get PHF to start an xterm on an arbitrary display To SubDomain PHF: –Specify all the files that PHF needs Effect: –access to all other files is denied –Including xterm :-) Place this file in /etc/subdomain.conf/phf /home/httpd/cgi-bin/phf { /bin/sh x, /etc/ld.so.cache r, /etc/nsswitch.conf r, /lib/ld-linux.so.2 r, /lib/libc.so.6 r, /lib/libtermcap.so.2 r, /usr/local/bin/ph ix, }

12. 4/23/2003 12 WireX Systems: Immunix Secure OS Linux system similar to Red Hat Linux RPM based All source-available programs compiled with StackGuard and FormatGuard –PointGuard in future releases Kernel equipped with SubDomain and RaceGuard All network-accessible daemons SubDomain-profiled

13. 4/23/2003 13 Experimentation... Some real-world red teaming Play an Immunix server in the Defcon Capture the Flag (CtF) games Almost no holds barred: –No flooding –No physical attacks New gaming rig designed by the Ghettohackers

14. 4/23/2003 14 Basic Defcon CtF Rules Player Nodes

15. 4/23/2003 15 Basic Defcon CtF Rules Player Nodes Score’bot Polls player nodes, Looking for req. services If all services found...

16. 4/23/2003 16 Basic Defcon CtF Rules Player Nodes Score’bot Polls player nodes, Looking for req. services If all services found, Score one point for the Flag currently on that node

17. 4/23/2003 17 Basic Defcon CtF Rules Player Nodes Score’bot Polls player nodes, Looking for req. services If all services found, Score one point for the Flag currently on that node … while each team tries to replace others’ flags

18. 4/23/2003 18 No Flooding DoS attacks are not interesting Explicit rule against flooding attacks –Game masters will make you stop if you are caught at it –Goal: ensure that all teams are actually able to play Penalties: –Kicked out for overt DoS attacks –Pay for bandwidth with a point penalty

19. 4/23/2003 19 Area View

20. 4/23/2003 20 Sporting Event Teams named funky colors Score obfuscated There was an official bookie :-) Score broadcast on hotel cable Immunix was white, hence “Weiss Labs”

21. 4/23/2003 21 The Catch The required services are secret Only a few clues: –They supply us with a VMWare/Linux image reference distribution that provides all required services It is also riddled with vulnerabilities –The score’bot polls for the required services But the score’bot stops its poll if it finds something it doesn’t like

22. 4/23/2003 22 The Reference Distribution Red Hat 6.2, unpatched nmap: shows nearly everything open –finger, POP, IMAP, SMTP, SNMP, Webmin... Apache running as root CGI’s for adduser and deleteuser –Anonymous can create a user login on your node –As any user number, including zero

23. 4/23/2003 23 Example Services the Score’bot Wanted Create a user Send that user mail Finger the user POP in to fetch the mail Delete the user Note: no crypto protocols –No proper authentication of the score’bot –Must heuristically distinguish score’bot from attacks using behavior signatures

24. 4/23/2003 24 Interesting Challenge Not just survive severe attack, but also –Protect bad code –A lot of it –Vague functional specification –Rapid deployment Great new game infrastructure from Ghettohackers –Interesting challenge –Engaging scoreboard

25. 4/23/2003 25 Captain’s Meeting Explain the rules in detail Hand us the reference distribution

26. 4/23/2003 26 Setting Up

27. 4/23/2003 27 The Popular Strategy: Human Intrusion Detection Launch the reference Linux distribution Ad hoc patch as stuff happens Defend: –look for logins, I.e. non-score’bot behavior –kill them off ASAP –very labor-intensive

28. 4/23/2003 28 The Immunix Strategy: Protect Bad Code with Immunix Tools Port all plausible services to Immunix 7+ distribution –Use our own fingerd, httpd, etc., up-to-date and compiled with StackGuard and FormatGuard –Run on an Immunix kernel with SubDomain and RaceGuard –Wrap vulnerable services & CGI’s with SubDomain profiles to limit access to least privilege necessary Launch only when we were reasonably confident that the Immunix machine was configured securely

29. 4/23/2003 29 Dealing with Logins: the SubDomain Shim Change adduser CGI to use a special default shell: /bin/fubush –/bin/fubush is just a hard link to /bin/bash –Restrict /bin/fubush to only the operations needed by the score’bot Attackers can go ahead and create a login with uid 0 and it still won’t do them any good –They get a root shell, stuck in a tiny sandbox

30. 4/23/2003 30 Immunix Team

31. 4/23/2003 31 Immunix Team Me Chris Wright Seth Arnold Steve Beattie Plus 15 volunteers

32. 4/23/2003 32 From Our Corner

33. 4/23/2003 33 From Our Corner John Viega Me Chris Wright Seth Arnold Steve Beattie

34. 4/23/2003 34 Mental Stress This is a tough game to play –Head-to-head competition with a lot of very smart people –Real-time, continuous The intensity of qualifying exams –That go on for 22 hours in a 48 hour period –… set in the middle of a rave Hydrate or die :-)

35. 4/23/2003 35 Rave Loud music Smoking Gawkers Social engineering Periodic “news breaks”

36. 4/23/2003 36 Our Strategic Error What We Did For first 4 hours –No server at all –Porting services to Immunix ASAP, based largely on nmap and source inspection Next 4 hours –Launch Immunix server –It’s secure, but is not making the score’bot happy Cost us massive points –Too focused on the science of “can we defend Immunix?” and not enough on the game rules What We Should Have Done Launch reference system immediately –Defend ad hoc like everyone else –Run network sniffer to determine what the score’bot wants Would have: –Put us over the top on points –Learned what score’bot wants much faster We eventually did this

37. 4/23/2003 37 Immunix Server Not Up Yet 6 th place

38. 4/23/2003 38 Once Immunix Server Up … in the Score’bot’s Opinion :) Our score quickly rose 2 nd place

39. 4/23/2003 39 Once Immunix Server Up … in the Score’bot’s Opinion :) Close 2 nd place

40. 4/23/2003 40 Once Immunix Server Up … in the Score’bot’s Opinion :) 1 st place Stayed there most of Saturday

41. 4/23/2003 41 Late Saturday: New Service Requirement With 4 hours of play to go, the score’bot changed: now it wanted Webmin –Open source web-GUI for Linux administration –Competitor to WireX’s commercial server appliance software –Rather famously vulnerable :) Took us 2 hours Sunday morning to make the score’bot happy again –Lost our lead

42. 4/23/2003 42 Some of Our Creative Attacks Lock Out the Owner Once we root the machine, install a back door Also replace root’s login shell with /sbin/halt –Owner can’t log in to their own machine –But we can Spam’bot Add user to their server User sends spam mail to all the other teams Costs them penalty points Penalties are per connection –Spam’bot sends 1-byte e-mails

43. 4/23/2003 43 Final Score: 2 nd Place

44. 4/23/2003 44 Lesson: Symmetric Red Teaming Solves Rules Issues Everyone is both an attacker and defender Bad: everyone needs to learn how to attack Good: –Everyone should learn how attacks are done :-) –Rule fussing about how hard or easy it is for the attacker apply to all parties -> less fussing Ghettohackers have designed a great game –Looking for technology transfer to Government

45. 4/23/2003 45 Lesson: Mandatory Access Control is Not Enough telnetd was a required service WireX never bothered to patch a vulnerability in telnetd for Immunix –Only idiots run telnetd :-) Someone hacked our telnetd –Didn’t get out of the SubDomain sandbox –Did make our telnetd stop working –Cost us a point that round General case: MAC protects your system, but not your individual services

46. 4/23/2003 46 Lesson: Resource Management is a Security Attribute SubDomain confined attacker logins to only run prescribed code –Including PERL Attacker launched a PERL fork bomb –Consumed all of real and virtual memory –While our machine is thrashing, the score’bot passes us by –Costs us a point that round

47. 4/23/2003 47 Lesson: Redundancy Helps When You Are Vulnerable Penetration attacks take a long time to recover –Must clean up state, find & fix vulnerability DoS attacks take a long time to recover –If machine crashes, must fsck file system; can take 10 minutes Hot spare can be on-line in seconds –Heterogeneous hot spare keeps attacker from immediately deploying the same attack

48. 4/23/2003 48 Lesson: Redundancy is Resource-Constrained Must have humans on watch to clean up the compromised machine –The hot spare will not protect you for long Presumption that hot spare prevents repeat attacks assumes resource limit at the attacker’s end –If attacker has lots of exploits/resources, they will hack your heterogeneous server just as quickly We had a hot spare, but not enough of them

49. 4/23/2003 49 Lesson: Immunix was Impenetrable, but not Incorruptible No one ever “flagged” the Immunix server –Others did plant enemy flags on our reference server (as expected) But they did hit the Immunix server hard enough to compromise availability –Take out one required service, and the score’bot doesn’t award a point –We missed first place by less than 4 points out of 55

50. 4/23/2003 50 Immunix Secure Server Appliances Base: Immunix Secure OS –Appliances that defend themselves Turn-key installation: –WireX installer turns PC hardware into server appliance in 5 minutes –No technical skills required Graphical ease of use –WireX Secure Server Manager Web-GUI management system

51. 4/23/2003 51

52. 4/23/2003 52 Immunix Secure Server Solutions Partner with HP First product: Trend Micro Antivirus Mail scanner http://h18000.www1.hp.com/products/servers/solutions/iis Websense content filter Secure Webmail Appliance –Customized for USTRANSCOM –Provides secure webmail access to Microsoft Exchange –Supports Microsoft Exchange/Outlook calendar events –Partner with Secure Computing All being demonstrated in our booth

53. 4/23/2003 53 Summary WireX Immunix: –Secure Linux system with low overhead and administrative hassle Defcon fun: –Couldn’t crack Immunix –Integrated undocumented code into Immunix in 12 hours Available as –Secure Linux system –Turn-key secure server solutions