1. 02/07/26 1 Autonomix: Autonomic Defenses for Vulnerable Software Crispin Cowan, Ph.D WireX Communications, Inc wirex.com

2. 02/07/26 2 Talk Outline Progress: LSM Experimentation: Defcon Technology Transition Conclusions

3. 02/07/26 3 LSM: Linux Security Modules Linux’s open source & broad popularity make it a great target for security research –SubDomain, DTE, SELinux, etc. But one has to have a custom kernel to use these packages Solution: security modules for Linux –Standardized interface in the Linux kernel for security modules –Get Linus et al to adopt LSM Expected result: –Can load advanced security into standard Linux kernels

4. 02/07/26 4 LSM: Linux Security Module Unfortunately, none are standard to Linux –Maintained as kernel patches –To deploy them, must acquire a custom kernel Linus would like to support advanced security policy, but not willing to endorse one project. –Too political… “My security policy is better than yours.” –Linus is not a security expert, and doesn’t want to be –Linux is about choice anyway Solution: enrich Linux’s module interface to support security policy modules

5. 02/07/26 5 LSM Design syscall interposition, i.e. wrappers at the syscall interface –not appropriate: leads to module bloat –already available by re-writing Linux syscall table Instead, we mediate access to internal kernel objects “May subject X access object Y for operation Z?”

6. 02/07/26 6 LSM - Architecture User-level process Kernel LSM Module Open syscall Std. error checks Std. Security checks LSM hook: Complete request Policy engine examine context does request pass policy? grant or deny

7. 02/07/26 7 LSM - Architecture User-level process Kernel LSM Module Open syscall Std. error checks Std. Security checks LSM hook: Complete request Policy engine examine context does request pass policy? grant or deny

8. 02/07/26 8 LSM - Architecture User-level process Kernel LSM Module Open syscall Std. error checks Std. Security checks LSM hook: Complete request Policy engine examine context does request pass policy? grant or deny “ok with you?”

9. 02/07/26 9 LSM - Architecture User-level process Kernel LSM Module Open syscall Std. error checks Std. Security checks LSM hook: Complete request Policy engine examine context does request pass policy? grant or deny “ok with you?” Yes or no

10. 02/07/26 10 Hook Style Restrictive: module may only reject a request about to be granted Permissive: module may only permit a request about to be rejected Authoritative: module may totally over-rule standard kernel logic We chose restrictive hooks only, except for capabilities –Simplifies LSM patch for maximum acceptability to Linux community

11. 02/07/26 11 Module Stacking Strong desire to compose modules However, composition in general is intractable Solution: stacking left to modules that want to stack –Stackable module must export an LSM-like interface “out the back” –Stackable module responsible for composing policy by taking down-chain module’s results under advisement –Module-stacking module (MUX) in development

12. 02/07/26 12 Hook Location

13. 02/07/26 13 LSM: Linux Security Module Progress Since February Implemented & working Running various WireX servers + other LSM activists SELinux shipping exclusively LSM packages Heard all that last PI meeting...

14. 02/07/26 14 LSM Progress Paper presented at USENIX Security –Plus IBM has a paper on LSM hook placement correctness Paper presented at Ottawa Linux Symposium –Plus three other papers with LSM content We were invited to the Linux Kernel Summit –Linus has accepted LSM; –Linux 2.5.27 has LSM in it

15. 02/07/26 15 LSM ToDo’s Cut the LSM patch into bite-sized pieces –Easier for the Linux maintainers to digest Work with Al Viro to get the VFS patch we need –He’s working on it, but not quickly Address the network performance problem –Leverage network hooks out of LSM –It’s Netfilter’s fault

16. 02/07/26 16 LSM Lesson:How to Get a Feature Into Linux Linux allows you to do it your way, but to be in Linus’ kernel, you have to do it in a way acceptable to Linus –Do something that makes him happy –Linus trusts his major subsystem maintainers, so work with them –Keep someone interacting constructively with the LKML (Linux Kernel Mailing List), especially core developers Have to actually solve a problem –LSM is effectively a direct response to Linus wishing out loud

17. 02/07/26 17 LSM Lesson: Hard Choices To keep Linus happy, we had to make some tough choices –Security people largely would prefer authoritative hooks, and many more of them –That would enable full POSIX audit logs But that also would have killed LSM’s chances of getting into Linux –Linus (and many others) do not like BSM –They would have regarded LSM as unnecessary bloat

18. 02/07/26 18 LSM Lesson: Collaboration Before LSM: –Competing security projects all working independently With LSM: –Lots of collaboration among many projects –Working to provide a common infrastructure –Creating a common market for composable and competing security features The trick: –Finding the right layer to abstract –Political engineering: make sure there is something in it for everyone

19. 02/07/26 19 LSM Collaboration Community 500 people on the mailing list Major contributions coming from: –3 IBM sites (2 LTC sites + T.J. Watson Lab) –SELinux (NAI/NSA) –SGI –Assorted open source people Commercial LSM modules in the offing: –WireX: SubDomain RaceGuard CryptoMark –HP Secure Linux –Ericsson Research

20. 02/07/26 20 Experimentation... Some real-world red teaming Play an Immunix server in the Defcon Capture the Flag (CtF) games Almost no holds barred: –No flooding –No physical attacks New gaming rig designed by the Ghettohackers

21. 02/07/26 21 Basic Defcon CtF Rules Player Nodes

22. 02/07/26 22 Basic Defcon CtF Rules Player Nodes Score’bot Polls player nodes, Looking for req. services If all services found...

23. 02/07/26 23 Basic Defcon CtF Rules Player Nodes Score’bot Polls player nodes, Looking for req. services If all services found, Score one point for the Flag currently on that node

24. 02/07/26 24 Basic Defcon CtF Rules Player Nodes Score’bot Polls player nodes, Looking for req. services If all services found, Score one point for the Flag currently on that node … while each team tries to replace others’ flags

25. 02/07/26 25 No Flooding DoS attacks are not interesting Explicit rule against flooding attacks –Game masters will make you stop if you are caught at it –Goal: ensure that all teams are actually able to play Penalties: –Kicked out for overt DoS attacks –Pay for bandwidth with a point penalty

26. 02/07/26 26 Area View

27. 02/07/26 27 Sporting Event Teams named funky colors Score obfuscated There was an official bookie :-) Score broadcast on hotel cable Immunix was white, hence “Weiss Labs”

28. 02/07/26 28 The Catch The required services are secret Only a few clues: –They supply us with a VMWare/Linux image reference distribution that provides all required services It is also riddled with vulnerabilities –The score’bot polls for the required services But the score’bot stops its poll if it finds something it doesn’t like

29. 02/07/26 29 The Reference Distribution Red Hat 6.2, unpatched nmap: shows nearly everything open –finger, POP, IMAP, SMTP, SNMP, Webmin... Apache running as root CGI’s for adduser and deleteuser –Anonymous can create a user login on your node –As any user number, including zero

30. 02/07/26 30 Example Services the Score’bot Wanted Create a user Send that user mail Finger the user POP in to fetch the mail Delete the user Note: no crypto protocols –No proper authentication of the score’bot –Must heuristically distinguish score’bot from attacks using behavior signatures

31. 02/07/26 31 Interesting Challenge Not just survive severe attack, but also –Protect bad code –A lot of it –Vague functional specification –Rapid deployment Great new game infrastructure from Ghettohackers –Interesting challenge –Engaging scoreboard

32. 02/07/26 32 Captain’s Meeting Explain the rules in detail Hand us the reference distribution

33. 02/07/26 33 Setting Up

34. 02/07/26 34 The Popular Strategy: Human Intrusion Detection Launch the reference Linux distribution Ad hoc patch as stuff happens Defend: –look for logins, I.e. non-score’bot behavior –kill them off ASAP –very labor-intensive

35. 02/07/26 35 The Immunix Strategy: Protect Bad Code with Immunix Tools Port all plausible services to Immunix 7+ distribution –Use our own fingerd, httpd, etc., up-to-date and compiled with StackGuard and FormatGuard –Run on an Immunix kernel with SubDomain and RaceGuard –Wrap vulnerable services & CGI’s with SubDomain profiles to limit access to least privilege necessary Launch only when we were reasonably confident that the Immunix machine was configured securely

36. 02/07/26 36 Dealing with Logins: the SubDomain Shim Change adduser CGI to use a special default shell: /bin/fubush –/bin/fubush is just a hard link to /bin/bash –Restrict /bin/fubush to only the operations needed by the score’bot Attackers can go ahead and create a login with uid 0 and it still won’t do them any good –They get a root shell, stuck in a tiny sandbox

37. 02/07/26 37 Immunix Team

38. 02/07/26 38 Immunix Team Me Chris Wright Seth Arnold Steve Beattie Plus 15 volunteers

39. 02/07/26 39 From Our Corner

40. 02/07/26 40 From Our Corner John Viega Me Chris Wright Seth Arnold Steve Beattie

41. 02/07/26 41 Mental Stress This is a tough game to play –Head-to-head competition with a lot of very smart people –Real-time –Continuous The intensity of qualifying exams –That go on for 22 hours in a 48 hour period –… set in the middle of a rave Hydrate or die :-)

42. 02/07/26 42 Rave Loud music Smoking Gawkers Social engineering Periodic “news breaks”

43. 02/07/26 43 Our Strategic Error What We Did For first 4 hours –No server at all –Porting services to Immunix ASAP, based largely on nmap and source inspection Next 4 hours –Launch Immunix server –It’s secure, but is not making the score’bot happy Cost us massive points –Too focused on the science of “can we defend Immunix?” and not enough on the game rules What We Should Have Done Launch reference system immediately –Defend ad hoc like everyone else –Run network sniffer to determine what the score’bot wants Would have: –Put us over the top on points –Learned what score’bot wants much faster We eventually did this

44. 02/07/26 44 Immunix Server Not Up Yet 6 th place

45. 02/07/26 45 Once Immunix Server Up … in the Score’bot’s Opinion :) Our score quickly rose 2 nd place

46. 02/07/26 46 Once Immunix Server Up … in the Score’bot’s Opinion :) Close 2 nd place

47. 02/07/26 47 Once Immunix Server Up … in the Score’bot’s Opinion :) 1 st place Stayed there most of Saturday

48. 02/07/26 48 Late Saturday: New Service Requirement With 4 hours of play to go, the score’bot changed: now it wanted Webmin –Open source web-GUI for Linux administration –Competitor to WireX’s commercial server appliance software –Rather famously vulnerable :) Took us 2 hours Sunday morning to make the score’bot happy again –Lost our lead

49. 02/07/26 49 Some of Our Creative Attacks Lock Out the Owner Once we root the machine, install a back door Also replace root’s login shell with /sbin/halt –Owner can’t log in to their own machine –But we can Spam’bot Add user to their server User sends spam mail to all the other teams Costs them penalty points Penalties are per connection –Spam’bot sends 1-byte e-mails

50. 02/07/26 50 Final Score: 2 nd Place

51. 02/07/26 51 Lesson: Symmetric Red Teaming Solves Rules Issues Everyone is both an attacker and defender Bad: everyone needs to learn how to attack Good: –Everyone should learn how attacks are done :-) –Rule fussing about how hard or easy it is for the attacker apply to all parties -> less fussing Ghettohackers have designed a great game –Looking for technology transfer to Government

52. 02/07/26 52 Lesson: Mandatory Access Control is Not Enough telnetd was a required service WireX never bothered to patch a vulnerability in telnetd for Immunix –Only idiots run telnetd :-) Someone hacked our telnetd –Didn’t get out of the SubDomain sandbox –Did make our telnetd stop working –Cost us a point that round General case: MAC protects your system, but not your individual services

53. 02/07/26 53 Lesson: Resource Management is a Security Attribute SubDomain confined attacker logins to only run prescribed code –Including PERL Attacker launched a PERL fork bomb –Consumed all of real and virtual memory –While our machine is thrashing, the score’bot passes us by –Costs us a point that round

54. 02/07/26 54 Lesson: Redundancy Helps When You Are Vulnerable Penetration attacks take a long time to recover –Must clean up state, find & fix vulnerability DoS attacks take a long time to recover –If machine crashes, must fsck file system; can take 10 minutes Hot spare can be on-line in seconds –Heterogeneous hot spare keeps attacker from immediately deploying the same attack

55. 02/07/26 55 Lesson: Redundancy is Resource-Constrained Must have humans on watch to clean up the compromised machine –The hot spare will not protect you for long Presumption that hot spare prevents attacker from attacking again assumes resource limit at the attacker’s end –If attacker has lots of exploits/resources, they will hack your heterogeneous server just as quickly

56. 02/07/26 56 Lesson: Immunix was Impenetrable, but not Incorruptible No one ever “flagged” the Immunix server –Others did plant enemy flags on our reference server (as expected) But they did hit the Immunix server hard enough to compromise availability –Take out one required service, and the score’bot doesn’t award a point –We missed first place by less than 4 points out of approx. 55

57. 02/07/26 57 Technology Transfer LSM: Linux Security Module –Linus said “yes” Immunix: licensed to Compaq/HP –Build a product family of security appliances –Firewall, AV Mail server, NIDS, etc. –Compaq hardware –Immunix OS, user interface –3 rd party security applications –Press release this week

58. 02/07/26 58 Summary LSM: –Technical work stable –Political breakthrough –Technology transfer heavy lifting coming next; feed actual patches to Linus et al Other technologies: working on them... Experimentation: Defcon fun :) Technology Transfer: going well –LSM in Linux –Immunix licensed to HP

59. 02/07/26 59 Web Resources LSM: http://lsm.immunix.org –Active mailing lists Sign up if interested –If you are hacking security into the Linux kernel, consider making it an LSM module Defcon: –Defcon convention http://www.defcon.org –CtF game http://www.ghettohackers.net/ctf/