1. Packet Mangling for Fun & Profit A Brief Intro to Netfilter, User Mode Linux, and the Linux TCP/IP Stack

2. Topics SplitTCP Summary (Context) Netfilter Introduction One buffer to rule them all: sk_buffs User Mode Linux Current Status

3. SplitTCP In a Nutshell Work started by Michalis, Srikanth, and Swastik. The idea is to add transport layer proxies to long (in terms of hop count) TCP connections. If a link fails (due to mobility, etc.) the packet can be re-transmitted by the proxy closest to the failure. A “demo-quality” implementation was done which made many simplifying assumptions.

4. So What? My current task is to take that demo implementation and turn it into a general implementation. This presentation discusses the lessons learned up to this point in that process.

5. Design Goals Any and all modifications made to the behavior of TCP should be backwards compatible. I.e. nodes should interoperate regardless of weather or not they’re “split tcp enabled”.) If possible, no changes should be made to the kernel proper. It’s just better for everyone that way…

6. Enter Netfilter Fortunately, Linux has a plugin API, called Netfilter, which allows kernel modules to hook into strategic spots in the networking stack. Unfortunately, in the grand tradition of open source, documentation takes a back seat to implementation – so there’s a non-trivial learning curve.

7. Sideline: Plugin APIs An example of the Delegation design pattern (GoF) – a work unit is passed through a series of cooperating steps to produce the final result. A well-designed plugin architecture can facilitate otherwise impossible tasks and simply possible ones.

8. Sideline: Kernel Modules Most of the linux kernel can be built as a module – a bit of code that’s loaded into (and unloaded from) the kernel dynamically. Device drivers, file systems, even networking protocols (IPX, Appletalk) are candidates for modularization.

9. Writing a Module Just define a few preprocessor macros (__KERNEL__, MODULE) Include a few header files (linux/module.h, linux/version.h, linux/config.h) And honor a small interface (module_init(), module_exit()) You’ll have a no-op module.

10. Netfilter Overview Initial design by Paul “Rusty” Russell Netfilter is a series of callback functions within the network stack. The API is non-portable and appeared in linux 2.3.x Each protocol has it’s own set of callback points. We care about IPv4.

11. Netfilter Concepts A module expresses interest in being invoked at an arbitrary subset of the available callback points – specifying the function and the (global) priority in which it should be called. That function is passed (among other things) a pointer to a pointer to a packet buffer ( sk_buff ** ).

12. Return Values The netfilter function has five possible return values: NF_ACCEPT: continue callback chain NF_DROP: drop the packet and stop the chain NF_STOLEN: stop the chain NF_QUEUE: send the packet to userspace NF_REPEAT: call the hook again

13. Netfilter Hooks in IPv Routing Engine Local Sockets 1 In Out 2 3 4 5

14. Say that Again? 1: NF_IP_PREROUTING any received packet which checksums OK. 2: NF_IP_LOCAL_IN packets destined for local sockets 3: NF_IP_FORWARD foreign packets being forwarded 4: NF_IP_POST_ROUTING any outbound packet 5: NF_IP_LOCAL_OUT packets originating from local sockets Routing Engine Local Sockets 1 In Out 2 4 5

15. An sk_what?? Linux uses a structure called an sk_buff to store packet data internally. It contains a handful of pointers to other structures as well as a packet data region. Data Pointers Headroom Tailroom

16. Sk_buff’s and you The data area is like a stack, only you can insert at the head and the tail (deque?). The kernel provides a handful of helper functions to manage sk_buff’s and their data areas. The various header pointers point into the data area – which can be thought of as a serialized packet.

17. Why do We Care? An sk_buff is built as a packet travels down the stack – each layer (TCP, IP, Ethernet) adds their own special sauce. This means that each header is “squashed” in against the next – so while modifying existing data is relatively easy, adding new header data is a bit trickier.

18. Don’t leave me in suspense… Basically, you make a copy of the sk_buff, and ask it to “grow” a bit during the copy. Once you have the copy – you “slide” the IP and TCP headers backwards a bit, insert the new option bytes, and re- checksum the packet.

19. A tale of “n” checksum’s Sounds easy, right? Remember that these sk_buffs are built one layer at a time? There is no nice friendly function which will take a TCP sk_buff and compute all the needed checksums. Funny thing about checksums – almost isn’t good enough.

20. Ok, insmod and Remember developing on a system without memory protection and having to reboot ? Kernel modules execute in kernel space – so no one’s watching your back. If you goof, it’s time to reboot. User Mode Linux to the rescue

21. User Mode Linux (UML) A kernel patch that allows running the linux kernel as a user-mode process on a linux machine. If you crash the user-mode kernel, you just restart the process, no reboot required.

22. Where do I sign up? Setup is (in principal) fairly easy – only it turns out that the standard distribution doesn’t have netfilter enabled. So I re-built with the appropriate options and placed the binaries in ~swift/user-mode- linux/bin There’s a README there – some support executables must be installed as root on your workstation.

23. That’s It? Not quite – you also need a file system to boot this kernel off. Good News: you can download a file system image – you don’t have to make one. Bad News: it’s really big. (up to 700MB) Good News: you can share one among many people Bad News: you have to read the HOWTO.

24. Building Modules Under UML Building a kernel module is the same under UML as under “KML” – it is important that you build it against the source used to build the target kernel. For “our” UML build – that source is in ~swift/user-mode-linux/src

25. Current Status A skeleton of a splittcp module (tcpproxy.c) exists. It can inspect locally generated packets and add our newly defined PROXY option, rechecksum the packet, and send it on it’s way. It can inspect arriving packets, check for the option, and decide if that packet should be proxied.

26. So What’s Left? It doesn’t (yet) actually proxy the packet. Nor does it send an acknowledgement of receipt to the upstream proxy. There are also issues around ICMP error messages, and what should be done about them.

27. Conclusion Once the code is “alpha” quality, I’ll commit it to the swift CVS repository for your collective viewing pleasure. Until then, if you have questions or suggestions, see me.