1. 1 Combining Events and Threads for Scalable Network Services Peng Li and Steve Zdancewic University of Pennsylvania PLDI 2007, San Diego

2. 2 Overview  A Haskell framework for massively concurrent network applications  Servers, P2P systems, load generators  Massive concurrency ::= 1,000 threads? (easy) | 10,000 threads? (common) | 100,000 threads? (challenging) | 1,000,000 threads? (20 years later?) | 10,000,000 threads? (in 15 minutes)  How to write such programs?  The very first decision to make: the programming model Shall we use threads or events ? A lazy, purely functional programming language http://www.haskell.org

3. 3 Threads vs. Events The multithreaded model  One thread ↔ one client  Synchronous I/O  Scheduling: OS/runtime libs int send_data(int fd1, int fd2) { while (!EOF(fd1)) { size = read_chunk(fd, buf, count); write_chunk(fd, buf, size); } …  The event-driven model:  One thread ↔ 10000 clients  Asynchronous I/O  Scheduling: programmer while(1) { nfds=epoll_wait(kdpfd, events, MAXEVT,-1); for(n=0; n<nfds; ++n) handle_event(events[n]); … ThreadsEvents Expressiveness and Abstraction (for programming each client) Synchronous I/O + intuitive control flow primitives Finite state machines / Continuation-passing style (CPS) programming Flexibility and Control (for resource scheduling) Baked into OS/runtime, difficult to customize Programmer has complete control – tailored to each application’s needs “Why threads are a bad idea (for most purposes)” [USENIX ATC 1999] “Why events are a bad idea (for high-concurrency servers)” [HotOS 2003] 

4. 4 Can we get the best of both worlds? The bridge between threads/events? (some kind of “continuation” support) Resource scheduling: events Written as part of the application Tailored to application’s needs Programming with each client: threads Synchronous I/O Intuitive control-flow primitives One application program

5. 5 Roads to lightweight, application-level concurrency  Direct language support for continuations:  Good if you have them  Source-to-source CPS translations  Requires hacking on compiler/runtime  Often not very elegant  Other solutions?  (no language support)  (no compiler/runtime hacks)

6. 6 The poor man’s concurrency monad  “A poor man’s concurrency monad” by Koen Claessen, JFP 1999. (Functional Pearl)  The thread interface:  The CPS monad  The event interface:  A lazy, tree-like data structure called “trace” SYS_NBIO(write_nb)

7. 7 Questions on the poor man’s approach Does it work for high-performance network services? (using a pure, lazy, functional language?)  How does the design scale up to real systems?  Symmetrical multiprocessing? Synchronization? I/O?  How cheap is it?  How much does a poor man’s thread cost?  How poor is it?  Does it offer acceptable performance?

8. 8 Our experiment A high-performance Haskell framework for massively- concurrent network services!!!  Supported features:  Linux Asynchronous IO (AIO)  epoll() and nonblocking IO  OS thread pools  SMP support  Thread synchronization primitives  Applications developed  IO benchmarks on FIFO pipes / Disk head scheduling  A simple web server for static files  HTTP load generator  Prototype of an application-level TCP stack We used the Glasglow Haskell Compiler (GHC)

9. 9 Exception handling Nested function calls Conditional branches Synchronous call to I/O lib Recursion Multithreaded code example

10. 10 Event-driven code example A wrapper function to the C library call using the Haskell Foreign Function Interface (FFI) Put events in queues for processing in other OS threads An event loop running in a separate OS thread

11. 11 A complete event-driven I/O subsystem Haskell Foreign Function Inteface (FFI) Each event loop runs in a separate OS thread One “virtual processor” event loop for each CPU

12. 12 Modular and customizable I/O system (add a TCP stack if you like) Define / interpret TCP syscalls (22 lines) Event loop for incoming packets (7 lines) Event loop for timers (9 lines)

13. 13 How cheap is a poor man’s thread?  Minimal memory consumption: 48 bytes  Each thread just loops and does nothing  Actual size determined by thread-local states  Even an ethernet packet can be >1,000 bytes…  Pay as you go --- only pay for things needed In contrast:  A Linux POSIX thread’s stack has 2 MB by default  The state-of-the-art user-level thread system (Capriccio) use at least a few KBs for each thread Observation: The poor man’s thread is extremely memory-efficient (Challenging most event-driven systems) 48 bytes

14. 14 I/O scalability test  Comparison against the Linux POSIX Thread Library (NPTL)  Highly optimized OS thread implementation  Each NPTL thread’s stack limited to 32KB  Mini-benchmarks used:  Disk head scheduling (all threads running)  FIFO pipe scalability with idle threads (128 threads running)

15. 15 A simple web server

16. 16 How poor is the poor man’s monad?  Not too shabby  Benchmarks shows comparable (if not higher) performance to existing, optimized systems  An elegant design is more important than 10% performance improvement  Added benefit: type safety for many dangerous things  Continuations, thread queues, schedulers, asynchronous I/O

17. 17 Related Work  We are motivated by two projects:  Twisted: the python event-driven framework for scalable internet applications - The programmer must write code in CPS  Capriccio: a high-performance user-level thread system for network servers - Requires C compiler hacks - Difficult to customize (e.g. adding SMP support)  Continuation-based concurrency  [Wand 80], [Shivers 97], …  Other languages and programming models:  CML, Erlang, …

18. 18 Conclusion  Haskell and The Poor Man’s Concurrency Monad are a promising solution for high- performance, massively-concurrent networking applications: Get the best of both threads and events!  This poor man’s approach is actually very cheap, and not so poor! http://www.cis.upenn.edu/~lipeng/homepage/unify.html