Language Support for Fast and Reliable Message-based Communication in Singularity Manuel Fähndrich, Mark Aiken, Chris Hawblitzel, Orion Hodson, Galen Hunt, James R. Larus, and Steven Levi Microsoft Research EUROSYS 2006

Manuel FähndrichMicrosoft Research2 Singularity Research goal –Leverage state of the art to build more reliable software High-level languages, memory and type safety Advanced type systems, formal verification Optimizing compilers and typed assembly language –Don’t sacrifice performance, unless safety requires it The Singularity OS –Micro-kernel architecture separate processes for services (drivers) processes and kernel communicate solely via channels no shared memory –Ideal research platform –Written in Sing# (a C# extension) This talk: –Programming language support for Singularity’s process isolation

Manuel FähndrichMicrosoft Research3 VolMgr Process Isolation Isolation guarantee stems from static memory safety –No hardware protection –All code runs in ring0 on x86 processor –Each process has own heap + garbage collector Channels are sole form of communication –point-to-point –serve as APIs (default Kernel API is minimal) Kernel DiskDevFS NS App

Manuel FähndrichMicrosoft Research4 Challenges Pervasive message passing –Difficult programming model What’s the protocol? Are we sending the right message at the right time? Are we handling all messages? –Perceived as expensive Copying of message content between processes limits performance –Situation worse in Singularity due to fine grained isolation Queue, memory management Cost of using a high-level language and GC –Another talk

Manuel FähndrichMicrosoft Research5 Sing#: Easy message passing Know thy protocol –Explicit channel contracts describe messages message arguments valid message sequences (finite state machine) –Enables compile-time checking exhaustiveness of selects sends in correct state finite buffering requirement Pass ownership of message data, not a copy … but prove its correctness Direct support for channels in programming language

Manuel FähndrichMicrosoft Research6 Outline Motivation Channels Data ownership Implementation Performance Conclusion

Manuel FähndrichMicrosoft Research7 Channels Consist of two endpoints (point-to-point) Each endpoint owned by at most one thread at any time Communication is asynchronous, delivery in order –Sends never block (nor fail) –Receives can block (or indicate channel closure) Endpoints can be passed as messages –Network configuration evolves dynamically Governed by channel contract –Messages –Message arguments –Valid message sequences

Manuel FähndrichMicrosoft Research8 Channel model in pictures Kernel Name Server I III II create channel Kernel IV spawn

Manuel FähndrichMicrosoft Research9 P1 Kernel creates a process I IIIIV II Kernel Name Server Kernel Name Server Kernel Name Server send a over b a b Kernel Name Server spawn

Manuel FähndrichMicrosoft Research10 Channel contracts contract NIC { // Network device interface in message NICParameters(…); in message Buffer(char[] in ExHeap buf); in message Reconfigure(); in message Exit(); out message BufferAck(int count); out message Configured(); out message BadParameters(); state START: one { NICParameters?  CONFIGURING; } state CONFIGURING: one { Configured!  READY; BadParameters!  START; } state READY: one { Buffer?  BufferAck!  READY; Reconfigure?  START; Exit?  DONE; } state DONE: ; } START CONFREADY DONE RDY$0 NICParameters? BadParameters! ReConfigure? Configured! Exit? BufferAck! Buffer?

Manuel FähndrichMicrosoft Research11 Channel operations Endpoint types NIC.Imp (importing side) NIC.Exp (exporting side) Channel creation NIC.NewChannel( out Nic.Imp imp, out Nic.Exp exp); Sends are methods on endpoints Receive patterns block until a case is satisfied NIC.Exp:READY client; ServiceProvider.Exp:START provider; … switch receive { case client.Buffer(buf): queue.Add(buf); client.SendBufferAck(queue.Size); break; case client.Reconfigure(): … case client.ChannelClosed() && provider.ChannelClosed(): … break; … }

Manuel FähndrichMicrosoft Research12 B Ownership transfer Only exchangeable data can be sent Ownership invariant –Exchangeable data is owned by exactly one thread at any time. Transfer by pointer, not copy (but semantically equiv.) –Proof obligations No use after ownership transfer (e.g. free or send) No leaks (missing free/send) –Like manual memory management –Compiler checks invariant statically A A B

Manuel FähndrichMicrosoft Research13 Tracked Data Challenge: statically prove correct memory management Segregate tracked and non-tracked data Tracked types –ExHeap pointers: S* in ExHeap –ExHeap vectors: char[] in ExHeap –rep structs S: do not contain GC pointers Strongly typed at MSIL level –allows runtime bounds checking for vectors

Manuel FähndrichMicrosoft Research14 Ownership Tracking Method parameters –either borrowed for the call duration void FillBuffer(byte[] in ExHeap b); –or claimed void SendBuffer([Claims] byte[] in ExHeap b); void Free([Claims] void* in ExHeap b); Method result –ownership always passes to caller Complications –Data structures and vectors –Nested pointers owned by containing structure –Restricted to tree structures to make tractable

Manuel FähndrichMicrosoft Research15 Where are the efficiency gains? Channel endpoint invariants –Single owner: no locking on send/receives Channel contract invariants –Finite queue size: no allocation on send or receive –Endpoint passing: no complicated locking/races Data ownership invariants –No copying of data: pass by pointers Programmer productivity –Simple message passing model and contract enforcement

Manuel FähndrichMicrosoft Research16 Channel Implementation Pre-allocate space to hold message arguments in endpoints Channel= cross-linked structure in memory Send= write into peer + signal event Receive= check tag + read arguments CStates state; C.Exp* peer; M mbuffer0; M mbuffer1; N nbuffer0; CTags tag0; CTags tag1; … C.Imp CStates state; C.Imp* peer; … C.Exp

Manuel FähndrichMicrosoft Research17 Message size benchmark

Manuel FähndrichMicrosoft Research18 Summary and Future Work Singularity uses message passing pervasively –So far, channels have not been our performance bottle neck –Programmers have stopped complaining Channel contracts help specify interfaces > 43 distinct channel contracts with 135 distinct states > 500 distinct messages 146 messages transferring pointers 42 messages transferring endpoints Static verification of ownership invariant successful –Dangling pointer accesses, double frees or leaks not been a problem. Future work –Ownership and protocol verification on exception paths –Programmer defined asynchronous abstractions –Pre-condition, post-conditions and invariant verification

Manuel FähndrichMicrosoft Research19 Questions?

Manuel FähndrichMicrosoft Research20 Backups

Manuel FähndrichMicrosoft Research21 Transfer of ownership Transfer by pointer, not copy (but semantically equiv.) Proof obligations –No use after ownership transfer (e.g. free or send) –No leaks (missing free/send) Like manual memory management Process Heaps (GC) Exchange Heap (manual)

Manuel FähndrichMicrosoft Research22 Transfer of ownership Transfer by pointer, not copy (but semantically equiv.) Proof obligations –No use after ownership transfer (e.g. free or send) –No leaks (missing free/send) Like manual memory management Process Heaps (GC) Exchange Heap (manual)

Manuel FähndrichMicrosoft Research23 Endpoint Abstractions ESet and EMap Sets of endpoints in the same state Can add endpoint to sets void ESet.Add([Claims] E ep); Can wait for messages on any endpoint in a set ESet eset; switch receive { case ep.M(x) in eset: … // ep and x bound

Manuel FähndrichMicrosoft Research24 P1 2 processes connect I IIIIV II Kernel Name Server P2 P1 Kernel Name Server P2 P1 Kernel Name Server P2 P1 Kernel Name Server P2