User-Level Interprocess Communication for Shared Memory Multiprocessors Brian N. Bershad, Thomas E. Anderson, Edward D. Lazowska, and Henry M. Levy Presented By: Yahia Mahmoud

Yahia Mahmoud - CS5332 Introduction  IPC is central to OS design  Encourages decomposition across address space boundaries Failure Isolation – no leaking Extensibility – adding modules dynamically Modularity  Slow IPC => trade of between performance and decomposition

Yahia Mahmoud - CS5333 Problems  IPC has been kernel responsibility – two problems: Architectural performance barriers  Overhead of Kernel-mediated cross address space call – 70% of LRPC overhead Interaction with user-level threads  Partitioning of communication and thread management across protection boundaries has high performance cost.

Yahia Mahmoud - CS5334 Solution  Eliminate kernel from cross-address space communication  Use shared memory as data transfer channel  Avoid processor reallocation – use already active processor in target address space  This approach results in improved performance because…

Yahia Mahmoud - CS5335 Advantages  Messages sent without kernel involved  Avoid processor reallocation. Reduce call overhead and preserve cache  Processor reallocation overhead can be amortized over several calls  Exploit the parallelism of send and receive of messages

Yahia Mahmoud - CS5336 URPC  Messages are passed through logical memory channels – created and mapped for every client/server  User-level Thread management – no kernel involved in messages sent

Yahia Mahmoud - CS5337 Software Components

Yahia Mahmoud - CS5338 URPC  Separates IPC into three components 1. Data transfer 2. Thread management 3. Processor reallocation  Goals Move 1 and 2 into user-level 3 needs the kernel but try to avoid it. Why?  Scheduling cost to decide which address space runs on the processor  Virtual memory mapping costs  Long-term costs because of cache and TLB invalidated

Yahia Mahmoud - CS5339 URPC  Calls appear synchronous to programmer but asynchronous beneath the system  Client thread blocks and another ready thread runs  LRPC used the same blocked thread to run the ready thread  URPC always tries to schedule another thread Avoid context switching overhead  If load balancing is needed client lends its processor to the server via kernel and server returns back after processing messages

Yahia Mahmoud - CS53310 Processor Reallocation  Kernel uses pessimistic reallocation (handoff scheduling)  This policy does not always improve performance  Kernel centralized data structure creates performance bottleneck (lock contention, thread run queues and message channels)

Yahia Mahmoud - CS53311 Processor Reallocation  Use Optimistic reallocation policy The client has other work to do – no performance side effect of delaying message processing at server side => inexpensive context switch Server is not underpowered – has or will have processor to process message => client executes in parallel with server  This won’t hold in case of time sensitive service is needed (real time system, high latency I/O ops)  In case of reallocation is needed it’s done via kernel Idle processor can donate itself to underpowered address space The identity of the donating processor is made known to the receiver No guarantee that processor will be returned back to donor

Yahia Mahmoud - CS53312 Example

Yahia Mahmoud - CS53313 Data Transfer  Arguments can be passed using shared memory and still guarantee safety  Stubs are responsible for communication safety On receipt of a message, stubs unmarshal the data into parameters and do the needed copying and checking to ensure application safety  No need to use kernel stubs can do the copying directly Data are passed on stack or heap and stubs copy them directly In case of type safe language copying into kernel does not guarantee type checking of data  Shared memory queues are controlled with test-and-set locks with no spinning

Yahia Mahmoud - CS53314 Thread Management  Fine-grained parallel application needs high performance thread management which could only be achieved by implementing in user-level  Communication & Thread management can achieve very good performances when both are implemented at user-level  Threads block in order to: Synchronize their activities in same address space Wait for external events from different address space  Communication implemented at kernel level will result in synchronization at both user level and kernel level

Yahia Mahmoud - CS53315 Performance

Yahia Mahmoud - CS53316 Call Latency and Throughput  Call Latency is the time from which a thread calls into the stub until control returns from the stub.  These are load dependent, and depend on Number of Client Processors (C) Number of Server Processors (S) Number of runnable threads in the client’s Address Space (T)  The graphs measure how long it takes to make 100,000 “Null” procedure calls into the server in a “tight loop”

Yahia Mahmoud - CS53317 Call Latency and Throughput

Yahia Mahmoud - CS53318 Conclusions  Performance gains from moving features out of the kernel not vice-versa  URPC represents appropriate division for operating system kernels of shared memory multiprocessors  URPC showcases a design specific to a multiprocessor, not just uniprocessor design that runs on multiprocessor hardware