1. USER LEVEL INTERPROCESS COMMUNICATION FOR SHARED MEMORY MULTIPROCESSORS Presented by Elakkiya Pandian CS 533 OPERATING SYSTEMS – SPRING 2011 Brian N. Bershad Carnegie Mellon University Thomas E Anderson, Edward D. Lazowska, and Henry M.Levy University of Washington © ACM Transactions on Computer Systems, 9, 2, pgs. 175-198, May 1991

2. 2 QUESTIONS ANSWERED How to bypass kernel from the path of cross address space communication in shared memory multiprocessors and why? How does URPC mechanism work and what are the motivations and goals? What is the design rationale behind URPC? How does URPC perform compared to other approaches discussed earlier?

3. 3 OUTLINE OF THE PAPER Motivations for URPC Goals of URPC Design of URPC -Processor Reallocation -Data Transfer -Thread Management Performance of URPC Conclusion

4. 4 MOTIVATIONS FOR URPC Efficient communication facility encourages decomposition across address spaces. Why? Fault isolation, Extensibility, Modularity Problem: Traded away for “system performance” Solution : Efficient cross address space communication reducing overhead LRPC!! But why URPC? Improve performance further with Shared memory multiprocessors and avoid kernel intervention overhead

5. 5 MOTIVATIONS FOR URPC Exploiting the advantage of shared memory multiprocessor for IPC to avoid kernel based IPC issues Cost of invoking kernel, processor reallocation Eliminate Kernel intervention, use already active processor in target address space Cost of Thread Communication Use Shared memory as communication channel Cost of Separation of Communication and thread management Manage cross address space communication at user level along with threads

6. 6 MOTIVATIONS OF URPC Key Advantage – Doesn’t invoke kernel in normal case (Optimistic Allocation Policy) Context Switching – T1 blocks but K1 never blocks! No Processor Reallocation

7. 7 MOTIVATIONS FOR URPC Multiprocessor versus Uniprocessor URPC can be used for multi-threaded applications, it can reschedule within the process as much as possible and delay the processor reallocation Reallocation cannot be avoided as there is no other processor to communicate or work in parallel

8. 8 OUTLINE OF THE PAPER Motivations for URPC Goals of URPC Design of URPC -Processor Reallocation -Data Transfer -Thread Management Performance of URPC Conclusion

9. 9 GOALS OF URPC Appropriate Division of responsibilities for performance and flexibility to obtain the advantages mentioned. Components of IPC Processor Reallocation Thread Management Data Transfer - Communication KERNEL Application Level Libraries Application Level Libraries

10. 10 OUTLINE OF THE PAPER Motivations for URPC Goals of URPC Design of URPC -Processor Reallocation -Data Transfer -Thread Management Performance of URPC Conclusion

11. 11 DESIGN OF URPC – Sample Execution T1 Blocks Underpowered Server! Load Balancing Problem Underpowered Server! Load Balancing Problem T2 Blocks Client Processor Reallocation Client Processor Reallocation

12. 12 DESIGN OF URPC - Processor Reallocation Processor Reallocation: Context switching involves less overhead than processor reallocation - On C-Vax Context switching takes 15 µsecs and processor reallocation takes 55 µsecs. Avoiding Processor reallocation not always possible. Why? Load Balancing Problem! Solution?? Processor Reallocation – An idle processor in client reallocates itself, after processing it is returned to client via kernel.

13. 13 DESIGN OF URPC - Processor Reallocation How to reduce frequency of processor reallocation? Optimistic Reallocation Policy! Assume client has other runnable threads Assume server has (or will soon have) a processor to service a message Amoritization

14. 14 DESIGN OF URPC - Processor Reallocation Optimistic assumptions won’t always hold. When do they fail? Single threaded and real-time applications, high latency I/O operations and priority invocations. How does URPC handle this? Client’s address space can force reallocate to server’s Kernel using Processor.Donate procedure Processor is returned when all outstanding messages from the client has been serviced or when processor determines client to be underpowered.

15. 15 DESIGN OF URPC - Processor Reallocation Voluntary return of processors cannot be guaranteed There is no way to enforce the protocol – eg. Server could use processor to service other clients Effects?? Solution?? Preemptive policies that forcibly reallocate processors Direct Processor Reallocation – decision made by client

16. 16 OUTLINE OF THE PAPER Motivations for URPC Goals of URPC Design of URPC -Processor Reallocation -Data Transfer -Thread Management Performance of URPC Conclusion

17. 17 DESIGN OF URPC – Data Transfer Interacts with user level and Stubs (Synchronous calls and Asynchronous messages) Interacts with user level and Stubs (Synchronous calls and Asynchronous messages) Data flows between URPC packages in different address spaces over a bi-directional shared memory queue

18. 18 DESIGN OF URPC – Data Transfer As with RPC, “abusability factor” still exists. In URPC, safety of communication based on shared memory is responsibility of stubs  “Pair-wise shared memory can be used to transfer data between address spaces more efficiently but as safely as messages copied by kernel.” TSET non-spinning locks on either end  Prevents processors from waiting indefinitely  Receiver should never delay when sender holds the lock

19. 19 OUTLINE OF THE PAPER Motivations for URPC Goals of URPC Design of URPC -Processor Reallocation -Data Transfer -Thread Management Performance of URPC Conclusion

20. 20 DESIGN OF URPC – Thread Management Calling semantics of procedure call are synchronous with respect to thread that is performing the call  Strong interaction between system that manages threads and communication  Communication functions have corresponding thread management synchronization function ( send - start, receive – stop) Relationship between Communication and Thread Management  Efficient thread management is necessary for parallel programs and it can be provided at user level  Close interaction between the two should be exploited to achieve extremely good performance for both

21. 21 DESIGN OF URPC – Thread Management Light-weight Thread Management implies two level scheduling – application level scheduling lightweight threads on top of weightier threads scheduled by the kernel Communication and threads should both be implemented at user level. Why? Cost of Synchronization between activities in different address spaces

22. 22 OUTLINE OF THE PAPER Motivations for URPC Goals of URPC Design of URPC -Processor Reallocation -Data Transfer -Thread Management Performance of URPC Conclusion

23. 23 PERFORMANCE OF URPC 1.Comparison of thread management operations implemented at user level and kernel level 2. Influence of thread management overhead on communication performance

24. 24 PERFORMANCE OF URPC 1. Call Latency and throughput are load dependent - Number of server processors (S), number of client processors (C) and number of runnable threads in client address space (T) 2. For different values of C,S and T, 100,000 null procedure calls into server on a tight loop was timed and recorded. 3. When T> S+C, latency increases but throughput is less sensitive, it improves only until T>= S+C 4. Worst Case S=0, LRPC performs better than URPC in this case.

25. 25 PERFORMANCE OF URPC Comparison with LRPC LRPC reduces the amount of thread management but is kernel mediated communication which is an overhead. URPC performs better Like LRPC, URPC also passes parameters between shared memory that is pair-wise mapped between client and server In the worst case when there are no processors in the server space, LRPC performs better. This is attributed to 2 reasons - URPC was implemented on existing LRPC and it is integrated with two-level scheduling

26. 26 OUTLINE OF THE PAPER Motivations for URPC Goals of URPC Design of URPC -Processor Reallocation -Data Transfer -Thread Management Performance of URPC Conclusion

27. 27 CONCLUSION URPC solves the problem of kernel based communication by moving thread management and communication out of the kernel by implementing it at user level It is found to perform better than other traditional systems and LRPC with shared memory multiprocessors

28. 28 REFERENCES Operating System Concepts, 7th edition, Silberschatz, Galvin and Gagne "Lightweight Remote Procedure Call" by B. N. Bershad, T. E. Anderson, E. D. Lazowska, and H. M. Levy, Proceedings of the 12th Symposium on Operating Systems Principles, pp. 102-113, December 1989.

29. 29 Thank You