1. Read-Copy-Update Synchronization in the Linux Kernel 1 David Ferry, Chris Gill CSE 522S - Advanced Operating Systems Washington University in St. Louis St. Louis, MO 63143

2. The Synchronization Problem A common kernel problem: Multiple threads share a data structure. – Some are reading – Some are writing Reads and writes should not interfere! CSE 522S – Advanced Operating Systems2 Shared data

3. Synchronization Design Tradeoffs All synchronization methods need to balance the needs of readers and writers: Reads tend to be more common CSE 522S – Advanced Operating Systems3 Lots of reads Few writes Balanced Reads and writes Lots of writes Few reads Synchronization can prevent concurrency… Reader/writer locks Mutual exclusion Or it can allow concurrency at the expense of overheads: Lock free / wait free algorithms Transactional memory

4. RCU Philosophy Under RCU: – Concurrent reads are synchronization-free (which means scalability!) – Writers must guarantee that all readers only ever see a consistent view of memory – Similar to a publish-subscribe model Before the details, let’s look at the API… CSE 522S – Advanced Operating Systems4

5. RCU Writer API Even if pointer write is atomic: 1struct foo *ptr = NULL; 2p = kmalloc(...); 3p->A = 1; 4p->B = 2; 5p->C = 3; 6ptr = p; Overall code is not safe! Compiler may re-order lines 3-6 CSE 522S – Advanced Operating Systems5

6. RCU Writer API RCU encapsulates memory fences 1struct foo *ptr = NULL; 2p = kmalloc(...); 3p->A = 1; 4p->B = 2; 5p->C = 3; 6rcu_assign_ptr(ptr,p); rcu_assign_ptr method publishes P CSE 522S – Advanced Operating Systems6

7. RCU Reader API Consider reading a data structure: p = ptr; if (p != NULL) do_something(p->A, p->B, p->C); This is also not safe! The values of A, B, and C could change between reads! CSE 522S – Advanced Operating Systems7

8. RCU Reader API Consider reading a data structure: rcu_read_lock(); p = rcu_dereference(ptr); if (p != NULL) do_something(p->A, p->B, p->C); rcu_read_unlock(); rcu_dereference() can be thought of as subscribing to a specific, valid version of ptr lock/unlock defines RCU critical section CSE 522S – Advanced Operating Systems8

9. RCU Encapsulation Note, RCU semantics can be encapsulated for specific data structures: rcu_list_add() rcu_for_each_read() But not: rcu_for_each_write() RCU does not allow for concurrent writes! CSE 522S – Advanced Operating Systems9

10. What does RCU actually do? CSE 522S – Advanced Operating Systems10 CreationRemovalGrace PeriodReclamation Time RCU Read Critical Section

11. wait_for_readers() CSE 522S – Advanced Operating Systems11 CreationRemoval RCU Read Critical Section Grace PeriodReclamation Time A critical implementation detail is how to wait for outstanding reads. Explicit: reference counting reader locks Implicit: Linux context switch

12. RCU Linked List - Delete CSE 522S – Advanced Operating Systems12 H Ptr H H D D D 1. Want to delete node D 2. Atomically assign pointer over D 3. D is safe to delete when all reader locks are done

13. RCU Linked List - Update CSE 522S – Advanced Operating Systems13 H P H P Q H P Q M H P Q M H Q M H M 1 2 3 4 5 6

14. RCU usage in sched/core.c CSE 522S – Advanced Operating Systems14 RCU-iterates over list of processes Writes back to RCU-protected structure