Verifying Concurrent Programs with Relaxed Conflict Detection Tayfun Elmas, Ismail Kuru, Serdar Taşıran, Omer Subasi Koç University Istanbul, Turkey

Relaxed Conflict Detection Pattern – Traverse/Take snapshot of (large portion of) global state – Do local computation – Update small portion of global state Very common – Concurrent data structures – Parallelized optimization algorithms – Cloud computing – Performance optimizations for transactional memory Ignore WAR conflicts (Titos et al., HiPEAC ‘12) Early release/early discard (e.g., Kozyrakis et al., WTW ‘06) Performance problem: Most operations conflict Solution: Program so that some conflicts can be ignored Verification problem: How do you reason about correctness? 2

Motivating Example: Sorted Linked List Head 5 16 Insert 5 Insert 16

Linked List: Insert list_insert(list_t *listPtr, node_t *node) { atomic { *curr = listPtr->head; do { prev = curr; curr = curr->next; } while (curr != NULL && curr->key key); node->next = curr; prev->next = node; } 4 Transactional memory: Optimistic concurrency Track read and write accesses a transaction performs – ReadSet(Tx1) – WriteSet(Tx1) Enforce conflict serializability For all concurrent transactions Tx2, check WriteSet(Tx2)  (ReadSet(Tx1)  WriteSet(Tx1)) =  Abort and retry Tx1 otherwise

Motivating Example: Sorted Linked List Head 5 16 Insert 5 Insert 16

Head READ

Head 5 WRITE READ

Write- After- Read conflict Head 16 WRITE READ 5 WRITE

Write- After- Read conflict Head 16 WRITE READ Conventional TM conflict detection enforces conflict serializability – Does WriteSet(Tx1) intersect (ReadSet(Tx1) + WriteSet(Tx2)) ? – Any two concurrent insertions conflict! 5 WRITE

Head 5 16 But, both allowing insertions OK even if we ignore WAR conflict Conflict serializability too strict Options: – Fine-grain, hand-crafted concurrency – Transactional memory + relaxed conflict detection

Relaxed conflict detection Programmer tells TM to ignore this kind of conflict Need to reason that this still results in a correct program Write- After- Read conflict Head 16 WRITE READ 5 WRITE

Linked List: Insert list_insert(list_t *listPtr, node_t *node) { atomic { *curr = listPtr->head; Read do { phase prev = curr; curr = curr->next; } while (curr != NULL && curr->key key); node->next = curr; Write prev->next = node; phase } 12

Linked List: Insert list_insert(list_t *listPtr, node_t *node) { atomic { *curr = listPtr->head; do { prev = curr; curr = curr->next; } while (curr != NULL && curr->key key); node->next = curr; prev->next = node; assert(node is in the list && list is sorted); } 13 Strict conflict detection: Can reason about transaction code sequentially.

Linked List: Insert list_insert(list_t *listPtr, node_t *node) { atomic [!WAR]{ *curr = listPtr->head; do { prev = curr; curr = curr->next; } while (curr != NULL && curr->key key); node->next = curr; prev->next = node; assert(node is in the list && list is sorted); } 14 Ignore Write-After-Read conflicts: Writes by others interfere with transaction Cannot reason sequentially Ignore Write-After-Read conflicts: Writes by others interfere with transaction Cannot reason sequentially READ 6; WRITE 5

Arguing that the !WAR block is atomic list_insert(list_t *listPtr, node_t *node) { atomic [!WAR]{ *curr = listPtr->head; do { prev = curr; curr = curr->next; } while (curr != NULL && curr->key key); node->next = curr; prev->next = node; assert(node is in the list && list is sorted); } 15 Would like these actions to be “right movers” Can commute to the right of any action by another thread Would like these actions to be “right movers” Can commute to the right of any action by another thread

Making !WAR block atomic list_insert(list_t *listPtr, node_t *node) { atomic [!WAR]{ *curr = listPtr->head; do { prev = curr; curr T1 = curr T1 ->next; } while (curr != NULL && curr->key key); node->next = curr; prev->next = node; assert(node is in the list && list is sorted); } 16 node T2 ->next = curr T2 ; READ 6; WRITE 5

Making !WAR block atomic list_insert(list_t *listPtr, node_t *node) { atomic [!WAR]{ *curr = listPtr->head; do { prev = curr; curr T1 = curr T1 ->next; } while (curr != NULL && curr->key key); node->next = curr; prev->next = node; assert(node is in the list && list is sorted); } 17 node T2 ->next = curr T2 ; Ignored WAR conflict: curr T1 = curr T1 ->next; does not move to the right of node T2 ->next = curr T2 ;

Abstraction list_insert(list_t *listPtr, node_t *node) { atomic [!WAR]{ *curr = listPtr->head; do { prev = curr; curr T1 = curr T1 ->next + ; assume prev->next = curr; assume prev->key key } while (curr != NULL && curr->key key); node->next = curr; prev->next = node; assert(node is in the list && list is sorted); } 18 Solution: Abstraction Replace curr T1 = curr T1 ->next; with [curr T1 = curr T1 ->next + ; assume prev->next = curr; assume prev->key key;] Solution: Abstraction Replace curr T1 = curr T1 ->next; with [curr T1 = curr T1 ->next + ; assume prev->next = curr; assume prev->key key;] Now block is atomic. Sequential verification

1. Sequentially verify the original code list_insert(list_t *listPtr, node_t *node) { *curr = listPtr->head; do { prev = curr; curr = curr->next; } while (curr != NULL && curr->key key); node->next = curr; prev->next = node; assert(node is in the list && list is sorted); } 19

2. Apply program transformation list_insert(list_t *listPtr, node_t *node) { *curr = listPtr->head; do { prev = curr; curr = curr->next*; assume (prev->next = curr); assume (prev->key key); } while (curr != NULL && curr->key key); node->next = curr; prev->next = node; assert(node is in the list && list is sorted); } 20 Do global read abstractions  Abstract transaction becomes atomic.

3. Prove abstract code sequentially list_insert(list_t *listPtr, node_t *node) { *curr = listPtr->head; do { prev = curr; curr = curr->next*; assume (prev->next = curr); assume (prev->key key); } while (curr != NULL && curr->key key); node->next = curr; prev->next = node; assert(node is in the list && list is sorted); } 21

Relaxed Conflict Detection Pattern – Traverse/Take snapshot of (large portion of) global state – Do local computation – Update small portion of global state Very common – Concurrent data structures, parallelized optimization algorithms – Cloud computing – Performance optimizations for transactional memory Ignore WAR conflicts (Titos et al., HiPEAC ‘12) Early release/early discard (e.g., Kozyrakis et al., WTW ‘06) Issue: Snapshot may be stale – But programmer thinks this is OK (as long as there is no write-write conflict with another operation) More general problem: – How to verify properties of operations running under a relaxed consistency model 22

WAR Conflict Pattern read x read y read x write x write y 23

WAR Conflict Pattern read x read y read x write x write y 24 WAR conflict OK

WAR Conflict Pattern read x read y read x write x write y 25 WAR conflict OK as long as no WAW conflict

WAR Conflict Pattern read x read y read x write x write y 26 WAR conflict OK as long as no WAW conflict read x read y read x write x write x WAR conflict WAW conflict

WAR Conflict Pattern read x read y read x write x write y 27 WAR conflict OK as long as no WAW conflict read x read y read x write x write x WAR conflict WAW conflict A B O R T

Labyrinth: Grid Routing FindRoute(p1,p2) – Route a shortest wire on the grid connecting points p1 and p2 – Wires must not touch FindRoute operation – Take snapshot of grid – Find shortest path from p1 to p2 – Write path to shared memory if path does not overlap others Overlap = write-write conflict Stale snapshot OK as long as computed path does not overlap others – Same path could have been computed even with up-to-date snapshot 28

Write- After- Read conflict Head 16 WRITE READ 5 WRITE

snapshotGS := GlobSt n ; (x, xVal) := localComp(snapshotGS, args); atomic { GlobSt := GlobSt [x->xVal]; } 30 Desired Interleaving GlobSt 1 := GlobSt 0 [y->yVal]; … GlobSt 2 := GlobSt 1 [z->zVal]; … GlobSt n := GlobSt n-1 [w->wVal]; …

31 Actual Interleaving GlobSt 1 := GlobSt 0 [y->yVal]; … GlobSt 2 := GlobSt 1 [z->zVal]; … GlobSt n := GlobSt n-1 [w->wVal]; … snapshotGS := GlobSt 0 ; snapshotGS := GlobSt n ; (x, xVal) := localComp(snapshotGS, args); atomic { GlobSt := GlobSt [x->xVal]; } This is OK because localComp(GlobSt n, args) and localComp(GlobSt 0,args) would/might have produced same result.

Correctness Argument 32 Blue transaction’s snapshot is stale Blue and green accesses conflict But, end state consistent with –green executed serially, then –blue executed serially Head 16 WRITE READ 5 WRITE

QED: A Proof System for Concurrent Programs Shaz Qadeer Microsoft Research Redmond, WA Serdar Taşıran, Tayfun Elmas, Ali Sezgin Koç University Istanbul, Turkey [POPL ’09, TACAS ‘10, PADTAD ‘10, VSTTE ‘10]

Intuition for proof steps Abstract some actions Prove non-interference Commute Prove larger blocks atomic Abstract some actions Prove non-interference Commute Prove larger blocks atomic

Coarser Atomic Actions check P1P1 PnPn P2P2 Correct Difficult to prove Fine-grain concurrency Annotations at every interleaving point Easy to prove Larger atomic blocks Local, sequential analysis within atomic blocks

36 QED’s Idea of Abstraction  If for all : error s1s1 s1s1 1. If then s1s1 2. Else, if then s2s2 s1s1 s2s2 or error s1s1 s1s1      abstracted by 36

37 Flavors of Abstraction if (x == 1) y := y + 1; if ( * ) y := y + 1; Adding non-determinism Adding assertions (more behaviors that might go wrong) t := x; havoc t; assume x != t; skip; assert (lock_owner == tid); x := t + 1; 37 “Read abstraction”

QED Transformations: Reduction [ S1; S2] [ S1 ] ; [ S2 ] I,PI,P’ 38 If [S1] is a right mover or [S2] is a left mover P P’P’

39 Reduction  ;  ;  ...   1   2 ...   n   ;  ...  right-mover: For each execution: Exist equivalent executions:...     1   2 ...   n      1     2 ...   n      1   2 ...     n   ...  ;  39

Mover check in QED: Static, local, semantic First-order verification condition For each ;... Right-mover ? A AB ; BA B : S1S1 S2S2 S3S3 S1S1 T2T2 S3S3 AB BA All actions in program run by different thread

41 Atomicity Proof Idea GlobSt 1 := GlobSt 0 [y->yVal]; … GlobSt 2 := GlobSt 1 [z->zVal]; … GlobSt n := GlobSt n-1 [w->wVal]; … sshotGS := GlobSt 0 ; (x, xVal) := localComp(sshotGS, args); atomic { GlobSt := GlobSt [x->xVal]; } Need this to be a “Right mover” Must commute to the right of all other concurrent actions Atomicity ensured by strict conflict detection

42 Atomicity Proof Idea GlobSt 1 := GlobSt 0 [y->yVal]; … GlobSt 2 := GlobSt 1 [z->zVal]; … GlobSt n := GlobSt n-1 [w->wVal]; … sshotGS := GlobSt 0 ; (x, xVal) := localComp(sshotGS, args); atomic { GlobSt := GlobSt [x->xVal]; } Need this to be a “Right mover” Must commute to the right of all other concurrent actions Atomicity ensured by strict conflict detection Reasoning using Lipton’s reduction and abstraction QED-like proof (“A Calculus of Atomic Actions”, POPL ’09) Costly pairwise mover checks avoided

Atomicity Proof Idea GlobSt 1 := GlobSt 0 [y->yVal]; … GlobSt 2 := GlobSt 1 [z->zVal]; … GlobSt n := GlobSt n-1 [w->wVal]; … sshotGS := GlobSt 0 ; (x, xVal) := localComp(sshotGS, args); atomic { GlobSt := GlobSt [x->xVal]; } Not a “Right mover” Does not commute to the right of global state updates

Reads and Updates do not Commute READ 6; WRITE WRITE 5; READ 5 Want to read 6 again!

Atomicity Proof Idea GlobSt 1 := GlobSt 0 [y->yVal]; … GlobSt 2 := GlobSt 1 [z->zVal]; … GlobSt n := GlobSt n-1 [w->wVal]; … sshotGS := GlobSt 0 ; (x, xVal) := localComp(sshotGS, args); atomic { GlobSt := GlobSt [x->xVal]; } Not a “Right mover” Does not commute to the right of global state updates Idea: Abstract this read Add non-determinism so it can read GlobSt 1, GlobSt 2, …, GlobSt n

Abstraction​ intuition ABSTRACT READ 6; WRITE WRITE 5; ABSTRACT READ 6 Need to jump over READ 6; WRITE WRITE 5; READ 5 Want to read 6 again!

Abstraction​ intuition ABSTRACT READ 6; WRITE WRITE 5; ABSTRACT READ 6 Need to jump over READ 6; WRITE WRITE 5; READ 5 Want to read 6 again! curr = curr->next; abstracted by curr = curr->next*; but don’t go past key.

Abstracting Read Accesses Abstraction invariant Abs x (l): For variable x, a predicate about its value l – Provided manually (for now) l := x becomes – “Read returns a non-deterministic value consistent with abstraction invariant” For the linked list insertion example Abs node (l) = Reachable(listPtr->head, l)   n  ReachableSet(listPtr->head) \ {l} (l->key < key ) 48

Abstracting Accesses Abstract read accesses: – l := x becomes Annotate write accesses: – x := m becomes – Assertion: Proof obligation, ensures the abstraction invariant By construction, abstract reads commute to the right of annotated writes Assertions become proof obligations – Discharged using sequential reasoning – Abstraction invariant verified when operations are atomic Soundness guaranteed by theorem – Special case of QED soundness theorem – No pairwise mover check needed! 49

Ongoing Work: Obtaining the (Sequential) Specification When interpreted sequentially, the program may already have a specification – Spec already existed for Labyrinth and Genome Specifications (invariants) may be inferred mechanically Done for the Genome benchmark using the Celia tool – Invariant Synthesis for Programs Manipulating Lists with Unbounded Data A. Bouajjani, C. Dragoi, C. Enea, A. Rezine, and M. Sighireanu CAV'10. – On Inter-Procedural Analysis of Programs with Lists and Data A. Bouajjani, C. Dragoi, C. Enea, and M. Sighireanu PLDI'11. 50

Future Work: Obtaining the Abstraction Invariant Sometimes “true” is sufficient Otherwise, need an over-approximation for conflicting_updates* o read_access Genome: curr = curr-> next abstracted by curr = curr->next* “while leaving initial curr->next reachable” Can use annotation inference tool on the sequential code for while (*) conflicting_updates; read_access; 51

Summary Common pattern – Traverse/Take snapshot of (large portion of) global state – Do local computation – Update small portion of global state Conflict serializability too costly – Fine-grain locking – Relaxed conflict detection Static proof approach for such programs – Read abstraction to capture effects of conflicting writes – Enables sequential proof on abstract program Ongoing work: Automation for inferring spec, abstraction invariant 52