1. Verification of architectural memory models by model checking Shaz Qadeer Compaq Systems Research Center Shaz.Qadeer@compaq.com

2. Outline Introduction to shared-memory systems and models Model checking method for verifying models on systems

3. Compiler Multiprocessor Multithreaded program Executable code Language memory model (Java, Modula-3, C with threads)

4. Multiprocessor PPP PPP INTERCONNECT NETWORK

5. Architectural memory model (SC, Alpha, Sun) Multiprocessor

6. Compiler Multiprocessor Multithreaded program Executable code Language memory model (Java, Modula-3, C with threads)

7. Compiler Multithreaded program Executable code Language memory model (Java, Modula-3, C with threads) Architectural memory model (SC, Alpha, Sun)

8. Verification Problem Architectural memory model (SC, Alpha, Sun) Multiprocessor ?

9. Uniprocessor Memory P A := 1; if (B = 0) {... } Initially A = B = 0

10. Shared-memory multiprocessor Memory P A := 1; if (B = 0) {... } Initially A = B = 0 P B := 1; if (A = 0) {... }

11. Memory P W(A, 1) R(B, ?) Initially A = B = 0 P W(B, 1) R(A, ?) Shared-memory model

12. Sequential consistency Memory P W(A, 1) R(B, 1) Initially A = B = 0 P W(B, 1) R(A, 1)

13. Sequential consistency Memory P W(A, 1) R(B, 0) Initially A = B = 0 P W(B, 1) R(A, 1)

14. Sequential consistency Memory P W(A, 1) R(B, 1) Initially A = B = 0 P W(B, 1) R(A, 0)

15. Sequential consistency Memory P W(A, 1) R(B, 0) Initially A = B = 0 P W(B, 1) R(A, 0)

16. Dekkers algorithm Memory P A := 1; if (B = 0) { CS } Initially A = B = 0 P B := 1; if (A = 0) { CS }

17. Interconnect network P1 C1 Memory + Directory P2 C2 state[A] = INVstate[A] = EXC WR_REQ(A) RDEX(A)FWD_RDEX(A, P1)

18. Interconnect network P1 C1 Memory + Directory P2 C2 state[A] = INV WR_REQ(A) RDEX(A)FWD_RDEX(A, P1) RDEX_ACK(A)

19. Interconnect network P1 C1 Memory + Directory P2 C2 state[A] = EXCstate[A] = INV WR_REQ(A) RDEX(A)FWD_RDEX(A, P1) RDEX_ACK(A) WR_ACK(A)

20. Programmers program according to a memory model System must satisfy memory model for software correctness Shared-memory systems are very complex

21. Parameterized shared-memory systems Parameters: processors n, addresses m Memory actions: {R,W} {1,..,n} {1,..,m} Val Internal actions: I {1,..,n} {1,..,m} State transition system: State variables Initial predicate Guarded command for each action

22. State transition system cache: array [1..n] of array [1..m] of {s: State, d: Val} queue: array [1..n] of Queue[m: Msg, a: [1..m]] … (R,i,j,v) [] cache[i][j].s INV cache[i][j].d = v (W,i,j,v) [] cache[i][j].s = EXC cache[i][j].d := v (RRQ,i,j) [] cache[i][j].s = INV queue[i].enqueue(RD_REQ, j) …

23. (EventId, Proc, Addr, Data) (WRQ, 2, 1) (WRP, 2, 1) (W, 2, 1, 1) (R, 1, 1, 0) (R, 2, 1, 1) (WRQ, 1, 1) (WRP, 1, 1) (R, 1, 1, 1) (W, 1, 1, 2) (RRQ, 2, 1) (RRP, 2, 1) (R, 2, 1, 2) Run: finite action sequence executable from initial state

24. Verification problem Impl: state transition system with actions Spec: 1. Invariants, e.g., 1 i, j n. cache[i].s = EXC i j cache[j].s = INV 2. Memory models, e.g., sequential consistency, Alpha memory model Does Impl satisfy Spec?

25. (EventId, Proc, Addr, Data) (WRQ, 2, 1) (WRP, 2, 1) (W, 2, 1, 1) (R, 1, 1, 0) (R, 2, 1, 1) (WRQ, 1, 1) (WRP, 1, 1) (R, 1, 1, 1) (W, 1, 1, 2) (RRQ, 2, 1) (RRP, 2, 1) (R, 2, 1, 2) Run: finite action sequence executable from initial state

26. Memory model (R,1,1,0) (R,1,1,1) (W,1,1,2) (EventId, Proc, Addr, Data) (W,2,1,1) (R,2,1,1) (R,2,1,2) Processor 1Processor 2 n partial orders, one for each processor i th partial order on memory actions at processor i

27. Sequential consistency (R,1,1,0) (R,1,1,1) (W,1,1,2) (EventId, Proc, Addr, Data) (W,2,1,1) (R,2,1,1) (R,2,1,2) Processor 1Processor 2

28. Sequential consistency (EventId, Proc, Addr, Data) (R,1,1,0) (W,2,1,1) (R,2,1,1) (R,1,1,1) (W,1,1,2) (R,2,1,2) 011122011122 Addr 1 (W,2,1,1) (R,1,1,0) (R,2,1,1) (R,1,1,1) (W,1,1,2) (R,2,1,2) swap!

29. Sequential consistency (EventId, Proc, Addr, Data) (R,1,1,0) (W,2,1,1) (R,2,1,1) (R,1,1,1) (W,1,1,2) (R,2,1,2) Witness order System S satisfies Model M iff there is a witness order for every run

30. Debugging vs. Verification McMillan, Schwalbe 91 Clarke et al. 93 Eiriksson, McMillan 95 Ip, Dill 96 Katz, Peled 92 Alur et al. 96 Nalumasu et al. 98 Henzinger et al. 99 Loewenstein, Dill 92 Pong, Dubois 95 Park, Dill 96 Delzanno 00 Graf 94 Henzinger et al. 99 TLA Plakal et al. 98 Invariants Impl Spec Memory models Fixed parameters Arbitrary parameters needed in practice

31. Verifying Memory Models is Hard Alur, McMillan, Peled 96 : Checking sequential consistency for finite parameter values is undecidable.

32. Contribution Model checking algorithm to verify a number of shared-memory models on a useful class of shared-memory systems for finite number of processors and addresses by reduction to invariant verification.

33. Outline Introduction to shared-memory systems and models Model checking method for verifying models on systems

34. State transition system cache: array [1..n] of array [1..m] of {s: State, d: Val} queue: array [1..n] of Queue[m: Msg, a: [1..m]] … (R,i,j,v) [] cache[i][j].s INV cache[i][j].d = v (W,i,j,v) [] cache[i][j].s = EXC cache[i][j].d := v (RRQ,i,j) [] cache[i][j].s = INV queue[i].enqueue(RD_REQ, j) …

35. (EventId, Proc, Addr, Data) (WRQ, 2, 1) (WRP, 2, 1) (W, 2, 1, 1) (R, 1, 1, 0) (R, 2, 1, 1) (WRQ, 1, 1) (WRP, 1, 1) (R, 1, 1, 1) (W, 1, 1, 2) (RRQ, 2, 1) (RRP, 2, 1) (R, 2, 1, 2) Run: finite action sequence executable from initial state

36. Data independence Memory systems do not conjure up data values Data values copied but not examined by actions (except for read and write actions) Every run can be generated from an unambiguous run by suitably renaming data values. (R,1,1,0) (R,1,1,1) (W,1,1,2) (W,2,1,1) (R,2,1,1) (R,2,1,2) Unambiguous run: Suffices to analyze unambiguous runs! (EventId, Proc, Addr, Data)

37. (R,1,1,0) (R,1,1,1) (W,1,1,2) (W,2,1,1) (R,2,1,1) (R,2,1,2) Unambiguous run: Witness write order for address 1 (W,2,1,1) (W,1,1,2) System S satisfies Model M iff for every run there are witness write orders for all addresses acyclic graph witness order (EventId, Proc, Addr, Data)

38. Recipe for verification For every unambiguous run, 1. guess write order for each address 2. generate graph and check for cycles

39. Interconnect network P1 C1 Memory + Directory P2 C2 state[A] = EXCstate[A] = INV WR_REQ(A) RDEX(A)FWD_RDEX(A, P1) RDEX_ACK(A) WR_ACK(A)

40. Simple write order For each location, order write events according to actual occurrence order !!

41. Examples Piranha chip multiprocessor (Compaq) Wildfire challenge problem (Compaq) DASH multiprocessor (Stanford)

42. Recipe for verification For every unambiguous run, 1. guess write order for each address simple write order 2. generate graph and check for cycles

44. ... (*,i,z,*)... (*,i,x,*)... (*,j,x,*)... (*,j,y,*)... (*,k,y,*)... (*,k,z,*)... Nice cycles... 3-nice cycle: 3 processors i, j, k 3 addresses x, y, z k-nice cycle involves k processors and k addresses (EventId, Proc, Addr, Data)

45. Nice cycles S has a cycle iff S has a k-nice cycle for 1 k min(n,m) S: memory system with n processors, m addresses Lemma:

46. Recipe for verification For every unambiguous run, 1. guess write orders for each address simple write order 2. generate graph and check for cycles reduce to nice cycles detecting nice cycles by model checking

47. Detecting nice cycles min(n,m) model checking lemmas: kth lemma checks for k-nice cycles S has a cycle iff S has a k-nice cycle for 1 k min(n,m) S: memory system with n processors, m addresses Lemma:

48. Sup j supplies write values for address j. Mon i monitors memory events at processor i. Memory system Sup 1 Mon 1 Model checker Property = i. Mon i @err 1-nice cycle Detecting nice cycles

49. Memory system Sup 1 Sup 2 Mon 1 Mon 2 Model checker Property = i. Mon i @err Detecting nice cycles 2-nice cycle Sup j supplies write values for address j. Mon i monitors memory events at processor i.

50. Memory system Sup 1 Sup 2 Sup 3 Mon 1 Mon 2 Mon 3 Model checker Property = i. Mon i @err Detecting nice cycles 3-nice cycle Sup j supplies write values for address j. Mon i monitors memory events at processor i.

51. ... (*,i,z,*)... (*,i,x,*)... (*,j,x,*)... (*,j,y,*)... (*,k,y,*)... (*,k,z,*)... Nice cycles... 3-nice cycle: 3 processors i, j, k 3 addresses x, y, z (EventId, Proc, Addr, Data)

52. Unambiguous run: (R,1,1,0) (R,1,1,1) (W,1,1,2) (W,2,1,1) (R,2,1,1) (R,2,1,2) (EventId, Proc, Addr, Data) Witness write order for address 1 (W,2,1,1) (W,1,1,2) Causal edges Anti-causal edges

53. Supplier automata Supplies 0 upto some nondeterministically chosen write and then supplies 1 forever. 0 1 1 Sup x (supplier for address x):

54. ... (*,i,z,1)... (*,i,x,0)... (*,j,x,1)... (*,j,y,0)... (*,k,y,1)... (*,k,z,0)... Nice cycles... 3-nice cycle: 3 processors i, j, k 3 addresses x, y, z (EventId, Proc, Addr, Data)

55. Monitor automata Mon j : (*,j,x,1) (*,j,y,0) err

56. Analysis for sequential consistency Number of processors = n Number of addresses = m Number of model checking runs = min(n,m) Number of Sup automata = k Number of Mon automata = k For all j, number of states in Sup j = 3 For all i, number of states in Mon i = 3 States in model checked system = |S| 3 k 3 k kth model checking run (1 k min(n,m)):

57. Model checker Property = i. Mon i @err Other memory models? Alpha memory model, partial store order, release consistency, weak ordering Memory system Sup 1 Sup 2 Sup 3 Mon 1 Mon 2 Mon 3

58. Model checker Property = i. Mon i @err Other write orders? Can be generalized !! Memory system Sup 1 Sup 2 Sup 3 Mon 1 Mon 2 Mon 3

59. Wildfire challenge problem 2 processor, 2 location system –Supplier and monitor automata constructed –Seeded bug was found by invariant verification on composed system by model checker TLC

60. Summary Model checking algorithm to verify a number of shared-memory models on a useful class of shared-memory systems for finite number of processors and addresses by reduction to invariant verification.