1. 1 Lecture 07 – Shape Analysis Eran Yahav

2. Previously  LFP computation and join-over-all-paths  Inter-procedural analysis  call-string approach  functional approach 2

3. Today  Shape Analysis  Typestate Verification 3

4. Shape Analysis Automatically verify properties of programs manipulating dynamically allocated storage Identify all possible shapes (layout) of the heap 4

5. Timeline: Shape Analysis Parametric Shape Analysis via 3-valued Logic 199920012007 Verifying Concurrent Heap Manipulating Programs Verifying Linearizability Interprocedural / Recursive Programs 200520002002200320042006 Numerical Abstractions Logical Char. of Heap Abstractions Procedure Local Heaps 2008 Verifying Linearizability with Heap Decomposition Thread Modular Shape Analysis Flow analysis and optimization of Lisp-like structures 1981 Analysis of pointers and structures 1990 A Local Shape Analysis Based on Separation Logic 5

6. Timeline: Shape Analysis via 3-Valued Logic Parametric Shape Analysis via 3-valued Logic [Sagiv, Reps, Wilhelm POPL’99,TOPLAS’02] 199920012007 Verifying Concurrent Heap Manipulating Programs [Yahav, Sagiv, POPL’01] Verifying Linearizability [Amit et.al., CAV’07] Interprocedural / Recursive Programs 200520002002200320042006 Numerical Abstractions Logical Char. of Heap Abstractions Procedure Local Heaps 2008 Verifying Linearizability with Heap Decomposition Thread Modular Shape Analysis A Local Shape Analysis Based on Separation Logic 6

7. Sequential Stack void push (int v) { Node  x = malloc(sizeof(Node)); x->d = v; x->n = Top; Top = x; } int pop() { if (Top == NULL) return EMPTY; Node *s = Top->n; int r = Top->d; Top = s; return r; } Want to Verify No Null Dereference Underlying list remains acyclic after each operation 7

8. Shape Analysis via 3-valued Logic 1) Abstraction  3-valued logical structure  canonical abstraction 2) Transformers  via logical formulae  soundness by construction  embedding theorem, [SRW02] 8

9. Concrete State  represent a concrete state as a two-valued logical structure  Individuals = heap allocated objects  Unary predicates = object properties  Binary predicates = relations  parametric vocabulary Top nn u1u2u3 (storeless, no heap addresses) 9

10. Concrete State  S = over a vocabulary P  U – universe   - interpretation, mapping each predicate from p to its truth value in S Top nn u1u2u3 10  U = { u1, u2, u3}  P = { Top, n }   (n)(u1,u2) = 1,  (n)(u1,u3)= 0,  (n)(u2,u1)= 0,…   (Top)(u1)= 1,  (Top)(u2)= 0,  (Top)(u3)= 0

11.  v1,v2: n(v1, v2)  n*(v2, v1) void push (int v) { Node  x = malloc(sizeof(Node)); x  d = v; x  n = Top; Top = x; } Formulae for Observing Properties Top nn u1u2u3 1 No Cycles  v1,v2: n(v1, v2)  n*(v2, v1) 1  w:Top(w) Top != null  w: x(w) No node precedes Top  v1,v2: n(v1, v2)  Top(v2) 1 11

12. Concrete Interpretation Rules StatementUpdate formula x =NULLx’(v)= 0 x= malloc()x’(v) = IsNew(v) x=yx’(v)= y(v) x=y  nextx’(v)=  w: y(w)  n(w, v) x  next=yn’(v, w) = (  x(v)  n(v, w))  (x(v)  y(w)) 12

13. Example: s = Topn Top s ’ (v) =  v1: Top(v1)  n(v1,v) u1 u2u3 Top s u1 u2u3 Top u1 1 u2 0 u3 0 nu1u2U3 u1 010 u2 001 u3 000 s u1 0 u2 0 u3 0 Top u1 1 u2 0 u3 0 nu1u2U3 u1 010 u2 001 u3 000 s u1 0 u2 1 u3 0 13

14. Collecting Semantics 14  {  st(w)  (S) | S  CSS[w] }  (w,v)  E(G), w  Assignments(G) { } CSS [v] = if v = entry  { S | S  CSS[w] }  (w,v)  E(G), w  Skip(G)  { S | S  CSS[w] and S  cond(w)}  (w,v)  True-Branches(G)  { S | S  CSS[w] and S   cond(w)} (w,v)  False-Branches(G) othrewise

15. Collecting Semantics  At every program point – a potentially infinite set of two-valued logical structures  Representing (at least) all possible heaps that can arise at the program point  Next step: find a bounded abstract representation 15

16.  1 = true  0 = false  1/2 = unknown  A join semi-lattice, 0  1 = 1/2 3-Valued Logic 1/2 0 1 information order logical order 16

17. 3-Valued Logical Structures  A set of individuals (nodes) U  Relation meaning  Interpretation of relation symbols in P p 0 ()  { 0, 1, 1/2 } p 1 (v)  { 0, 1, 1/2 } p 2 (u,v)  { 0, 1, 1/2 }  A join semi-lattice: 0  1 = 1/2 17

18. Boolean Connectives [Kleene]

19. Property Space  3-struct[P] = the set of 3-valued logical structures over a vocabulary (set of predicates) P  Abstract domain   (3-Struct[P])   is   We will see alternatives later (maybe) 19

20. Embedding Order  Given two structures S =, S’ = and an onto function f : U  U’ mapping individuals in U to individuals in U’  We say that f embeds S in S’ (denoted by S  f S’) if  for every predicate symbol p  P of arity k: u1, …, uk  U,  (p)(u1,..., uk)   ’(p)(f(u1),..., f (uk))  and for all u’  U’ ( | { u | f (u) = u’ } | > 1)   ’(sm)(u’)  We say that S can be embedded in S’ (denoted by S  S’) if there exists a function f such that S  f S’ 20

21. 21

22. Tight Embedding  S’ = is a tight embedding of S= with respect to a function f if:  S’ does not lose unnecessary information  ’(u’ 1,…, u’ k ) =  {  (u 1..., u k ) | f(u 1 )=u’ 1,..., f(u k )=u’ k }  One way to get tight embedding is canonical abstraction 22

23. Canonical Abstraction Top nn u1u2u3 u1u2u3 [Sagiv, Reps, Wilhelm, TOPLAS02] 23

24. Canonical Abstraction Top nn u1u2u3 u1u2u3 Top [Sagiv, Reps, Wilhelm, TOPLAS02] 24

25. Canonical Abstraction Top nn u1u2u3 u1u2u3 Top 25

26. Canonical Abstraction Top nn u1u2u3 u1u2u3 Top 26

27. Canonical Abstraction Top nn u1u2u3 u1u2u3 Top 27

28. Canonical Abstraction Top nn u1u2u3 u1u2u3 Top 28

29. Canonical Abstraction (  )  Merge all nodes with the same unary predicate values into a single summary node  Join predicate values  ’(u’ 1,..., u’ k ) =  {  (u 1,..., u k ) | f(u 1 )=u’ 1,..., f(u k )=u’ k }  Converts a state of arbitrary size into a 3-valued abstract state of bounded size   (C) =  {  (c) | c  C } 29

30. Information Loss Top nn nn n Canonical abstraction Top n n 30

31. Instrumentation Predicates  Record additional derived information via predicates r x (v) =  v1: x(v1)  n*(v1,v) c(v) =  v1: n(v1, v)  n*(v, v1) Top nn c c nn n c Canonical abstraction 31

32. Embedding Theorem: Conservatively Observing Properties No Cycles  v1,v2: n(v1, v2)  n*(v2, v1) 1/2 Top r Top No cycles (derived)  v:  c(v) 1 32

33. Operational Semantics void push (int v) { Node  x = malloc(sizeof(Node)); x->d = v; x->n = Top; Top = x; } int pop() { if (Top == NULL) return EMPTY; Node *s = Top->n; int r = Top->d; Top = s; return r; } s ’ (v) =  v1: Top(v1)  n(v1,v)  s = Top->n  Top nn u1u2u3 Top nn u1u2u3 s 33

34. Abstract Semantics Top s = Top  n ? r Top Top r Top s s ’ (v) =  v1: Top(v1)  n(v1,v)  s = Top->n  34

35. Top Best Transformer (s = Topn) Concrete Semantics  Canonical Abstraction Abstract Semantics ? r Top...... Top r Top Top r Top...... Top s r Top Top s r Top Top s r Top Top s r Top s ’ (v) =  v1: Top(v1)  n(v1,v) 35

36. Then a Miracle Occurs 36

37. Semantic Reduction  Improve the precision of the analysis by recovering properties of the program semantics  A Galois connection (C, , , A)  An operation op:A  A is a semantic reduction when   l  L 2 op(l)  l and   (op(l)) =  (l) l CA   op

38. The Focus Operation  Focus: Formula  (  (3-Struct)   (3-Struct))  Generalizes materialization  For every formula   Focus(  )(X) yields structure in which  evaluates to a definite values in all assignments  Only maximal in terms of embedding  Focus(  ) is a semantic reduction  But Focus(  )(X) may be undefined for some X

39. Top r Top Top r Top Partial Concretization Based on Transformer (s=Topn) Abstract Semantics Partial Concretization Canonical Abstraction Abstract Semantics Top s r Top r Top, r s Top s r Top Top r Top s Top s r Top Top r Top s ’ (v) =  v1: Top(v1)  n(v1,v) 39  u: top(u)  n(u, v) Focus (Top  n)

40. Partial Concretization  Locally refine the abstract domain per statement  Soundness is immediate  Employed in other shape analysis algorithms [Distefano et.al., TACAS’06, Evan et.al., SAS’07, POPL’08]  Emplyed in other analysis algorithms [Typestate verification, ISSTA’06] 40

41. The Coercion Principle  Another Semantic Reduction  Can be applied after Focus or after Update or both  Increase precision by exploiting some structural properties possessed by all stores (Global invariants)  Structural properties captured by constraints  Apply a constraint solver

42. Apply Constraint Solver Top r Top x Top r Top x x rxrx r x, r y n n n n y n Top r Top  x rxrx r x, r y n n y n

43. Sources of Constraints  Properties of the operational semantics  Domain specific knowledge  Instrumentation predicates  User supplied

44. Example Constraints x(v1)  x(v2)  eq(v1, v2) n(v, v1)  n(v,v2)  eq(v1, v2) n(v1, v)  n(v2,v)   eq(v1, v2)  is(v) n*(v3, v4)  t[n](v1, v2)

45. Abstract Transformers: Summary  Kleene evaluation yields sound solution  Focus is a statement-specific partial concretization  Coerce applies global constraints

46. Abstract Semantics 46  { t_embed(coerce(  st(w)  3(focus F(w) (SS[w] ))))  (w,v)  E(G), w  Assignments(G) { } SS [v] = if v = entry  { S | S  SS[w] }  (w,v)  E(G), w  Skip(G)  { t_embed(S) | S  coerce(  st(w)  3(focus F(w) (SS[w] ))) and S  3 cond(w)}  (w,v)  True-Branches(G)  { t_embed(S) | S  coerce(  st(w)  3(focus F(w) (SS[w] ))) and S  3  cond(w)}  (w,v)  False-Branches(G) othrewise

47. Recap  Abstraction  canonical abstraction  recording derived information  Transformers  partial concretization (focus)  constraint solver (coerce)  sound information extraction 47

48. void push (int v) { Node  x = alloc(sizeof(Node)); x  d = v; x  n = Top; Top = x; } Stack Push Top x x x emp xx x Top  v:  c(v)  v: x(v)  v1,v2: n(v1, v2)  Top(v2) … 48

49. Non-blocking Stack [Treiber 1986] [1] void push(Stack *S, data_type v) { [2] Node *x = alloc(sizeof(Node)); [3] x->d = v; [4] do { [5] Node *t = S->Top; [6] x->n = t; [7] } while (!CAS(&S->Top,t,x)); [8] } [9] data_type pop(Stack *S){ [10] do { [11] Node *t = S->Top; [12] if (t == NULL) [13] return EMPTY; [14] Node *s = t->n; [15] data_type r = t->d; [16] } while (!CAS(&S->Top,t,s)); [17] return r; [18] } #define EMPTY -1 typedef int data_type; typedef struct node t { data_type d; struct node t *n } Node; typedef struct stack t { struct node t *Top; } Stack; 49

50. Concurrent Shape Analysis  a thread is represented as a thread object  add predicates to vocabulary  Recipe 1) abstraction: canonical abstraction 2) transformers: interleaving + as before  Bounded threads  Static thread names  Unbounded threads  thread objects abstracted via canonical abstraction 50

51. U = { u1, u2, u3,…,u7 } isThread = { u1,u2 } at[pc=1] = { } … at[pc=6] = { u3 } at[pc=7]= { u1 } Top = { u5 } … x = { (u1,u4), (u2,u3)} t={ (u1,u5),(u2,u6)} n = { (u5,u6), (u6,u7) } t 1= { u1 } t2 = { u2 } Top n x n x t n pc=7 pc=6 t Concrete State u1 u2 u5 u6 u7 u3 u4 thread object with program counter thread-local variable list field list object 51

52. [1] void push(Stack *S, data_type v) { [2] Node *x = alloc(sizeof(Node)); [3] x->d = v; [4] do { [5] Node *t = S->Top; [6] x->n = t; [7] } while (!CAS(&S->Top,t,x)); [8] } Top n n 5 x 5 x n n x x t 6 5 n n x x t 5 6 n n x x t t 6 7 n n n x x t t 6 8 n n n x t 6 1 n n n x t 7 1 n n  v:  c(v) Exploration Top n n x x t t 6 6 … n n x t 5 1 n n 52

53. Top n n pc=5 Representing an Unbounded Number of Threads [1] void push(Stack *S, data_type v) { [2] Node *x = alloc(sizeof(Node)); [3] x->d = v; [4] do { [5] Node *t = S->Top; [6] x->n = t; [7] } while (!CAS(&S->Top,t,x)); [8] } pc=5 x x x 53

54. Top n n pc=5 Representing an Unbounded Number of Threads [1] void push(Stack *S, data_type v) { [2] Node *x = alloc(sizeof(Node)); [3] x->d = v; [4] do { [5] Node *t = S->Top; [6] x->n = t; [7] } while (!CAS(&S->Top,t,x)); [8] } pc=5 x x x Top n n pc=5 x rbyX hasX pc=5 n 54

55. Top n n Abstract Semantics [1] void push(Stack *S, data_type v) { [2] Node *x = alloc(sizeof(Node)); [3] x->d = v; [4] do { [5] Node *t = S->Top; [6] x->n = t; [7] } while (!CAS(&S->Top,t,x)); [8] } pc=5 x rbyX Top n n pc=5 x rbyX pc=6 x rbyX t hasX 55

56. Example - Mutual Exclusion 1 shared Initial configuration [1] while (true) { [2] lock(shared) [C] // critical actions [3] unlock(shared) [4] }  t 1,t 2 : (t 1  t 2 )   (at[pc=c](t 1 )  at[pc=c](t 2 )) 1 shared C held_by A thread enters the critical section 2 shared C held_by blocked Other threads may be blocked or just beginning execution 1 shared 56

57. Recap [1] void push(Stack *S, data_type v) { [2] Node *x = alloc(sizeof(Node)); [3] x->d = v; [4] do { [5] Node *t = S->Top; [6] x->n = t; [7] } while (!CAS(&S->Top,t,x)); [8] } [9] data_type pop(Stack *S){ [10] do { [11] Node *t = S->Top; [12] if (t == NULL) [13] return EMPTY; [14] Node *s = t->n; [15] data_type r = t->d; [16] } while (!CAS(&S->Top,t,s)); [17] return r; [18] } #define EMPTY -1 typedef int data_type; typedef struct node t { data_type d; struct node t *n } Node; typedef struct stack t { struct node t *Top; } Stack; Dynamic Allocation Destructive Updates Concurrency No null dereferences Structural shape invariantsLinearizability 57