SymDiff: Leveraging Program Verification for Comparing Programs Shuvendu Lahiri Research in Software Engineering (RiSE), Microsoft Research, Redmond Contributors: Chris Hawblitzel (Microsoft Research, Redmond), Ming Kawaguchi (UCSD), Henrique Rebelo (UPFE), Rahul Sharma (Stanford)

Motivation

Ensuring compatibility – Programmers spend a large fraction of their time ensuring (read praying) compatibility after changes Does my bug-fix introduce a regression? Does the refactoring change any observable behavior? How does the feature addition impact existing features?

Compatibility: applications f() { Print(foo); g(); } g() {... Print(foo); } f() { Print(foo); g(); } g() {... Print(foo); } g() {... Print(foo); Print(bar); } g() {... Print(foo); Print(bar); } New features Refactoring Compilers Library API changes Bug fixes Version Control

Compatibility: Microsoft Products – Windows APIs (Win32, ntdll) – Driver development kits –.NET frameworks, Base class library – Compilers (C#, JIT,…) – ….. Windows updates – Security patches – Bug fixes Every developer/tester /auditor

Problem Use static analysis to – Improve the productivity of users trying to ensure compatibility across program changes Potential benefits – Agility: fewer regressions, higher confidence in changes, smarter code review,..

Main challenge Hard to formalize (separate expected changes from unexpected changes) – Refactoring  behaviors intact – Bug fix  non-buggy behaviors intact – Feature add  existing feature behaviors intact – API change  ?? – Data change  ?? – Config changes  ?? – … Equivalence checking is too strong a spec – Most changes modify behavior

Challenge  Opportunity Hard to formalize (separate expected changes from unexpected changes) – Refactoring  behaviors intact – Bug fix  non-buggy behaviors intact – Feature add  existing feature behaviors intact – ……. Highlight “unexpected” changes

Our approach – Provide a tool for performing semantic diff (diff over behaviors) Semantic Diff Does my bug-fix introduce a regression? Does the refactoring change any observable behavior? How does the feature addition impact existing features?

Our approach – Provide a tool for performing semantic diff (diff over behaviors) Semantic Diff Does my bug-fix introduce a regression? Does the refactoring change any observable behavior? How does the feature addition impact existing features?

What is SymDiff? A framework to – Leverage and extend program verification for providing relative correctness

Overview Demo Equivalence checking – Application: Compiler compatibility Differential assertion checking Making SymDiff extensible with contracts – Users can express “expected” changes Ongoing works

Demo 1.Eval (bug1) 2.StringCopy (bug fix) 3.RtlString (regression)

SymDiff tool

SymDiff – Apply and extend program verification techniques towards comparing programs – Current form: Checks input/output partial equivalence Terminating executions from the same input result in the same output [CAV ’12 tool paper]

SymDiff tool: language independent SymDiff (Boogie+ Z3) P1 P2 ≠ P1 = P2 Works at Boogie intermediate language S1 C/.NET/ x86/ARM  Boogie S2 C/.NET/ x86/ARM  Boogie

Simple intermediate verification language – [Barnett et al. FMCO’05] Commands – x := E //assign – havoc x //change x to an arbitrary value – assert E //if E holds, skip; otherwise, go wrong – assume E // if E holds, skip; otherwise, block – S ; T //execute S, then T – goto L1, L2, … Ln //non-deterministic jump to labels – call x := Foo(e1,e2,..) //procedure call

Boogie (contd.) Two types of expressions – Scalars (bool, int, ref,..) – Arrays ([int]int, [ref]ref, …) Array expression sugar for SMT array theory – x[i] := y  x := upd(x, i, y) – y := x[i]  y := sel(x,i) Procedure calls sugar for modular specification call Foo(); assert pre; havoc V; assume post; procedure Foo(); requires pre; ensures post; modifies V;

Basic equivalence checking void swap1(ref int x, ref int y){ int z = x; x = y; y = z; } void swap2(ref int x, ref int y){ x = x + y; y = x - y; x = x - y; } z0 == x0 && x1 == y0 && y1 == z0 && swap1.x == x1 && swap1.y == y1 && x1' == x0 + y0 && y1' == x1' – y0 && x2' == x1' – y1' && swap2.x == x2' && swap2.y == y1' && ~ (swap1.x == swap2.x && swap1.y == swap2.y) Z3 theorem prover UNSAT (Equivalent) SAT (Counterexample)

Handling procedure calls Modular checking – Assume “matched” callees are deterministic and have the same I/O behaviors – Modeled by uninterpreted functions [Necula ‘00, …, Godlin & Strichman ‘08, …..] Addition of postcondition for Foo, Foo’ modifies g; free ensures g == UF_Foo_g(x, old(g)); free ensures ret == UF_Foo_ret(x, old(g)); procedure Foo(x) returns (ret); modifies g; free ensures g == UF_Foo_g(x, old(g)); free ensures ret == UF_Foo_ret(x, old(g)); procedure Foo’(x) returns (ret);

Modeling C/Java/C#/x86  Boogie Separation of concerns – Front end can be developed independently – Quite a few already exists HAVOC/VCC for C, Spec#/BCT for.NET, ?? for Java, … Heap usually modeled by arrays – x.f := y  Heap_f[x] := y Challenges – Deterministic modeling of I/O, malloc, ….. – The entire heap is passed around

Application: Compiler compatibility

Compiler validation X86 ARM ARM+opt Source v1v2v3 Versions X86+opt v4

Compatibility: x86 vs. x86 example G01: mov EAX, EDX G02: and EAX, 255 push EAX mov EDX, 0x call WriteInternalFlag2(int,bool) __epilog: ret G01: mov EAX, EDX G02: and EAX, 255 push EAX mov EDX, 0x call WriteInternalFlag2(int,bool) __epilog: ret G01: push ESI mov ESI, EDX G02: and ESI, 255 push ESI mov EDX, 0x call WriteInternalFlag2(int,bool) G03: pop ESI ret G01: push ESI mov ESI, EDX G02: and ESI, 255 push ESI mov EDX, 0x call WriteInternalFlag2(int,bool) G03: pop ESI ret v2v3 X86+opt 254

Compatibility: x86 vs. x86 example

Large x86 vs. ARM example

Summary of results Compiler tool took heavy dependency on the semantic diff tool for testing the compilers The compiler team found and fixed 12 bugs False alarm varies by configuration – Month to month (x86-x86, ARM-ARM): ~2-3% – Optimized vs. Unoptimized: ~19% – X86 vs. ARM: ~29% Root cause analysis was crucial – Bucketed between 50-96% (depending on configuration) – MAXSAT based rootcause analysis compares poorly Comparable to results of translator validators

Beyond equivalence

Type of changeCheck Refactoring/OptimizationsIn1 = In2  Out1’ = Out2’ Bug fixIn1 = In2  (Fail1’ || Out1’ = Out2’) Feature additionIn1 = In2  (UnImplemented1’ || Out1’ = Out2’) Performance optimizationIn1 = In2  (Measure2’ <= Measure1’) Differential assertion checking (DAC) In1 = In2  (Fail1’ || ~Fail2’)

Differential assertion checking (DAC) – [POPL’12, Interleaved bugs and underspecified harnesses] Given two programs V1 and V2 – Each program has an assertion A Traditionally: Does there exist an “input” n – for which V1(n) is not OK (violates A) DAC: Does there exist an input n – For which V1(n) is OK (for all/some internal choice) – And V2(n) is not OK Highlight warnings more relevant to the changes

Cost-benefit tradeoff in DAC Sound analysis: – Callee can crash – Compute equality information as a prepass Local analysis: Local analysis – Callee cannot crash – Compute equality information as a prepass Bogus analysis: Bogus analysis – Assume callees are equivalent and do not crash

Preliminary evaluation Compared different versions of Windows DDK (Vista vs. Win7) – Instrumented drivers with null pointer asserts Implemented single program local analysis Constructed joint program

Results on DDK

DAC evaluation DAC helps reduce the set of false alarms compared to single version checking Challenges remain to reduce false alarms

Mutual summaries: Making Symdiff extensible

Contracts over two programs Need an extensible contract mechanism for comparing two programs – Generalization of pre/post conditions Why – Allow users to express relative correctness specifications (e.g. conditional equivalence) – Automated methods may not always suffice (even for equivalence checking) Challenge – Should be able to leverage SMT-based program verifiers

Mutual summaries – A extensible framework for interprocedural program comparison Prior work (mostly automated): – Intraprocedural Translation validation [Pnueli et al. ‘98, Necula ‘00, Zuck et al. ’05,…] – Coarse intraprocedural (only track equalities) Regression verification [Strichman et al. ‘08]

Mutual summaries – [MSR-TR ] Mutual summaries (MS) Relative termination (RT) Dealing with loops and unstructured goto

Example: Feature addition int f1(int x1){ a1 = A1[x2]; a2 = A2[x2]; if (Op[x1] == 0) return Val[x1]; else if (Op[x1] == 1) return f1(a1) + f1(a2); else if (Op[x1] == 2) return f1(a1) - f1(a2); else return 0; } int f2(int x2, bool isU){ a1 = A1[x2]; a2 = A2[x2]; if (Op[x2] == 0) return Val[x2]; else if (Op[x2] == 1){ if (isU) return uAdd(f2(a1, T), f2(a2, T)); else return f2(a1, F) + f2(a2, F); } else if (Op[x2] == 2){ if (isU) return uSub(f2(a1, T), f2(a2, T)); else return f2(a1, F) – f2(a2, F); } else return 0; } The programs are equivalent when isU == False

Mutual summaries What is a mutual summary MS(F1, F2)? – An formula over two copies of parameters, globals (g), returns and next state of globals (g’) void F1(int x1){ if(x1 < 100){ g1 := g1 + x1; F1(x1 + 1); } void F2(int x2){ if(x2 < 100){ g2 := g2 + 2*x2; F2(x2 + 1); } MS(F1, F2): (x1 = x2 && g1 = 0) ==> g1’ <= g2’

Mutual summaries What does a mutual summary MS(F1, F2) mean? – For any pre/post state pairs (s1,t1) of F1, and (s2,t2) of F2, (s1,t1,s2,t2) satisfy MS(F1,F2) void F1(int x1){ if(x1 < 100){ g1 := g1 + x1; F1(x1 + 1); } void F2(int x2){ if(x2 < 100){ g2 := g2 + 2*x2; F2(x2 + 1); } MS(F1, F2): (x1 = x2 && g1 = 0) ==> g1’ <= g2’

Example int f1(int x1){ a1 = A1[x2]; a2 = A2[x2]; if (Op[x1] == 0) return Val[x1]; else if (Op[x1] == 1) return f1(a1) + f1(a2); else if (Op[x1] == 2) return f1(a1) - f1(a2); else return 0; } int f2(int x2, bool isU){ a1 = A1[x2]; a2 = A2[x2]; if (Op[x2] == 0) return Val[x2]; else if (Op[x2] == 1){ if (isU) return uAdd(f2(a1, T), f2(a2, T)); else return f2(a1, F) + f2(a2, F); } else if (Op[x2] == 2){ if (isU) return uSub(f2(a1, T), f2(a2, T)); else return f2(a1, F) – f2(a2, F); } else return 0; } MS(f1, f2) = (x1 == x2 && !isU) ==> ret1 == ret2

Checking mutual summaries Given F1, F2, MS(F1, F2), define the following procedure: void CheckMS_F1_F2(int x1, int x2){ inline F1(x1); inline F2(x2); assert MS(F1,F2); }

Modular checking: Instrumentation 1. Add “summary relations” R_F1, and R_F2 void F1(int x1); ensures R_F1(x1, old(g1)/g1, g1/g1’); 2. Use the summary relations to assume mutual summaries at call sites: axiom (forall x1, g1, g1’, x2, g2, g2’:: {R_F1(x1, g1, g1’), R_F2(x2, g2, g2’)} (R_F1(x1, g1, g1’) && R_F2(x2, g2, g2’)) ==> MS_F1_F2(x1, g1, g1’, x2, g2, g2’) );

Leveraging program verifiers Mutual Summary checking – Encode using contracts (postconditions), axioms – Verification condition generation (Boogie) – Checking using SMT solver (Z3) Future work – Inferring the mutual summaries

Relative termination Specification relating the terminating behaviors of P2 wrt P1 Not just for proving termination – Required for composing transformations – MS1(f,f’) && MS2(f’,f’’)  (MS1  MS2) (f,f’’) – E.g. P_Eq(f,f’) && P_Eq(f’,f’’)  P_Eq(f,f’’)

Relative termination condition What is a relative termination condition RT(F1, F2)? – An formula over two copies of parameters, globals (g) void F1(int x1){ if(x1 < 100){ g1 := g1 + x1; F1(x1 + 1); } void F2(int x2){ if(x2 < 100){ g2 := g2 + 2*x2; F2(x2 + 1); } RT(F1, F2): (x1 <= x2)

Relative termination condition What does relative termination condition RT(F1, F2) mean? – For pair of inputs states (s1,s2), if F1 terminates on s1, and (s1,s2) satisfies RT(F1,F2), then F2 terminates on s2 void F1(int x1){ if(x1 < 100){ g1 := g1 + x1; F1(x1 + 1); } void F2(int x2){ if(x2 < 100){ g2 := g2 + 2*x2; F2(x2 + 1); } RT(F1, F2): (x1 <= x2)

What about loops? int Foo2() { i = 0; if (n > 0) { t = g; v = 3; do2: a[i] := v; i := i + 1; v := v + t; //FLABEL While2: //FLABEL if (i < n) goto do2; } return i; } int Foo2() { i = 0; if (n > 0) { t = g; v = 3; do2: a[i] := v; i := i + 1; v := v + t; return While2(i, t, v); } return i; } (int,int) While2(i2, t2, v2) { i2' := i2; v2' := v2; if (i2' < n) { a2[i2'] := v2'; i2' := i2' + 1; v2' := v2' + t2; return While2(i2', t2,v2'); } return (i2‘,v2’); }

Unrolling optimizations void F2(int i2) { if (i2 < n) { a2[i2] = 1; F2(i2+1); return; } return; } void F3(int i3) { if (i3 + 1 < n) { a3[i3] := 1; a3[i3+1] := 1; F3(i3+2); return; } if (i3 < n) a3[i3] := 1; return; } Extra step Inline F2 once inside F2 to “match up” with F3 MS(F2, F3) = (i2 == i3 && a2 == a3) ==> a2’ == a3’

Using mutual summaries Flow 1.Specify the FLABELS to remove loops and gotos into procedures 2.Write mutual summaries for pairs of resulting procedures 3.Specify the inlining limit (if needed)

Express translation validation proofs of many compiler optimizations – Copy propagation – Constant propagation – Common sub-expression elimination – Partial redundancy elimination – Loop invariant code hoisting – Conditional speculation – Speculation – Software pipelining – Loop unswitching – Loop unrolling – Loop peeling – Loop splitting – Loop alignment – Loop interchange – Loop reversal – Loop skewing – Loop fusion – Loop distribution [Kundu, Tatlock, Lerner ‘09] Order of updates differ in two versions

A nice example that uses MS, RT void A(ref x){ if(x != nil){ A(next[x]); D(x); } next: ref  ref; data: ref  int; void B(ref x){ if(x != nil){ D(x); B(next[x]); } void C(ref x){ ref i := x; if(i != nil){ Do: D(i); i := next[i]; if (i != nil) goto Do; } void D(ref x){ data[x] := U(data[x]); } Recursive Tail-recursive Do-while

Overview Demo Equivalence checking Application: Compiler compatibility Differential assertion checking Making SymDiff extensible with contracts Users can express “expected” changes Ongoing works

SMT theory of array isomorphism – Required to deal with out of order mallocs Inferring mutual summaries automatically MAXSAT based root cause analysis

In summary Checking compatibility (statically) is a huge opportunity – Both formalizing the problem – Tools/techniques to solve it Likely to have impact on development cycle – Existing static analysis tools has failed to do so cost- effectively, in spite of all the progress Combining with dynamic analysis – To generate test cases when possible, or aid testing achieve higher differential coverage

Resources SymDiff website Binary release available! – Contains C front end RiSE website for trying tools

Questions

Research challenges Difficult for changes across – Loop boundaries (use loop unrolling) – Procedure boundaries (try use inlining) – Module boundaries (??) Assumptions – procedure names/parameters/globals remain the same – underlying runtime or external APIs are the same and deterministic – object layouts are the same – single threaded executions

Deployment challenges Requires more than just the source code – Need to be able to build Need source depot integration – Currently need both versions to be in two directories Approximation due to modeling of C/C#/x86 – false positives/negatives as the modeling will never be perfect Need some user guidance for bug-fixes/features – Tell the tool what the assertion/feature was

Questions