1. Generating High-Quality Tests for Boolean Circuits by Treating Tests as Proof Encoding Eugene Goldberg, Pete Manolios Northeastern University, USA TAP-2010, July 1-2, Malaga, Spain

2. Summary Introduction Test generation algorithm based on TTPE Experimental results and conclusion

3. Motivation Testing is easy (scalable) but incomplete Formal verification is complete but hard How do we get the best of both worlds (e.g. by generating a small test set that is complete or close to such)? We study testing by the example of combinational circuits We describe circuits by propositional logic

4. Main Idea Q: When does one stop generating tests? A: When the test set encodes a proof Q: Why does testing work at all? A: Short proof  small encoding test set Q: What is the best coverage metric? A: A formal proof is an ideal coverage metric (contains a complete set of corner cases) TTPE: Treating Tests As Proof Encoding

5. Description of the Setup Circuit N  CNF formula F N (X,Y,z), N  0  (F N  z)  0 …. N Y z Point p is (x,y,z) i.e. a complete assignment to X  Y  {z} Test t extracted from p is the assignment to X, i.e. t = x X N is a combinational circuit, It is correct if N  0, It has a bug if N(x) = 1.

6. A Naive Testing Procedure p 1, t 1, N(t 1 )=0, Enc(S 1 ) = no …. N Y z X S i = {p 1,…,p i } p 2, t 2, N(t 2 )=0, Enc(S 2 ) = no …….. p i, t i, N(t i )=0, Enc(S i ) = no p i+1, t i+1, N(t i+1 )=1, Enc(S i+1 ) = no p i+1, t i+1, N(t i+1 )=0, Enc(S i+1 ) = yes

7. Summary Introduction Test generation algorithm based on TTPE Experimental results and conclusion

8. High-Level Description We use the resolution proof system Checking if a set of points encodes a resolution proof is hard Instead, we generate points (called boundary) that encode mandatory fragments of a proof So, instead of trying to encode an entire proof we target essential parts of it We stop when a counterexample is found or a resource is exceeded

9. Encoding a Resolution Proof C = x 1  x 2  ~x 5, C  =~x 1  x 7 p, p  legalize the resolution of C and C  on v iff a) p and p  are different only in v b) C (p ) = 0, C  (p  ) = 0 c) C(p ) = 0, C(p  ) = 0, where C is the resolvent S={p 1,..,p k } encodes a proof if the resolutions legalized by points of S are sufficient to derive an empty clause. Tests T encode a proof if they are extracted from S encoding a proof  C = x 2  ~x 5  x 7

10. Small Complete Test Sets A proof of (F N  z)  0 of k resolutions is encoded by 2  k points (at most) A complete set of  2  k tests for checking N  0. N with 1000 inputs: 10 6 tests instead of 2 1000

11. Boundary Points Given a CNF formula F, an unsatisfying assignment to Vars(F) is a v-boundary point p iff If C  F and C(p)=0, then v  Vars(C) If p is a v-boundary point of F and F is unsatisfiable, any proof has to have a resolution such that it is a resolution on v, the resolvent is falsified by p Adding the resolvent of this resolution to F eliminates p as a v-boundary point. If F is satisfiable, there is a boundary point of F that cannot be eliminated.

12. Extracting Tests from Boundary Points 1.Generate a v-boundary point p i of F (finding p i requires a SAT-check) 2.Extract test t i from p i 3. N(t i )=1? If so, stop. 4.Add to F a resolvent on v (mandated by p i ) 5.Go to step 1 (to generate p i+1 ) Circuit N, CNF formula F=F N  z, Is N(t)=1?

13. Summary Introduction Test generation algorithm based on TTPE Experimental results and conclusion

14. Faults in Arithmetic Components G is a 24-input AND gate

15. Comparing SAT, Random Tests and Tests from Boundary Points Precosat Tests from boundary pnts time (s.) #tests total54, 1152,919562 average3,18317233 median93583 17 faults. Random tests failed (10 6 per fault) Reused tests generated for previous faults Precosat is a winner of SAT-2009 competition Used Precosat for finding boundary points too

16. Some Concluding Remarks TTPE is not a trick. There is a deep relation between proofs and tests Many ways to use TTPE (e.g. encoding mandatory parts of a proof) TTPE for more expressive logics (describing sequential circuits and software)?