On Bridging Simulation and Formal Verification Eugene Goldberg Cadence Research Labs (USA) VMCAI-2008, San Francisco, USA

Summary Introduction –motivation –main idea Stable sets of points (SSPs) and proofs –checking if a set of points contains an SSP –simulation based proof system –extraction of sufficient test sets from proofs Generation of practical tests –test generation for design change/manufacturing faults –theoretical and experimental justification of our approach Conclusion

Motivation Simulation is still the main workhorse of verification. It works surprisingly well taking into account that only negligible part of the search space is sampled!! Given a design, formal verification is to check if property  holds at all points of the search space. Simulation is to “informally” check if property  holds by testing it for a (relatively small) set of points. In this paper, we study this phenomenon by the example of the satisfiability problem. As an application, we consider testing combinational circuits.

Simulation in Terms of SAT Simulation: Given a CNF formula F, find a set of points T s.t. if F(T) = 0, then F is (most likely) unsatisfiable. Satisfiability problem (SAT): Given F, find an assignment p such that F(p)=1 or prove that F(all_space)=0. In other words, find T such that F(T)=0 implies F(all_space) = 0 Let F be a CNF formula (i.e. conjunction of disjunctions also called clauses).

Interpretation of Points as Tests Let CNF formula F N (X,Y,z) specify N. Checking satisfiability of N reduces to checking the satisfiability of F N  z. Let N be a circuit. Let X,Y describe the input and internal variables of N. Let z describe the output variable of N. …. N Y z A point p is a complete assignment to the variables X  Y  {z}. The corresponding test t is the projection of p to X. So p consists of a test t and a computation vector (y,z). Since p falsifies clauses of F N it is computation with a fault. X

Main Idea We formulate simulation as a proof system. In contrast to “regular” simulation, we use points not only for sampling search space but for derivation. In this derivation we employ the machinery of Stable Sets of Points. Instead of proving one “big” theorem (F implies an empty clause), we deduce a sequence of simpler lemmas (F implies a clause L). Each derived clause is added to F.

Summary Introduction –motivation –main idea Stable sets of points (SSPs) and proofs –checking if a set of points contains an SSP –simulation based proof system –extraction of sufficient test sets from proofs Generation of practical tests –test generation for design change/manufacturing faults –theoretical and experimental justification of our approach Conclusion

A Stable Set of Points (SSP) Example of 1-neighborhood of point p w.r.t a clause C (written Nbhd(p,C)) such that C(p)=0. C = x  z, p=(x=0,y=0,z=0). Nbhd(p,C) = {p 1,p 2 }, where p 1 =(x=1,y=0,z=0) and p 2 =(x=0,y=0,z=1). A complete assignment  a point. Let F be a CNF formula and T be a set of points. T is called an SSP if  p  T,  clause C of F such that Nbhd(p,C)  T. A CNF formula F is unsatisfiable iff there is a set of points that is an SSP.

Testing if a Set of Points Contains SSP Let F be a CNF formula and T be a set of points. To check if T is an SSP : For every point p of T, check if there is a clause C of F s.t. Nbhd(p,C)  T. This procedure is linear both in T and F. Even if T is not an SSP, a subset T* of T may be an SSP. To check if T contains an SSP : For every point p of T, check if there is a clause C of F s.t. Nbhd(p,C)  T. If C does not exist, remove p from T and restart this procedure. This procedure is quadratic in T and linear in F.

Simulation as a Proof System We formulate simulation as a proof system with two derivation rules. Rule 1: Generate a point p i and derive the value of F(p i ). Rule 2: Let T={p 1,…,p k } be the current set of points. Derive a clause C such that F implies C and add it to F. Derivation of C is valid only if T contains an SSP of the CNF formula F  ~C (and so F  C holds).

A Sufficient Test Set Given a CNF formula F, a set of points T is called sufficient if there is a set of lemma clauses L 1,…,L k (L k =  ) such that all k derivations F  L 1, F  L 1  L 2, …. F  L 1  …  L k-1  L k are valid (i.e. they can be proved by “simulation” using T) In other words, T is a sufficient test set if the set of valid derivations is sufficient to prove that F is unsatisfiable by applying only the second rule (no new points are needed).

Extracting a Sufficient Test Set from a Resolution Proof Let L={L 1,..,L k } be the set of resolvents of a resolution proof R that F is unsatisfiable. We assume that resolvents L i are numbered in their derivation order (so L k =  ). Then there is always a sufficient test set whose size is  2  k (proving that F  L 1  …  L i-1  L i needs only two points of T). Sufficient test sets can be very small even for very large formulas!!!!

“Proving” Resolvent by Building a Stable Set of Two Points Let C be the resolvent of C and C . One can prove that C  C   C by building an SSP of only two points. Example: Let C = ~x  y  z, C  = x  w. The resolvent C is y  z  w. To prove C  C   C it suffices to show the CNF formula C  C   ~C is unsatisfiable. (Here ~C= ~y  ~z  ~w). After making assignments y=0,z=0,w=0, the formula C  C   ~C turns into ~x  x. Any two points p and p  with y=0,z=0,w=0, that are different only in the value of variable x is an SSP for C  C   ~C.

Summary Introduction –motivation –main idea Stable sets of points (SSPs) and proofs –checking if a set of points contains an SSP –simulation based proof system –extraction of sufficient test sets from proofs Generation of practical tests –test generation for design change/manufacturing faults –theoretical and experimental justification of our approach Conclusion

Applying Our Theory to (design change/manufacturing testing) We cannot apply our theory directly to checking if F is unsatisfiable. (To prove that a set of points is sufficient we need a set of lemma clauses i.e. another proof) However, there are numerous indirect ways to apply our theory. One application is to detect incorrect design changes/ manufacturing faults. Let F be a CNF formula. Let R be a proof that F is unsatisfiable. Find a set of points such that most likely detect satisfiable variations of F. (These variations describe tech. faults/design changes.) Such a test set can be extracted from a sufficient set of points specified by R.

Tight Sufficient Tests Let T be a sufficient test set for F. Informally, T is tight if every point p  T falsifies as few clauses as possible. Given a resolution proof L 1,..,L k, a sufficient test set T is built as T 1  …  T k. Here T i is a two point SSP that proves that C  C   L i where C  C  are the parents of resolvent L i. When building T i one can arbitrarily assign variables that are not in C or C . To build a tight sufficient test, free variables are assigned so as to minimize the number of falsified clauses.

Big Picture Let F be an unsatisfiable formula. To detect satisfiable variations (“faults” in F) we generate a tight sufficient test set T. Such a set can be extracted, for example, from a resolution proof that F is unsatisfiable. The sufficiency of T guarantees its “completeness”. The tightness of T increases the probability of detec- ting small satisfiable variations. Suppose, for example, that a point p of T falsifies only one clause C of F. Then if F* is a variation of F consisting of disappearance of C (along with some other changes) it is likely that F*(p) = 1. Hence p detects that F* is satisfiable.

“Theoretical” Justification of Our Approach Let N 1 and N 2 be identical copies of circuit N. Let R nat be a “natural” resolution proof that their miter is unsatisfiable. Let T={p 1,..,p m } be a tight sufficient test set specified by R nat. We show that the set inp(T)= {inp(p 1 ),…,inp(p m )} detects all (testable) stuck-at faults of N. So, in a sense, our theory “predicts” the high quality of the stuck-at fault model for circuit testing.

Experimental Justification We experimentally compared quality of random test sets and test sets extracted from resolution proofs that two copies of a circuit were identical. Resolution proofs were generated by a state-of-the-art SAT- solver. We used MCNC benchmark circuits. Experiments show that tests extracted from resolution proofs significantly outperform random tests. We applied these two kinds of test sets (of the same size) to detecting literal appearance faults. Such faults are more “subtle” than stuck-at faults and so are harder to detect.

Conclusion We use SAT to show that there is a close relation between high quality test sets and formal proofs. We show how high quality tests can be extracted from proofs of unsatisfiability represented as a sequence of lemma clauses. Our approach answers (at least in principle) two important questions. a)When does one stop simulation? (When the set of generated points “encrypts” a formal proof). b)What is an “ideal” metric in functional verification? (A formal proof.)