1. Easier and More Informative Vacuity Checks Hana ChocklerandOfer Strichman IBM ResearchTechnion Technion - Israel Institute of Technology not at the same time no free lunch.

2. IBM HRL 2 Preliminaries  The players: s.t. M ²    affects  in M if M 2  [  <- false].  If 9  s.t.  does not affect M, then  is satisfied vacuously in M. [BBER01,KV99]  = G(req -> ack) M M ²  [ack <- false] An LTL formula:  A structure: M A subformula  of  T “satisfies vacuously” = “satisfies from the wrong reasons”

3. IBM HRL 3 Main Results  Cheap vacuity checks:  Redundancy of properties  Vacuity without design  Enhancing vacuity information  Mutual vacuity:  Maximal set of literals that can be replaced with false without falsifying the specification (complexity results)  Iterative algorithm to find this set  Approximation algorithm  Responsibility more expensive, more information

4. IBM HRL 4 Cheap-Checks (1): Redundancy  Redundancy of properties:  is redundant w.r.t a set  of properties if  Each check is easy (there is no system).  The result of a sequential redundancy check is sensitive to the order of checking. Example: either pRq or Gq are redundant, depending on the order.

5. IBM HRL 5 Complexity:  Problem Min-Redundancy: find a minimum subset of  that is equivalent to .  Complexity: -hard  The corresponding decision problem is  2 -hard  Naïve algorithm checks all subsets. Can still be feasible in practice because:   can be small, and  each check is easy. Cheap-Checks (1): Redundancy log n queries to oracle

6. IBM HRL 6 An approximation algorithm (results are sensitive to the order of checks): Remove-redundancy(  ){ For each  2  if then  =  n  } Cheap-Checks (1): Redundancy Some possible improvements: start with checking all sets of the most likely size, start by removing longest formulas, etc.

7. IBM HRL 7  Vacuity without design:  Checking vacuity of a property w.r.t the set of properties:  Assume  ² . Then vacuity without design implies vacuity in the design: Cheap-Checks (2): vacuity without design We save a costly model checking and vacuity check in the design!

8. IBM HRL 8 The algorithm: Vacuity-without-design(  ){ For each property  2  For each subformula  in  if  ²  [  Ã false] then  Ã  [  Ã false] }  Note that the order of vacuity checks does not matter: the substitution does not remove behaviors from . Cheap-Checks (2): vacuity without design Thus, the check is polynomial!

9. IBM HRL 9 Cheap-Checks (3): more info  Suppose that:  ² v   Let  be a subformula of  such that M ²  [  à false]  After “cheap check 2”, we know that  2 v   Let  be the counterexample (an interesting witness).  Perhaps the bug is in the model:  demonstrates an interesting behavior that should be added to .  So… we can suggest  to enhance   Example:  = G(req ->F ack)  = {  :  req }  2  ack à false]  = {} ¢ req ¢ ack ¢ {}  M req ack Constructing a new spec from the new behavior – future work satisfies vacuously

10. IBM HRL 10  The algorithm: Cheap-Checks (3): more info

11. IBM HRL 11 So far...  The vacuity algorithm: check-vacuity( ,M){ Remove-Redundancy(  ); Vacuity-without-design(  ); Informative-Vacuity( ,M); }

12. IBM HRL 12 Mutual Vacuity  Several subformulas can be replaced with false at the same time without falsifying the property.  Vacuity does not check that.  Why is it important ?  Next slide...

13. IBM HRL 13 Mutual Vacuity  Why is it important ?  A result of fixing one vacuity problem, may mask a bigger vacuity problem. That’s because vacuity is not monotonic: there may be ,  s.t.:   ²  [  Ã false],  ²  [  Ã false], but   2  [  Ã false,  Ã false].  Example: AG(a Ç b Ç c)   ² AG(a Ç b Ç false)   ² AG(false Ç false Ç c)   2 AG(false) ababab... cccccc...  ’s single trace

14. IBM HRL 14 Mutual vacuity and Responsibility  We can ask ‘what is the largest set of subformulas of  that can be replaced with false simultaneously without falsifying  ?’  We refer to this problem as MAX-VACUOUS (defined in [CG04]). We will discuss its complexity and how to compute it…  But we can also try to measure the ‘importance’ of each subformula in   This will give us fine-grain vacuity.  The motivation: the added information (beyond what can be given to us by MAX-VACUOUS) can be useful for debugging the model.  To quantify `importance’, we will use the notion of responsibility (defined in [CH04]).

15. IBM HRL 15 Mutual Vacuity  In the discussion that follows we check vacuity and responsibility only for literals (and not for general subformulas). The justification:  [CG04] Theorem: Given a CTL* formula , A subformula  of  can be replaced with false iff all literals of  can be replaced with false.  Example:  let  = GX(a  : b)  8 .  ²  [  à false] $  ²  [a à false, b à true]

16. IBM HRL 16 Mutual Vacuity  (Decision) Problem MAX-VACUOUS: is there a subset of k literals of  that can be replaced with false without falsifying  in M ?  Complexity: NP-complete  By a reduction from CLIQUE  (Computing) Problem MAX-VACUOUS: find a maximum subset of literals of  that can be replaced with false without falsifying  in M.  Complexity: - complete for the size of the maximum subset.  (By a reduction from MAX-CLIQUE-SIZE) the complexity above refers to the size, not to the actual subset; to find the actual subset we need an FNP oracle

17. IBM HRL 17 Mutual Vacuity (2)  Preprocessing for the algorithm:  Construct a Buchi automaton B :  for :   Convert to a conjunctive Buchi automaton CB :   Case-split on disjunctions in the edge labels; add an edge for each case s0s0 s2s2 s2s2 s0s0 s2s2 s2s2 XF((p 1 Æ p 2 ) Ç q) BuchiConjunctive Buchi

18. IBM HRL 18 Example  We know that M ²  thus, no path satisfies CB :   But replacing literals with false (and thus labels on edges with true) creates counterexamples Example:  = p U (q U r); :  = : p R ( : q R : r); s0s0 s2s2 s1s1 s3s3 s4s4 s5s5 s6s6 CB :  

19. IBM HRL 19 Example  Let  Accepting path: s 0 s 1 s 4 s 5 s 6 …  Eliminating set (of occurrences): {p,q} Put back either p or q in order to get rid of this trace s6s6 s0s0 s2s2 s1s1 s3s3 s4s4 s5s5 s0s0 s1s1 s4s4 s5s5 s6s6

20. IBM HRL 20 Example Suppose we choose to return p: s0s0 s2s2 s1s1 s3s3 s4s4 s5s5 s6s6 Example

21. IBM HRL 21 Example - problems  Let  Accepting path: s 0 s 3 s 6 s 6 s 6 …  Eliminating set (of occurrences): {q} Now we have to return q in order to remove the red trace s0s0 s2s2 s1s1 s3s3 s4s4 s5s5 s6s6 s0s0 s3s3 s6s6 Example

22. IBM HRL 22 Example (Conclusions)  Too bad…  We did not find any vacuous satisfaction because we first chose p, rather than q.  Wrong order  monotonic increase in ‘allegedly required’ literals  suboptimal vacuity detection.  What we need:  Given a set of literal sets (e.g. {{p,q},{q}})  Choose the minimum set S of literals that intersects all sets (e.g. {q})   can be replaced with  8 s  S. s à false]  This is a minimum hitting set problem!  The algorithm constructs the minimum set of literals that blocks all “bad” transitions in an iterative way based on counter-example traces The number of iterations is bounded by exp(|L|) Can be approximated using known approximation algorithms for min_hitting_set

23. IBM HRL 23 Algorithm literal-set Max_Vacuous( CB :  ) { literal-set S = ; set-of-literal-sets CE = ; literal-set L = the literals of CB :  while (true) { CB ’ :  = CB :  [ 8 l 2 L n S. l à true ]; if M £ CB ’ :  is empty then return S ; Let ce be the counterexample projected to L; Let elim-set(ce) be the set of literals that can eliminate ce; CE = CE [ elim-set(ce); // minimal set of literals that intersect all sets in CE S = minimum-hitting-set(CE);} Algorithm

24. IBM HRL 24 Responsibility  When M ²  Ã false  we say that  is vacuous in M due to   The check is: is there a counterfactual dependence between  and  in  If no – then  causes vacuity.  We would like to quantify  ’s importance when there is no counterfactual dependence.  Counterfactual dependence (=not vacuous): responsibility = 1.  What happens if there is no counterfactual dependence, but there is some influence?

25. IBM HRL 25 Responsibility  Let  be a formula in positive normal form s.t.  ²   Let l be a literal in   The degree of responsibility of l, dr(l, ,M) of l in the satisfaction of  in M is 1/( k +1) if  k is the smallest number for which there exists a subset S = { l 1 … l  } of  ’s literals that maintains:  l  S   ²  [ l  à false, …, l  à false]   2  [ l  à false, …, l  à false, l à false]  In other words, k is the size of the minimal subset of literals of  that need to be replaced with false in order to make the value of  in M depend on l.

26. IBM HRL 26  Examples:  AG(a Ç b Ç c) responsibility of a, b, c = ½.  AG(a Ç b Ç c);all states in M satisfy a,c, some satisfy b. responsibility of a,c: ½, of b: 0. Responsibility M (has a single trace) a,c a,b,c a, c a,b,c a,c b,c

27. IBM HRL 27 Responsibility: usability  Complexity of computing the responsibility of a literal: FP NPlog(n) – complete, n =  Reduction from CLIQUE-SIZE.  After examining responsibility and debugging the model, we simply would like to get a shorter formula in which all signals have responsibility 1.  Replacing any maximal subset of literals with false without changing the satisfiability of , makes all remaining signals ‘fully responsible’, i.e., for all , responsibility(  ) = 1. that is, it eliminates vacuity

28. IBM HRL 28 Experimental Results  We applied our easy vacuity checks to specifications of a real hardware block (from European project PROSYD).  The hardware block implements a producer, a consumer, and a data receiver.  The set of 17 properties describes its correct behavior.  The results:  9 properties out of 17 are redundant.  After removing the redundant properties, 1 property is vacuous with respect to others.

29. IBM HRL 29 Can this work be used in RuleBase?  The most appropriate place for easy (preliminary) vacuity checks is the “property visualization tool”  Removing redundant properties and eliminating vacuity without design increases performance of model checking  If vacuity without design is not found, the witnesses can be saved for future information to the user in case of vacuous pass in the model  Writing “assumes” iteratively (to eliminate false counterexamples) can lead to redundant assumptions – checkable with redundancy and vacuity without design  Mutual vacuity – is probably too expensive... but the approximate algorithm might be efficient