1. Hierarchical Hardness Models for SAT Lin Xu, Holger H. Hoos and Kevin Leyton-Brown University of British Columbia {xulin730, hoos, kevinlb}@cs.ubc.ca

2. 22 Hierarchical Hardness Models --- SATzilla-07 One Application of Hierarchical Hardness Models --- SATzilla-07  Old SATzilla [ Nudelman, Devkar, et. al, 2003 ]  2nd Random  2nd Handmade (SAT)  3rd Handmade  SATzilla-07 [Xu, et.al, 2007]  1 st Handmade  1 st Handmade (UNSAT)  1 st Random  2 nd Handmade (SAT)  3 rd Random (UNSAT)

3. 3 Outline  Introduction  Motivation  Predicting the Satisfiability of SAT Instances  Hierarchical Hardness Models  Conclusions and Futures Work

4. Introduction

5. 5  NP-hardness and solving big problems  Using empirical hardness model to predict algorithm’s runtime Identification of factors underlying algorithm’s performance Induce hard problem distribution Automated algorithm tuning [Hutter, et al. 2006] Automatic algorithm selection [Xu, et al. 2007] …

6. 6 Empirical Hardness Model (EHM)  Predicted algorithm’s runtime based on poly-time computable features Features: anything can characterize the problem instance and can be presented by a real number  9 category features [Nudelman, et al, 2004] Prediction: any machine learning technique can return prediction of a continuous value  Linear basis function regression [Leyton-Brown et al, 2002; Nudelman, et al, 2004]

7. 7 Linear Basis Function Regression Features () Runtime (y) f w () = w T  23.34 7.21 …

8. Motivation

9. 9  If we train EHM for SAT/UNSAT separately (M sat, M unsat ), Models are more accurate and much simpler [Nudelman, et al, 2004] M oracular : always use good modelM uncond. : Trained on mixture of instance Solver: satelite; Dataset: Quasi-group completion problem

10. 10 Motivation  If we train EHM for SAT/UNSAT separately (M sat, M unsat ), Models are more accurate and much simpler Using wrong model can cause huge error M unsat : only use UNSAT modelM sat : only use SAT model Solver: satelite; Dataset: Quasi-group completion problem

11. 11 Motivation Question: How to select the right model for a given problem? Attempt to guess Satisfiability of given instance!

12. Predicting Satisfiability of SAT Instances

13. 13 Predict Satisfiability of SAT Instances  NP-complete No poly-time method has 100% accuracy of classification  BUT For many types of SAT instances, classification with poly-time features is possible Classification accuracies are very high

14. 14 Performance of Classification  Classifier: SMLR  Features: same as regression  Datasets rand3sat-var rand3sat-fix QCP SW-GCP DatasetClassification Accuracy sat.unsat.overall rand3sat-var0.9790.9890.984 rand3sat-fix0.8480.8810.865 QCP0.9800.9320.960 SW-QCP0.7520.7110.734 [Krishnapuram, et al. 2005]

15. 15 Performance of Classification (QCP)

16. 16 Performance of Classification For all datasets:  Much higher accuracies than random guessing  Classifier output correlates with classification accuracies  High accuracies with few features

17. Hierarchical Hardness Models

18. 18 Back to Question Question: How to select the right model for a given problem?  Can we just use the output of classifier? NO, need to consider the error distribution  Note: best performance would be achieved by model selection Oracle! Question: How to approximate model selection oracle based on features

19. 19 Hierarchical Hardness Models , s Features & Classifier output s: P(sat) Z Model selection Oracle y Runtime Mixture of experts problem with FIXED experts, use EM to find the parameters for z [Murphy, 2001]

20. 20 B A Importance of Classifier’s Output Two ways: A: Using classifier’s output as a feature B: Using classifier’s output for EM initialization A+B

21. 21 Big Picture of HHM Performance OracularUncond.Hier.OracularUncond.Hier. SolverRMSE (rand3-var)RMSE (rand3-fix) satz0.3290.3580.3440.3430.4200.413 march_dl0.2830.3960.3060.4440.5420.533 kcnfs0.2940.3730.3120.3970.4910.486 Oksolver0.3560.4430.3780.4790.5960.587 SolverRMSE (QCP)RMSE (SW-GCP) Zchaff0.3030.6750.5770.6570.9930.983 Minisat0.3050.5740.5000.6821.0221.024 Satzoo0.2400.3970.3340.3840.581 Satelite0.2470.4260.3720.6180.9700.978 Sato0.3750.7110.6350.7231.3521.345 oksolver0.4270.5480.5060.6011.3371.331

22. 22 Example for rand3-var (satz) Left: unconditional model Right: hierarchical model

23. 23 Example for rand3-fix (satz) Left: unconditional model Right: hierarchical model

24. 24 Example for QCP (satelite) Left: unconditional model Right: hierarchical model

25. 25 Example for SW-GCP (zchaff) Left: unconditional model Right: hierarchical model

26. 26 Correlation Between Prediction Error and Classifier’s Confidence Left: Classifier’s output vs runtime prediction error Right: Classifier’s output vs RMSE Solver: satelite; Dataset: QCP

27. Conclusions and Futures Work

28. 28 Conclusions  Models conditioned on SAT/UNSAT have much better runtime prediction accuracies.  A classifier can be used to distinguish SAT/UNSAT with high accuracy.  Conditional models can be combined into hierarchical model with better performance.  Classifier’s confidence correlates with prediction error.

29. 29 Future Work  Better features for SW-GCP  Test on more real world problems  Extend underlying experts beyond satisfiability