SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British.

Slides:

Advertisements

Similar presentations

Systematic Data Selection to Mine Concept Drifting Data Streams Wei Fan IBM T.J.Watson.

Advertisements

Automated Parameter Setting Based on Runtime Prediction: Towards an Instance-Aware Problem Solver Frank Hutter, Univ. of British Columbia, Vancouver, Canada.

Lazy Paired Hyper-Parameter Tuning

SATzilla: Portfolio-based Algorithm Selection for SAT Lin Xu, Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown Department of Computer Science University.

Hydra-MIP: Automated Algorithm Configuration and Selection for Mixed Integer Programming Lin Xu, Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown Department.

10/7/2014 Constrainedness of Search Toby Walsh NICTA and UNSW

Face Alignment by Explicit Shape Regression

1 Local Restarts in SAT Solvers Vadim Ryvchin and Ofer Strichman Technion, Haifa, Israel.

1 Backdoor Sets in SAT Instances Ryan Williams Carnegie Mellon University Joint work in IJCAI03 with: Carla Gomes and Bart Selman Cornell University.

IBM Labs in Haifa © 2005 IBM Corporation Adaptive Application of SAT Solving Techniques Ohad Shacham and Karen Yorav Presented by Sharon Barner.

Dynamic Restarts Optimal Randomized Restart Policies with Observation Henry Kautz, Eric Horvitz, Yongshao Ruan, Carla Gomes and Bart Selman.

Automatic Tuning1/33 Boosting Verification by Automatic Tuning of Decision Procedures Domagoj Babić joint work with Frank Hutter, Holger H. Hoos, Alan.

On the Potential of Automated Algorithm Configuration Frank Hutter, University of British Columbia, Vancouver, Canada. Motivation for automated tuning.

Automatic Algorithm Configuration based on Local Search EARG presentation December 13, 2006 Frank Hutter.

Paper presentation for CSI5388 PENGCHENG XI Mar. 23, 2005

The loss function, the normal equation,

Hierarchical Hardness Models for SAT Lin Xu, Holger H. Hoos and Kevin Leyton-Brown University of British Columbia {xulin730, hoos,

Hrinking hrinking A signment tack tack. Agenda Introduction Algorithm Description Heuristics Experimental Results Conclusions.

CP Formal Models of Heavy-Tailed Behavior in Combinatorial Search Hubie Chen, Carla P. Gomes, and Bart Selman

Pieter Abbeel and Andrew Y. Ng Apprenticeship Learning via Inverse Reinforcement Learning Pieter Abbeel Stanford University [Joint work with Andrew Ng.]

Heavy-Tailed Behavior and Search Algorithms for SAT Tang Yi Based on [1][2][3]

Pieter Abbeel and Andrew Y. Ng Apprenticeship Learning via Inverse Reinforcement Learning Pieter Abbeel Stanford University [Joint work with Andrew Ng.]

Ensemble Learning: An Introduction

Frank Hutter, Holger Hoos, Kevin Leyton-Brown

AAAI00 Austin, Texas Generating Satisfiable Problem Instances Dimitris Achlioptas Microsoft Carla P. Gomes Cornell University Henry Kautz University of.

1 Hybrid Agent-Based Modeling: Architectures,Analyses and Applications (Stage One) Li, Hailin.

Pieter Abbeel and Andrew Y. Ng Reinforcement Learning and Apprenticeship Learning Pieter Abbeel and Andrew Y. Ng Stanford University.

SAT J. Gonzalez. Summary Introduction Algorithms for SAT –Splitting and Resolution –Local Search –Integer Programming Method –Global Optimization SAT.

Lukas Kroc, Ashish Sabharwal, Bart Selman Cornell University, USA SAT 2010 Conference Edinburgh, July 2010 An Empirical Study of Optimal Noise and Runtime.

Hierarchical Hardness Models for SAT Hierarchical Hardness Models for SAT Building the hierarchical hardness models 1.The classifier is trained on a set.

1 Combinatorial Problems in Cooperative Control: Complexity and Scalability Carla Gomes and Bart Selman Cornell University Muri Meeting March 2002.

Hardness-Aware Restart Policies Yongshao Ruan, Eric Horvitz, & Henry Kautz IJCAI 2003 Workshop on Stochastic Search.

Distributions of Randomized Backtrack Search Key Properties: I Erratic behavior of mean II Distributions have “heavy tails”.

Introduction to variable selection I Qi Yu. 2 Problems due to poor variable selection: Input dimension is too large; the curse of dimensionality problem.

CHALLENGING SCHEDULING PROBLEM IN THE FIELD OF SYSTEM DESIGN Alessio Guerri Michele Lombardi * Michela Milano DEIS, University of Bologna.

David Stern Ralf Herbrich Thore Graepel Microsoft Research Cambridge, UK Horst Samulowitz National ICT Australia University of Melbourne Melbourne, Australia.

1 MCMC Style Sampling / Counting for SAT Can we extend SAT/CSP techniques to solve harder counting/sampling problems? Such an extension would lead us to.

ANTs PI Meeting, Nov. 29, 2000W. Zhang, Washington University1 Flexible Methods for Multi-agent distributed resource Allocation by Exploiting Phase Transitions.

Sparse Gaussian Process Classification With Multiple Classes Matthias W. Seeger Michael I. Jordan University of California, Berkeley

Parallel Algorithm Configuration Frank Hutter, Holger Hoos, Kevin Leyton-Brown University of British Columbia, Vancouver, Canada.

Heavy-Tailed Phenomena in Satisfiability and Constraint Satisfaction Problems by Carla P. Gomes, Bart Selman, Nuno Crato and henry Kautz Presented by Yunho.

CLASSIFICATION: Ensemble Methods

Performance Prediction and Automated Tuning of Randomized and Parametric Algorithms Frank Hutter 1, Youssef Hamadi 2, Holger Hoos 1, and Kevin Leyton-Brown.

Tony Jebara, Columbia University Advanced Machine Learning & Perception Instructor: Tony Jebara.

Paired Sampling in Density-Sensitive Active Learning Pinar Donmez joint work with Jaime G. Carbonell Language Technologies Institute School of Computer.

T. Messelis, S. Haspeslagh, P. De Causmaecker B. Bilgin, G. Vanden Berghe.

Performance Prediction and Automated Tuning of Randomized and Parametric Algorithms: An Initial Investigation Frank Hutter 1, Youssef Hamadi 2, Holger.

Boosted Particle Filter: Multitarget Detection and Tracking Fayin Li.

Quality of LP-based Approximations for Highly Combinatorial Problems Lucian Leahu and Carla Gomes Computer Science Department Cornell University.

Patrick De Causmaecker Stefaan Haspeslagh Tommy Messelis.

Balance and Filtering in Structured Satisfiability Problems Henry Kautz University of Washington joint work with Yongshao Ruan (UW), Dimitris Achlioptas.

Transfer Learning in Sequential Decision Problems: A Hierarchical Bayesian Approach Aaron Wilson, Alan Fern, Prasad Tadepalli School of EECS Oregon State.

Competition II: Springleaf Sha Li (Team leader) Xiaoyan Chong, Minglu Ma, Yue Wang CAMCOS Fall 2015 San Jose State University.

Identifying “Best Bet” Web Search Results by Mining Past User Behavior Author: Eugene Agichtein, Zijian Zheng (Microsoft Research) Source: KDD2006 Reporter:

Tommy Messelis * Stefaan Haspeslagh Burak Bilgin Patrick De Causmaecker Greet Vanden Berghe *

Deep Learning and Deep Reinforcement Learning. Topics 1.Deep learning with convolutional neural networks 2.Learning to play Atari video games with Deep.

SUPERVISED AND UNSUPERVISED LEARNING Presentation by Ege Saygıner CENG 784.

Predicting Winning Price In Real Time Bidding With Censored Data Tejaswini Veena Sambamurthy Weicong Chen.

Mustafa Gokce Baydogan, George Runger and Eugene Tuv INFORMS Annual Meeting 2011, Charlotte A Bag-of-Features Framework for Time Series Classification.

Frank DiMaio and Jude Shavlik Computer Sciences Department

Experience Report: System Log Analysis for Anomaly Detection

Applications of Propositional Reasoning Systems

Knot Pipatsrisawat and Adnan Darwiche Computer Science Department UCLA

Sofus A. Macskassy Fetch Technologies

Boosting and Additive Trees (2)

Parallelism in SAT Solvers

Lin Xu, Holger H. Hoos, Kevin Leyton-Brown

Constraint Programming and Backtracking Search Algorithms

Advisor: Dr.vahidipour Zahra salimian Shaghayegh jalali Dec 2017

Presentation transcript:

SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British Columbia {xulin730, hutter, hoos,

2 Outline  Motivation  History of SATzilla and related work  SATzilla methodology  Example Problem  SATzilla for the SAT Competition  Conclusions and ongoing research

Motivation  Lots of high performance solvers, but …. No single SAT solver dominates all others on all types of instances  Question: How to select the best solver for a given SAT instance?

4 Algorithm Selection Problem [Rice, 1976]  Reference: Select solvers based on previous experience or research papers  “Winner-Take-All”: Test solvers on some samples from target distribution; select the solver with best performance.  SATzilla: Automatically based on instance characteristics’

Related work:  Portfolio of stochastic algorithm [Gomes & Selman,1997] Running multiple algorithms at the same time  Reinforcement learning [Lagoudakis & Littman, 2001] Select branching rule at each decision point  Branch & bound algorithm selection [Lobjois & LemaÎter, 1998] Based on an estimation of search tree size

6 History of SATzilla  Old SATzilla [ Nudelman, Devkar, et. al, 2003 ]  2nd Random  2nd Handmade (SAT)  3rd Handmade  SATzilla-07  1 st Handmade  1 st Handmade (UNSAT)  1 st Random  2 nd Handmade (SAT)  3 rd Random (UNSAT)

7 SATzilla-07 Methodology (offline) Target Distribution Solvers Features Final solver selection Build Model Winner- Take-All Collect Data Pre-Solvers

8 SATzilla-07 Methodology (online) Run Pre-solver Compute Features Predict Runtime Run Winner- Take-All Run Best If error If error AND time left; run second best

9 Solvers Used  Eureka [Nadel, Gordon, Palti & Hanna, 2006]  Kcnfs2006 [Dubois & Dequen, 2006]  March_dl2004 [Heule & Maaren, 2006]  Minisat2.0 [Eén & Sörensson, 2006]  OKsolver [Kullmann, 2002]  Rsat [Pipatsrisawat & Darwiche, 2006]  Vallst [Vallstrom, 2005]  Zchaff_Rand [Mahajan, Fu & Malik, 2005]

SATzilla-07 Example Using quasi-group completion Problems (QCP) to validate our general approach

11 SATzilla-07 Example Problem  Problem distribution QCP problems generated near phase transition [Gomes & Selman, 1997]  Solvers Eureka, OKsolver, Zchaff_Rand  Features Same as in previous work [Nudelman, et al. 2004]  Collect Data Compute instances’ features and determine solvers’ runtime  Pre-Solver & “Winner take all”  Build Models  Final solver selection

12 Empirical Hardness Model (EHM)  The Core of SATzilla ---  EHM Accurately predict algorithm’s runtime based on cheaply computable features Linear basis function regression Features () Runtime (y) f w () = w T  …

13 Improve EHM (deal with censoring)  Heavy-tailed behavior and censoring  Three ways for censored data Drop them Keep them as if finished at cutoff Censored sampling  Schmee & Hahn ‘s approach [1979] REPEAT 1. Estimate runtime conditional on EHM and real runtime bigger than cutoff runtime 2. Build new EHM with estimated runtime UNTIL no more changes in EHM

14 How to deal with censored data A: Drop them B: Finished at cutoff C: Censor sampling Censor point

15 How to deal with censored data A: Drop them B: Finished at cutoff C: Censor sampling Censor point

16 How to deal with censored data A: Drop them B: Finished at cutoff C: Censor sampling Censor point

17 Improve EHM (using Hierarchal Hardness Models )  EHM often more accurate, much simpler when trained with SAT/UNSAT samples only [Nudelman, et al. 2004]  Building hierarchal hardness models by approximate model selection Oracle Mixture of experts problem with fixed experts

SATzilla-07 for QCP Average Runtime

SATzilla-07 for QCP Empirical CDF

2007 SAT Competition Three submissions for 2007 SAT Competition BIG_MIX for all three category (demo) RANDOM HANDMADE

21 SATzilla-07 for SAT Competition  Target Distribution Previous SAT competition and SAT Race  Solver (with/without preprocessing, Hyper) Eureka, Kcnfs2006, March_dl2004, Minisat2.0 Vallsat, Rsat, Zchaff_Rand  Features Reduce probing time to 1 second Only cheap features, total about 3 seconds  Pre-Solvers March_dl 5 seconds, SAPS 2 seconds

22 SATzilla-07 for SAT Competition  “Winner take all” solver March_dl2004  Final candidates BIG_MIX Eureka, Kcnfs2006, March_dl2004, Rsat RANDOM March_dl2004, Kcnfs2006, Minisat2.0+ HANDMADE March_dl2004, Vallst, March_dl2004+, Minisat2.0+, Zchaff_Random+

23 SATzilla-07 for BIG_MIX

24 SATzilla-07 for BIG_MIX Pre-solvers Feature time

25 SATzilla-07 for RANDOM

26 SATzilla-07 for RANDOM Pre-solvers Feature time

27 SATzilla-07 for HANDMADE

28 SATzilla-07 for HANDMADE Pre-solvers Feature time

Conclusions

30 Conclusions  Can combine algorithms into portfolios, improving performance and robustness  SATzilla approach has been proven to be successful in real world competition  With more training data and more solvers, SATzilla can be even better

Ongoing research  SATzilla for industrial category Use the same approach, SATzilla is 25% faster and solves 5% more instances  Score function Optimize objective function other than runtime  Local search Improve SATzilla performance by using local search solvers as component

32 Special Thanks  Creators of solvers Alexander Nadel, Moran Gordon, Amit Palti and Ziyad Hanna (Eureka) Marijn Heule, Hans van Maaren (March_dl2004) Niklas Eén, Niklas Sörensson (Minisat2.0) Oliver Kullmann (OKsolver) Knot Pipatsrisawat and Adnan Darwiche (Rsat 1.04) Daniel Vallstrom (Vallst) Yogesh S. Mahajan, Zhaohui Fu and Sharad Malik (Zchaff_Rand)

SATzilla Pick for BIG_MIX

SATzilla Pick for RANDOM

SATzilla Pick for HANDMADE