Frédéric Saubion LERIA Learning and Intelligent OptimizatioN Conference Autonomous Search

Based on joint works on this topic with : G. Di Tollo A. Fiahlo Y. Hamadi F. Lardeux J. Maturana E. Monfroy M. Schoenauer M. Sebag Learning and Intelligent OptimizatioN Conference

1.Introduction 2.Main Ideas 3.Taxonomy of AS 4.Focus on examples 5.Conclusion and challenges Outline

Introduction Generic modeling tools for engineers (Decision) Variables Domains Constraints Mathematical Model Mathematical Model Solver Solving Constraint Optimization and Satifaction Problems

Introduction Map coloring problem Satifaction Problems

Introduction Map coloring problem Satifaction Problems

Introduction Map coloring problem Satifaction Problems

Introduction Optimization Problems Travelling Salesman Problem : find a round trip across cities with minimal cost

Introduction Optimization Problems Travelling Salesman Problem : find a round trip across cities with minimal cost

Introduction Optimization Problems Travelling Salesman Problem : find a round trip across cities with minimal cost

Introduction Search landscapes are difficult to explore Many variables Complex constraints Problems are more and more complex

Introduction Search landscapes are difficult to explore Exploration vs. Exploitation Balance Problems are more and more complex

Introduction An illustrative example : solving SAT SAT CNF instance Devising more and more complex Solving algorithms Litterals Clauses Assignment (1 0 0)

Introduction Devising more and more complex Solving algorithms How to explore the binary search space (assignments) to find a solution ?

Introduction Devising more and more complex Solving algorithms Use Local Search

Introduction Devising more and more complex Solving algorithms Basic Local Search Choose a random initial assignment

Introduction Devising more and more complex Solving algorithms Basic Local Search Compute the number of true and false clauses

Introduction Devising more and more complex Solving algorithms Basic Local Search Try to improve by changing a value (flip) Move to a neighbor

Introduction Devising more and more complex Solving algorithms Basic Local Search Until finding a solution

Introduction Devising more and more complex Solving algorithms Short overview of the story : a first greedy version GSAT Bart Selman, Hector J. Levesque, David G. Mitchell: A New Method for Solving Hard Satisfiability Problems.AAAI 1992: A first boat for binary seas

Introduction Devising more and more complex Solving algorithms Problem : Many possible moves (many variables)

Introduction Devising more and more complex Solving algorithms Restrict neighborhood Select a false clause C abcdefg

Introduction Devising more and more complex Solving algorithms Get stuck in local optima

Introduction Devising more and more complex Solving algorithms Add pertubations Select a false clause C With a random probability p Perform a random flip for C With (1-p) Select the variable with best IMP Perform best move If solution then stop Else go on Parameter !

Introduction Devising more and more complex Solving algorithms Use restarts False Clauses Iterations Parameter !

Introduction Devising more and more complex Solving algorithms WalkSAT : adding a noise and random restart Henry A. Kautz, Bart Selman: Noise Strategies for Improving Local Search..AAAI 1994

Introduction Devising more and more complex Solving algorithms How to break ties ?

Introduction Devising more and more complex Solving algorithms Add more sophisticated heuristics Compute the age of the variable If the best variable is not the most recent then flip Else With a random probability p Perform a random flip the second best With (1-p) Flip the best Parameter !

Introduction Devising more and more complex Solving algorithms Novelty : using more strategies to perform improvements (age of the variable) D.A. McAllester, B. Selman and H. Kautz. Evidence for invariant in local search.In Proceedings of AAAI-97, AAAI Press 1997, pages

Introduction Devising more and more complex Solving algorithms And improvements go on … Novelty +,Novelty ++, …, TNM, Sattime…

Introduction Devising more and more complex Solving algorithms Captain Jack : many indicators and thus selection strategies Dave A. D. Tompkins, Adrian Balint, Holger H. Hoos: Captain Jack: New Variable Selection Heuristics in Local Search for SAT. SAT 2011:

Introduction Adding more parameters and heuristics Devising more and more complex Solving algorithms More flexible algorithms Fit to different instances Set parameters/heuristics values Understand the behavior

John Rice. The algorithm selection problem. Technical Report CSD-TR 152, Computer science department, Purdue University, The Algorithm Selection Problem Main ideas

John Rice. The algorithm selection problem. Technical Report CSD-TR 152, Computer science department, Purdue University, The Algorithm Selection Problem Main ideas

Related Questions Main ideas Using several algorithms for solving a class of problems

Related Questions Main ideas Adjusting the parameters of one algorithm

Main Objectives Main ideas Need for more autonomous solving tools Increasing number of works in this trend : LION, Special sessions in EA conferences (GECCO,…) …

Ideas for More Autonomous Solvers How to use an algorithm that includes Many parameters Many possible heuristics or components Ideas

Ideas for More Autonomous Solvers How to use an algorithm that include Many parameters Many possible heuristics or components How to automate all these choices ? Ideas

Off-line Automated Tuning Ideas Run your solver on some problems Experiment automatically parameters values

Off-line Automated Tuning Ideas Run your solver on new problems with these parameters values

Off-line Automated Tuning Ideas Question : Generality of the parameters ?

On-line Parameter Control Ideas Try to react during the resolution by changing the parameter

On-line Parameter Control Ideas Example : try to increase some parameter when possible

On-line Parameter Control Ideas Question : How to react efficiently ?

Hyper Heuristics Ideas Combine basic solving heuristics

Hyper Heuristics Ideas Get new solvers

Hyper Heuristics Ideas Question : How to learn the suitable solver ?

Portfolios Based Solvers Ideas Use different types of solvers

Portfolios Based Solvers Ideas Learn how to select the right solver for a given problem

Portfolios Based Solvers Ideas Question : Reliability of the learning process ?

Why introducing the concept of Autonomous Search ? Taxonomy Taxonomies

Classification : Solving Solving Methods Tree-Based Search Metaheuristics SLSEA On-line Off-line Auto Complete/incomplete search, Model representation Other optimization paradigms (e.g., ACO ) Taxonomy

Classification : Parameters Solving Methods Tree-Based Search SLSEA Parameter setting method On-line Off-line Auto Parameter type Structural Behavioral Numerical/discrete values Components of the solver Vs. Configuration of the solver Taxonomy

Classification : Settings Solving Methods Tree-Based Search EA Parameter setting method On-line Off-line Auto Parameter type Experiment-based Feedback Control Measures and learning techniques (reinforcement learning, statistical learning, case- base reasonning…) Taxonomy

Related Approaches Solving Methods Tree-Based Search Metaheuristics SLSEA Parameter setting method On-line Off-line Auto Parameter type Structural Behavioral Taxonomy Parameter Setting in Evolutionary Computation

Related Approaches Solving Methods Tree-Based Search Metaheuristics SLSEA Parameter setting method On-line Off-line Auto Parameter type Structural Behavioral Taxonomy Parameter Setting Parameter Tuning Parameter Control DeterministicAdaptiveSelf-adaptive

Related Approaches Solving Methods Tree-Based Search Metaheuristics SLSEA Parameter setting method On-line Off-line Auto Parameter type Structural Behavioral Taxonomy Optimization of algorithms (automated tuning)

Related Approaches Solving Methods Tree-Based Search Metaheuristics SLSEA Parameter setting method On-line Off-line Auto Parameter type Structural Behavioral Taxonomy Optimization of algorithms (automated tuning) SLS Based (ParamILS) Frank Hutter, Holger H. Hoos, Kevin Leyton-Brown, Thomas Stützle: ParamILS: An Automatic Algorithm Configuration Framework. J. Artif. Intell. Res. (JAIR) 36: (2009) GA Based (Revac) Volker Nannen, A. E. Eiben: Efficient relevance estimation and value calibration of evolutionary algorithm parameters. IEEE Congress on Evolutionary Computation 2007: Racing techniques Mauro Birattari, Thomas Stützle, Luis Paquete, Klaus Varrentrapp: A Racing Algorithm for Configuring Metaheuristics. GECCO 2002: …

Related Approaches Solving Methods Tree-Based Search Metaheuristics SLSEA Parameter setting method On-line Off-line Auto Parameter type Structural Behavioral Taxonomy Reactive Search Learning for SLS

Related Approaches Solving Methods Tree-Based Search Metaheuristics SLSEA Parameter setting method On-line Off-line Auto Parameter type Structural Behavioral Taxonomy Hyper heuristics

Hyper Heuristics Taxonomy Two possible views heuristics to choose heuristics heuristics to generate heuristics Burke EK, Hyde M, Kendall G, Ochoa G, Ozcan E, Woodward J Handbook of Meta-heuristics, A Classification of Hyper-heuristics Approaches

Proposing a general view of AS Solving Methods Tree-Based Search Metaheuristics SLSEA Parameter setting method On-line Off-line Auto Parameter type Structural Behavioral Taxonomy Autonomous Search

Autonomous Search Genesis Taxonomy Gather works from the CSP solving community Workshop Autonomous Search CP 2007, Providence (RI)

Autonomous Search Genesis Taxonomy Identify common concepts, goals and challenges for future works

Requirements for an Autonomous Search System Parameters Taxonomy Modify its internal components Parameters Fine grain heuristics Coarse grain solving techniques Model representation React to external forces and opportunities Search landscape analysis (quality, diversity,...) External knowledge (prediction models, rules,...)

CS Related Areas Taxonomy Solving Techniques Point of View Constraint Programming Operation Research Evolutionary Computation Adjustment Techniques Point of View Reinforcement Learning Statistical Learning Information Theory

CS Related Areas Taxonomy Solving Techniques Point of View Constraint Programming Operation Research Evolutionary Computation Adjustment Techniques Point of View Reinforcement Learning Statistical Learning Information Theory Not limited to…

Examples of works Solving Methods Tree-Based Search Metaheuristics SLSEA Parameter setting method On-line Off-line Auto Parameter type Structural Behavioral Examples

Examples of works Examples Solving Methods Tree-Based Search Metaheuristics SLS EA Parameter setting method On-line Off-line Auto Parameter type Structural Behavioral Manual Empirical Tuning Deciding the Size of a Tabu List Mazure B, Sais L, Gregoire E, Tabu search for sat. In : AAAI/IAAI, pp , 1997

Examples of works Examples Solving Methods Tree-Based Search Metaheuristics SLSEA Parameter setting method On-line Off-line Auto Parameter type Structural Behavioral Automated Parameter Tuning (SLS based) Hutter F, Hoos H, Stutzle T Automatic algorithm configuration based on local search. AAAI 2007

Examples of works Examples Solving Methods Tree-Based Search Metaheuristics SLSEA Parameter setting method On-line Off-line Auto Parameter type Structural Behavioral Portfolios Approaches Features based linear regression and classiers Xu L, Hutter F, Hoos HH, Leyton-Brown K Satzilla : portfolio-based algorithm selection for sat. JAIR 2008

Examples of works Examples Solving Methods Tree-Based Search Tree-Based Search Metaheuristics SLSEA Parameter setting method On-line Off-line Auto Parameter type Structural Behavioral Learning Combinations of Well- known Heuristics Epstein S, Freuder E, Wallace R Learning to support constraint programmers. Comput Intell 2005

Examples of works Examples Solving Methods Tree-Based Search Metaheuristics SLS EA Parameter setting method On-line Off-line Auto Parameter type Structural Behavioral Discovering heuristics (variable selection in SAT SLS) Alex S. Fukunaga : Automated Discovery of Local Search Heuristics for Satisability Testing. Evolutionary Computation 2008

Examples of works Examples Solving Methods Tree-Based Search Metaheuristics SLS EA Parameter setting method On-line Off-line Auto Parameter type Structural Behavioral Automated Tuning Adjusting the size of a Tabu List R. Battiti, G. Tecchiolli : The Reactive Tabu Search. INFORMS Journal on Computing 6(2): (1994)

Examples of works Examples Solving Methods Tree-Based Search Tree-Based Search Metaheuristics SLSEA Parameter setting method On-line Off-line Auto Parameter type Structural Behavioral Controlling Variable Orderings and Values Selection in Search Heuristics Boussemart F, Hemery F, Lecoutre C, Sais L Boosting systematic search by weighting constraints. ECAI

Examples of works Examples Solving Methods Tree-Based Search Metaheuristics SLS EA Parameter setting method On-line Off-line Auto Parameter type Structural Behavioral Adaptive Genetic Algorithms A.E. Eiben, Z. Michalewicz, M. Schoenauer, J. E. Smith Parameter Control in Evolutionary Algorithms. In Parameter Setting in Evolutionary Algorithms 2007

Focus on an example : how to design an AS system ? Generic Evolutionary Algorithm for Constraint Satisfaction Problem Focus Population Apply variation operator New Population Solution ?

Focus on an example Generic Search Algorithm for Constraint Satisfaction Problem Focus Configuration Apply variation operator New configuration Solution ? Question : How to choose the suitable operator at each iteration ?

Focus on an example Idea Associate a probability of application to each operator (parameter) Select an operator according to this probability scheme Focus

Focus on an example Question : how to set the probabilities (parameters of the algorithm) ? Focus

Focus on an example Use a principled approach to tune your parameter Search in the parameters space Assess the performance of each setting through runs of the algorithm on selected Instances ParamILS, REVAC, F-Race … Focus

Focus on an example Second idea: Control the probability during the run Evaluate the performance of each operator after application Adjust the parameters according to the performances Focus

General process for control (Automated Operator Selection) Focus

Focus on an example What are the suitable criteria ? -Quality -Fitness diversity -Genotypic diversity -Time -… Focus

Focus on an example What are the suitable criteria ? -Quality -Fitness diversity -Genotypic diversity -Time -… Focus

Different performnce mesearues Focus Sliding Windows Mean or Max ? How to measure the impact ?

Focus on an example What is the performance of the operators ? Fix a search policy Dynamic policy Values against rank … Focus

Focus on an example What is the performance of the operators ? Fix a search policy Focus

Focus on an example What is the performance of the operators ? No values : Pareto rank of the operators Area under the curve Focus

Estimating efficience of operators How to reward the operators ? Proportionally to their performance Focus

Estimating efficience of operators Using UCB (Upper Confidence Bound) (reinforcement learning technique) Exploration + Exploitation of the operators Choosing the operator having the best UCB Focus

Estimating efficience of operators Warning : UCB converge asymptotically to Gain for the MAB But here we have dynamic changes Use of statistical test to restart learning. Focus

Different selection processes Focus

How to assess the performances of your system ? Focus

Whats next ? Solving Methods Tree-Based Search Metaheuristics SLSEA Parameter setting method On-line Off-line Auto Parameter type Structural Behavioral Focus

Conclusion Many different possible approaches Guidelines for designing new autonomous Solvers Off-line/On-line Behavioural parameters/components Control of the efficient heuristics/discovering new heuristics … Conclusion

Challenges Comparing performances Autonomous vs. ad-hoc Off-line Tuning vs. On-line control Representative benchmarking Conclusion

Challenges Comparing performances Methodologies for comparisons New competitions Chesc ( Cross-domain Heuristic Search Challenge) G. Ochoa and her team Related to No Free-Lunch Theorems More reliable on more problems Conclusion

Challenges Parameters induced by the AS system Abstract parameters should be more easy to control (e.g., EvE balance) New parameters should be less sensitive than original ones Fewer paramaters are easier to adjust Conclusion

Challenges Learning Interactions solving-learning Improving learning off-line Short term (react) vs. long term (prediction) Continuous search (Arbelaez, Hamadi & Sebag) Conclusion

Challenges Distributed and parallel computing Improving algorithms space exploration Sharing information on parameters Sharing information on problems Conclusion

Challenges Towards more generic on-line control tools Identify generic control techniques and measures Control various components type (behavioral parameters, objective functions, heuristics…) Conclusion

Some books to read Conclusion

And ;-) Conclusion

So Sorry for missing references and works Conclusion I will not forget important works and references

So Questions Conclusion I will not forget important works and references