Experimental Algorithmics Reading Group, UBC, CS Presented paper: Fine-tuning of Algorithms Using Fractional Experimental Designs and Local Search by Belarmino Adenso-Díaz (Barcelona) and Manuel Laguna (Colorado) OR Journal 2006 Presenter: Frank Hutter, 23 Aug 2006

23 Aug, 2006Fine-tuning of algorithms2 Motivation Anecdotal evidence that of the total time for designing and testing a new (meta) heuristic  10% is spent on development  90% is spent on fine-tuning parameter  (In my opinion, 90% is maybe a little bit too high)  If you see any real stats about this sometime, please let me know !

23 Aug, 2006Fine-tuning of algorithms3 Motivation (2) Barr et al (1995) (we read this Nov 2004) “The selection of parameter values that drive heuristics is itself a scientific endeavor and deserves more attention than it has received in the operations research literature. This is an area where the scientific method and statistical analysis could and should be employed.”

23 Aug, 2006Fine-tuning of algorithms4 Motivation (3) Parameter tuning in the OR literature  1) “Parameter values have been established experimentally” (without stating the procedure)  2) Just give parameters without explanation, often different for problem classes or even for each instance  3) Use parameter values that were previously determined to be effective (simulated annealing, Guided Local Search for MPE)  4) Sometimes employed experimental design is stated

23 Aug, 2006Fine-tuning of algorithms5 Objective function to be minimized Runtime for solving a training set of decision problems  They only do one run per instance and (wrongly?) refer to runs on different instances as replication Average deviation from optimal solution in optimization algorithms In general: some combination of speed and accuracy

23 Aug, 2006Fine-tuning of algorithms6 Design of experiments Includes  1) The set of treatments included in the study  2) The set of experimental units included in the study  3) The rules and procedures by which treatments are assigned to experimental units  4) Analysis (measurements that are made on the experimental units after the treatments have been applied)

23 Aug, 2006Fine-tuning of algorithms7 Different designs Full factorial experimental design  2 k factorial – 2 levels (critical values) per variable  3 k factorial Fractional factorial experiment  Orthogonal array with n=8 runs, k=5 factors, s=2 levels and strength t=2  n x k array with entries 0 to s-1 and property that in any t columns the s t possible combinations appear equally often (the projections to lower dimensions are balanced)

23 Aug, 2006Fine-tuning of algorithms8 Aside: Taguchi design of experiments Genichi Taguchi  Robust parameter design Set controllable parameters to achieve maximum output with low variance

23 Aug, 2006Fine-tuning of algorithms9 Taguchi design applied here L9(3 4 ) is a design with nine runs, 4 variables with 3 values each and strength 2 (for each combination of variables, each of the 9 value combinations occurs exactly once) Based on this, you can estimate the “optimal condition”, even if it’s not one of the 9 runs performed (how ?  separate topic)

23 Aug, 2006Fine-tuning of algorithms10 The CALIBRA software Limited to 5 parameters Starts with full factorial bi-level design (32 runs)  25% and 75% “quantiles” of each parameter as levels  Fix parameter with least significant main effect to its best value From then on, do “local search”  Choose 3 levels around last best setting  Do L9(3 4 ) Taguchi design  Narrow down the levels around a the best predicted solution When local optimum is reached  Build new starting point for local search by combining previous local optima/previous worst solutions  This is meant to trade off exploration and exploitation but seem fairly ad-hoc

23 Aug, 2006Fine-tuning of algorithms11 The CALIBRA software (2) Only in Windows Requires the algorithm to be tuned as a.exe file  Just write a.exe wrapper

23 Aug, 2006Fine-tuning of algorithms12 The CALIBRA software (3) Objective function can be based on multiple instances Deal with that inside the algorithm  Well, in the wrapper

23 Aug, 2006Fine-tuning of algorithms13 The CALIBRA software - Live demo Let’s hope it works … They do some caching that’s not mentioned in the paper

23 Aug, 2006Fine-tuning of algorithms14 Backup in case it doesn’t work

23 Aug, 2006Fine-tuning of algorithms15 Experimental analysis Pretty straight-forward MAXEX is a parameter of major importance !! They do a little bit better than the manually found parameter settings (or those found with Taguchi designs) For these domains, not too much promise in per-instance tuning (Table 5 compared to Table 2) Figure 9 vs 10 probably only shows that their performance metric means different things for different domains

23 Aug, 2006Fine-tuning of algorithms16 Points for improvements Objective function evaluation requires solving many instances (possibly many times)  Takes lots of time even if results are abysmal  Can stop evaluation if it’s (statistically) clear that the result won’t be better than the best one we already have “CALIBRA should be more effectice in situations when the interactions among parameters are negligible”  But then you really don’t need anything like this !  Related work (DACE) builds a model of the whole surface - I expect that to work better