1. Copyright © ILOG 2008 IBM ILOG CPLEX Advanced Performance Tuning Optimization Technologies Workshop John Gregory CPLEX Product Manager IBM jgregor@us.ibm.com

2. 2 Copyright © ILOG 2008 CPLEX Performance Tuning Overview Speed Continuous Models (LP, QP, QCP/SOCP) MIP Memory Operating System Deployment Application CPLEX Numerical Ill conditioning Numerical Instability Summary

3. 3 Copyright © ILOG 2008 Tuning for Speed: Continuous Models Try each algorithm Primal Simplex Dual Simplex Network Simplex Barrier (only option for QCP/SOCP) Sifting (LP only) Concurrentopt (LP only, parallel processors)

4. 4 Copyright © ILOG 2008 Tuning for Speed: Continuous Models Other tactics Presolve dual on (apply presolve, then solve the dual model) LP only Constraints : 1972170 [Greater: 1972170] Variables : 275036 [Free: 275036] Constraint nonzeros: 9234686 Objective nonzeros: 156817 RHS nonzeros: 1487481 Parallel barrier method Check for numerical difficulties

5. 5 Copyright © ILOG 2008 Tuning for Speed: Continuous Models Iteration logs Degeneracy (simplex methods) Excessive time per iteration

6. 6 Copyright © ILOG 2008 Tuning for Speed : Continuous Models Degeneracy, sample log Iter: 4751 Infeasibility = 8.000000 Iter: 4870 Infeasibility = 8.000000 Iter: 4976 Infeasibility = 6.999999 Iter: 5083 Infeasibility = 6.000000 Iter: 5191 Infeasibility = 6.000000... Iter: 5862 Infeasibility = 6.000000 Iter: 5968 Infeasibility = 6.000000 Perturbation started.

7. 7 Copyright © ILOG 2008 Tuning for Speed: Continuous Models Excessive time per iteration, sample log Elapsed time = 138.23 sec. (37000 iterations) Iter: 37969 Infeasibility = 387849.999786 Iter: 39121 Infeasibility = 379979.999768 Iter: 40295 Infeasibility = 375639.999998 Elapsed time = 150.41 sec. (41000 iterations)... Elapsed time = 24318.58 sec. (138000 iterations) Iter: 138958 Infeasibility = 23.754244 Elapsed time = 24736.98 sec. (139000 iterations) Elapsed time = 25145.43 sec. (140000 iterations) 12 sec. 827 sec.

8. 8 Copyright © ILOG 2008 Tuning for Speed: Continuous Models Degeneracy Try each algorithm Instruct CPLEX to perturb the model Excessive time per iteration Try each algorithm Check for virtual memory usage Run with more memory available Turn on memory emphasis parameter Consider alternate pricing (pgradient/dgradient) Time per iteration depends on pricing scheme

9. 9 Copyright © ILOG 2008 Tuning for Speed: MIP Try the Tuning Tool (CPLEX 11 or later) Primarily for MIPs, but works for all problem types Works with all CPLEX interfaces, including interactive optimizer Can make use of internal algorithm information Let CPLEX and your computer do the work

10. 10 Copyright © ILOG 2008 Tuning for Speed: MIP Input: Model(s) to tune, tuning instructions Problem in memory or set of problems in files Fixed parameter set, parameters which should not be tuned, e.g., tolerances, threads Parameters, metrics to control tuning process (e.g. total tuning time, average or worst case run time) Output: Tuned and fixed parameter settings

11. 11 Copyright © ILOG 2008 Tuning for Speed: MIP Examine node log (if Tuning Tool doesnt help) Excessive iteration counts for node relaxations. Use different start/sub algorithm. Barrier method doesnt support restarts. Set dual gradient to exact steepest edge norms, file parameter to store node file on disk and compress it. More progress needed in best integer or best node value?

12. 12 Copyright © ILOG 2008 Nodes Cuts/ Node Left Objective IInf Best Integer Best Node ItCnt 0 0 -89.0000 6 -89.0000 5278 0 0 -89.0000 6 Fract: 4 12799 0 2 -89.0000 6 -89.0000 12799 1 1 infeasible -89.0000 20767 2 2 -89.0000 5 -89.0000 27275 3 1 infeasible -89.0000 32502 … 8 2 -89.0000 8 -89.0000 65717 9 1 infeasible -89.0000 73714 … Solution time = 177.33 sec. Iterations = 73714 Nodes = 10 (1) Tuning for Speed: MIP Excessive iteration counts for node LPs, sample node log #1 (dual simplex to solve node LPs)

13. 13 Copyright © ILOG 2008 Tuning for Speed: MIP Excessive iteration counts for node LPs, sample node log #2 (primal simplex to solve node LPs) Nodes Cuts/ Node Left Objective IInf Best Integer Best Node ItCnt 0 0 -89.0000 5 -89.0000 6603 0 0 -89.0000 5 Fract: 5 7120 0 2 -89.0000 5 -89.0000 7120 1 1 infeasible -89.0000 9621 2 2 -89.0000 5 -89.0000 10616 3 1 infeasible -89.0000 12963 … 8 2 -89.0000 8 -89.0000 21522 9 1 infeasible -89.0000 23891 … Solution time = 54.37 sec. Iterations = 23891 Nodes = 10 (1)

14. 14 Copyright © ILOG 2008 Tuning for Speed: MIP Progress in best node but not best integer: Nodes Cuts/ Node Left Objective IInf Best Integer Best Node ItCnt Gap... 300 229 22.6667 40 31.0000 22.0000 4433 29.03% 400 309 cutoff 31.0000 22.3333 5196 27.96% 500 387 26.5000 31 31.0000 22.6667 6164 26.88%... 7800 5260 28.5000 23 31.0000 25.6667 55739 17.20% 7900 5324 28.2500 26 31.0000 25.6667 56424 17.20% 8000 5385 27.3750 30 31.0000 25.7778 57267 16.85%

15. 15 Copyright © ILOG 2008 Tuning for Speed: MIP Progress in best integer but not best node: Nodes Cuts/ Node Left Objective IInf Best Integer Best Node ItCnt Gap … 300 296 2018.0000 27 3780.0000 560.0000 3703 85.19% * 300+ 296 0 2626.0000 560.0000 3703 78.67% * 393 368 0 2590.0000 560.0000 4405 78.38% 400 372 560.0000 291 2590.0000 560.0000 4553 78.38% 500 472 810.0000 175 2590.0000 560.0000 5747 78.38%... * 7740+ 5183 0 1710.0000 560.0000 66026 67.25% 7800 5240 1544.0000 110 1710.0000 560.0000 66279 67.25% 7900 5325 944.0000 176 1710.0000 560.0000 66801 67.25% 8000 5424 1468.0000 93 1710.0000 560.0000 67732 67.25%

16. 16 Copyright © ILOG 2008 Tuning for Speed: MIP Tactics for lack of progress in best integer Use especially if customer accepts a good, perhaps sub optimal solution. MIP emphasis Set to 1 (emphasize integer feasibility) Set to 4 (emphasize finding hidden integer solutions) Solution Polishing Requires an integer feasible solution Use to improve sub optimal solutions at end of MIP search Or use as replacement for majority of MIP search when - local search works well - node LPs are time consuming - improving best node appears hopeless

17. 17 Copyright © ILOG 2008 Tuning for Speed: MIP Lack of progress in best integer (continued) RINS Heuristic Local search heuristic - Needs an integer feasible solution MIP Starts and Solution Repair CPLEX 10.0 and later accepts partial MIP starts - Solution repair for (partial) infeasible MIP starts May make RINS, Solution Polishing more effective Feasibility Pump Heuristic (CPLEX 11 and later) Applied first at root node

18. 18 Copyright © ILOG 2008 Tuning for Speed: MIP Lack of progress in best integer (continued) Probing Set to 3 (maximum probing) Cuts Set to 2 (aggressive cut generation) Cuts and Probing dont explicitly help find feasible solutions, but the resulting tighter formulation may help. Can be useful when combined with setting MIP emphasis to 1 (emphasize integer feasibility)

19. 19 Copyright © ILOG 2008 Tuning for Speed: MIP Tactics for lack of progress in best node MIP emphasis Set to 2 or 3 to emphasize proving optimality and moving best node respectively. - May need Partial/Infeasible MIP start, Feasibility Pump or RINS Heuristic to compensate for fewer integer feasible solutions Probing Set to 3 (maximum probing) Cuts Set to 2 (aggressive cut generation) Cuts and Probing often tighten the model formulation

20. 20 Copyright © ILOG 2008 Tuning for Speed: MIP Tactics for lack of progress in best node and best integer Set MIP emphasis to 3 May cause difficulties finding integer feasible solutions Compensate by using (partial or infeasible) MIP starts, Feasibility Pump, RINS Heuristic, Solution Polishing to help find more integer feasible solutions

21. 21 Copyright © ILOG 2008 Tuning for Speed: MIP Tactics for lack of progress in best node and best integer (continued) Provide a starting solution, regardless of objective value Can help CPLEXs local search heuristics Can yield additional tightening of the model Consider external methods (e.g. Constraint Programming, Metaheuristics like Simulated Annealing, model specific heuristics) to find a starting solution Try CPLEXs feasopt feature

22. 22 Copyright © ILOG 2008 Tuning for Speed: MIP Other tactics Specific model knowledge determines parameter settings Try branching up first

23. 23 Copyright © ILOG 2008 Tuning for Speed: MIP Other tactics (continued) Specific model knowledge determines priority order. Time period models - Higher priority to earlier time periods. Dependencies among integer variables

24. 24 Copyright © ILOG 2008 Tuning for Speed: MIP Other tactics (continued) Specific model knowledge determines changes to model Adding model specific cuts - Tighten formulation - Remove symmetry Sherali, Smith Improving Zero-One Model Representations Via Symmetry Considerations Model reformulation http://www.ilog.com/products/cplex/news/newsletters/cplex_newsletter_9410.cfm

25. 25 Copyright © ILOG 2008 Memory Tuning: Operating System Virtual memory slows performance Task Manager (Windows) Top, ps, vmstat under Unix Make sure you can allocate all available memory Java VM limitations Use –Xmx option to increase heap size On a 32-bit machine, the real limit is 2GB Limits on individual accounts Compiler/Linker options 32 bit AIX has default of 256 MB for memory allocations. Write a simple program to test available memory (or ask us for one).

26. 26 Copyright © ILOG 2008 Memory Tuning: Deployment Considerations Compare memory usage Solve the Deployment application model(s) with: Interactive CPLEX Simple example programs Purify, Insure, Bounds Checker, Valgrind (Linux only) can help with memory leaks Make sure you solve the correct model Display problem statistics Data check parameter Diagnostic routines in check.c

27. 27 Copyright © ILOG 2008 Memory Tuning: CPLEX Considerations Sometimes CPLEX really uses all available memory Default settings emphasize speed over memory conservation. Is CPLEX memory usage reasonable? 1 GB per million constraints of an LP - Exceptions: large aspect ratios, very dense constraints 1 GB per million (constraints + quadratic variables) of a QP - Exception: dense Q matrix lacking sparse factorization At least that much for MIP, MIQP

28. 28 Copyright © ILOG 2008 Memory Tuning: CPLEX Considerations Memory conservation techniques General Turn memory emphasis on (slight extra run time) Use different algorithm (LP and QP) As a last resort, turn presolve off (potential big increase in run time) LP Set presolve dual indicator on

29. 29 Copyright © ILOG 2008 Memory Tuning: CPLEX Considerations Memory conservation techniques (continued) MIP MIP search tree often dominates memory usage - Performance tuning to reduce node count. - Setting node file parameter to 3 efficiently writes nodes to disk. - Setting variable select parameter to 3 (strong branching). Fewer nodes required to make the same amount of progress.

30. 30 Copyright © ILOG 2008 Memory Tuning: Other Tactics Buy more memory (but only after accurately estimating requirements) Specialized algorithms Example: Column generation for models with large aspect ratios.

31. 31 Copyright © ILOG 2008 Memory Tuning: Other Tactics Review of model development process Model reformulation Reduce problem dimensions Reduce difficulty while increasing problem size. MIP reformulations for column generation. Increase in problem size in exchange for sparser problem data

32. 32 Copyright © ILOG 2008 Numerical Tuning: Ill conditioning Sensitivity of computed solutions to perturbations in problem data Condition number of a matrix CPLEX solves systems of equations of the form Exact solution is

33. 33 Copyright © ILOG 2008 Numerical Tuning: Ill conditioning Condition number of a matrix (continued) To measure effects of perturbations, consider Exact solution is Combine with exact solution of x:

34. 34 Copyright © ILOG 2008 Numerical Tuning: Ill conditioning Condition number of a matrix (ctd). Using Cauchy - Schwarz inequality This gives max. absolute change in x relative to change in b. Now consider relative change by using Cauchy - Schwarz inequality in original system:

35. 35 Copyright © ILOG 2008 Numerical Tuning: Ill conditioning Condition number of a matrix (continued) Combine and rearrange to get Similar procedure for perturbing B yields

36. 36 Copyright © ILOG 2008 Numerical Tuning: Ill conditioning Ill conditioning can occur under perfect arithmetic But, finite precision can perturb the exact system of equations we wish to solve, resulting in significant changes to the computed solution

37. 37 Copyright © ILOG 2008 Numerical Tuning: Ill conditioning Sources of perturbations due to finite precision Rounding of data Near singular matrices Avoid computing data in single precision Round off error in the simplex method computations, particularly the basis factorization. Round off error due to scaling of large data values Floating point calculations involving values of greatly different orders of magnitude Avoid mixtures of very large and small coefficients

38. 38 Copyright © ILOG 2008 Numerical Tuning: Ill conditioning Rounding of data Maximize x1 + x2 c1: 1/3 x1 + 2/3 x2= 1 c2: x1 + 2 x2 = 3 Solution: x1 = 3.0 Maximize x1 + x2 c1:.333333 x1 +.666667 x2 = 1 c2: x1 + 2 x2 = 3 Solution: x1 = x2 = 1.0, cond(B) = 6.66667e+06 Maximize x1 + x2 c1:.333333333 x1 +.666666667 x2 = 1 c2: x1 + 2 x2 = 3 Solution: x1 = 3.0, cond(B) = 6.333333

39. 39 Copyright © ILOG 2008 Numerical Tuning: Ill conditioning Avoid rounding whenever possible Exact formulation: Maximize x1 + x2 c1: 1/3 x1 + 2/3 x2= 1 c2: x1 + 2 x2 = 3 Imprecisely rounded: Maximize x1 + x2 c1:.333333 x1 +.666667 x2 = 1 c2: x1 + 2 x2 = 3 Scale to integral value whenever possible: Maximize x1 + x2 c1: x1 + 2 x2 = 3 c2: x1 + 2 x2 = 3

40. 40 Copyright © ILOG 2008 Numerical Tuning: Ill conditioning Rounding of data (continued) Changes in right hand side can have similar effect Maximize x1 + x2 c1:.333333 x1 +.666667 x2 = 1 c2: x1 + 2 x2 = 3 Solution: x1 = x2 = 1.0 Maximize x1 + x2 c1:.333333 x1 +.666667 x2 =.999999 c2: x1 + 2 x2 = 3 Solution: x1 = 3.0

41. 41 Copyright © ILOG 2008 Numerical Tuning: Numerical Instability Large values in problem data Bounds can magnify round off error: Large coefficient ratios in matrix rows or columns:

42. 42 Copyright © ILOG 2008 Symptoms of Numerical Instability Sample iteration log Iter: 6389 Objective = 13137.039899 Iter: 6390 Objective = 13137.039899 Iter: 6391 Objective = 13726.011591 Markowitz threshold set to 0.3. Iter: 6456 Scaled infeas = 300615.030682... Iter: 6752 Scaled infeas = 0.000002 Iter: 6754 Objective = -23870.812630

43. 43 Copyright © ILOG 2008 Numerical Tuning Tactics General Turn numerical emphasis on Rounding of data Compute data as accurately as possible Round off error in simplex method Set Markowitz tolerance to.90 (or higher if necessary) Large values in problem data Reduce large coefficients if possible Set scaling parameter to 1 (aggressive scaling)

44. 44 Copyright © ILOG 2008 CPLEX Performance Tuning Summary CPLEX solves most problems with no trouble Organized tactics for the outliers can yield dramatic savings in time. Different categories of performance problems. Use all available information Iteration and node logs for LP and MIP Interactive CPLEX Simple test programs (CPLEX and your own) OS memory management tools Purify/Insure/Bounds Checker/Valgrind ILOG Technical Support and Consulting