1. Exact Differentiable Exterior Penalty for Linear Programming Olvi Mangasarian UW Madison & UCSD La Jolla Edward Wild UW Madison December 20, 2015 TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A A A A

2. Preliminaries Exterior penalty functions in linear and nonlinear programming –Exact (penalty parameter remains finite) Nondifferentiable –Asymptotic (penalty parameter approaches infinity) Differentiable Are there exact exterior penalty functions that are differentiable? –Yes for linear programs Which is the topic of this talk

3. Outline Sufficient exactness condition for dual exterior penalty function Exact primal solution computation from inexact dual exterior penalty function Independence of dual penalty function on penalty parameter Generalized Newton algorithm & its convergence DLE: Direct Linear Equation algorithm & its convergence Computational results Conclusion & outlook

4. The Primal & Dual Linear Programs Primal linear program Dual linear program

5. The Dual Exterior Penalty Problem Divide by  2 and let: Penalty problem becomes:

6. Exact Primal Solution Computation Any solution of the dual penalty problem: generates an exact solution y of the primal LP: for sufficiently large but finite  as follows: In addition this solution minimizes: over the solution set of the primal LP. Ref: Journal of Machine Learning Research 2006

7. Optimality Condition for Dual Exterior Penalty Problem & Exact Primal LP Solution A nasc for solving the dual penalty problem: is: where P 2 R m £ m is a diagonal matrix of ones and zeros defined as follows: Solving for u gives: which gives the following exact primal solution,

8. Sufficient Condition for Penalty Parameter  Note that in y = B 0 ( (BB 0 +P)\b ) +  ( B 0 ( (BB 0 + P) \ (Bd) ) - d ) y depends on  only through –The implicit dependence of P on u –The explicit dependence on  above Thus, if  is sufficiently large to ensure y is an exact solution of the linear program, then –P (i.e., the active constraint set) does not change with increasing  –B 0 ( (BB 0 + P) \ (Bd) ) - d = 0 Assumed to hold Ensured computationally

9. Generalized Newton Algorithm Solve the unconstrained problem f(u) = -b 0 u + ½(||B 0 u -  d|| 2 + ||(-u) + || 2 ) using a generalized Newton method Ordinary Newton method requires gradient and Hessian to compute the Newton direction - ( r 2 f(u) ) -1 r f(u), but f is not twice differentiable Instead of ordinary Hessian, we use the generalized Hessian, ∂ 2 f(u) and the generalized Newton direction (∂ 2 f(u)) -1 r f(u) – r f(u) = -b + B(B 0 u -  d) - (-u) + – ∂ 2 f(u) = BB 0 + diag ( sign ( (-u) + ) )

10. Generalized Newton Algorithm (JMLR 2006) minimize f(u) = -b 0 u + ½(||B 0 u -  d|| 2 + ||(-u) + || 2 ) 1)u i + 1 = u i + i t i t i = i (∂ 2 f(u i )) -1 r f(u i ) (generalized Newton direction) i = max {1, ½, ¼, …} s.t. f(u i ) - f(u i + i t i ) ¸ - i ¼ r f(u i ) 0 t i (Armijo stepsize) 2)Stop if || r f(u i )|| · tol & ||B 0 ((BB 0 +P i ) \ (Bd)) - d|| · tol P i = diag(sign((-u i ) + )) 3)If i = imax then  ! 10 , imax ! 2 ¢ imax 4)i ! i + 1 and go to (1)

11. Generalized Newton Algorithm Convergence Assume tol = 0 Assume B 0 ((BB 0 + P) \ (Bd)) - d = 0 implies that  is large enough that an exact solution to the primal is obtained Then either –The Generalized Newton Algorithm terminates at u i such that y = B 0 u i -  d is an exact solution to the primal, or –For any accumulation point ū of the sequence of iterates {u i }, y = B 0 ū -  d is an exact solution to the primal Exactness condition is incorporated as a termination criterion

12. Direct Linear Equation Algorithm f(u) = -b 0 u + ½(||B 0 u -  d|| 2 + ||(-u) + || 2 ) r f(u) = -b + B(B 0 u -  d) - (-u) + = -b + B(B 0 u -  d) + Pu r f(u) = 0, u = (BB 0 + P) -1 (  Bd + b) Successively solve r f(u) = 0 for updated values of the diagonal matrix P = diag ( sign ( (-u) + ) )

13. Direct Linear Equation Algorithm minimize f(u) = -b 0 u + ½(||B 0 u -  d|| 2 + ||(-u) + || 2 ) 1)P i = diag ( sign ( (-u i ) + ) ) 2)u i+1 = (BB 0 + P i ) \ (b +  Bd) 3)u i+1 ! u i + i (u i+1 - u i ) i is the Armijo stepsize 4)Stop if ||u i+1 - u i || · tol & ||B 0 ( (BB 0 +P i ) \ (Bd) ) - d|| · tol 5)If i = imax then  ! 10 , imax ! 2 ¢ imax 6)i ! i + 1 and go to (1)

14. Direct Linear Equation Algorithm Convergence Assume tol = 0 Assume B 0 ( (BB 0 + P) \ (Bd) ) - d = 0 implies that  is large enough that an exact solution to the primal is obtained, and that each matrix in the sequence {BB 0 + P i } is nonsingular Then either –The Direct Linear Equation Algorithm terminates at u i such that y = B 0 u i -  d is an exact solution to the primal, or –For any accumulation point ū of the sequence of iterates {u i }, y = B 0 ū -  d is an exact solution to the primal Exactness condition is incorporated as a termination criterion

15. Solving Primal LPs with More Constraints than Variables Difficulty: factoring BB 0 Solution: get exact solution to the dual which requires factoring a smaller matrix the size of B 0 B Given an exact solution of the dual, find the exact solution of the primal by solving where B 1 and B 2 correspond to u 1 > 0 and u 2 = 0 –Requires factoring matrices only of size B 0 B

16. Primal exterior penalty problem The Primal & Dual Linear Programs Dual linear program Primal linear program For sufficiently large , u = (-By +  b) + is an exact solution of the dual linear program Furthermore, this solution minimizes ||u|| 2 over the solution set of the dual linear program

17. Optimality Condition for the Primal Exterior Penalty Problem & Exact Dual LP Solution A nasc for solving the primal penalty problem: is: where Q 2 R ` £ ` is a diagonal matrix of ones and zeros defined as follows: Solving for y gives: y = (B 0 QB) \ (  B 0 Qb - d) which gives the following exact dual solution, u = (-By +  b) +, u = ( B ( (B 0 QB) \ d ) -  ( B ( (B 0 QB) \ (B 0 Qb) ) - b ) ) +

18. Sufficient Condition for Penalty Parameter  Note that in u = ( B ( (B 0 QB) \ d ) -  ( B ( (B 0 QB) \ (B 0 Qb) ) - b ) ) + u depends on  only through –Q, which depends on  through y and  –The explicit dependence on  above Thus,  is sufficiently large to ensure u is an exact solution of the linear program if –Q does not change with increasing  –diag ( sign(u) )( B ( (B 0 QB) \ (B 0 Qb) ) - b ) = 0 The subgradient with respect to 

19. Computational Details Cholesky factorization used for both methods –Ensure factorizability by adding a small multiple of the identity matrix –For example, BB 0 + P +  I for some small  –Other approaches left to future work Start with  = 100 for both methods –Newton method: occasionally increased to 1000 –Direct method:  not increased in our examples

20. Computational Results When B 0 ( (BB 0 +P) \ (Bd) ) - d = 0, optimal solution obtained –Tested on randomly generated linear programs –We know the optimal objective values –This condition is used as a stopping criterion –Relative difference from the true objective value and maximum constraint violation less then 1e-3, and often smaller than 1e-6 B 0 ( (BB 0 + P) \ (Bd) ) - d = 0 satisfied efficiently –Our algorithms are compared against the commercial LP package CPLEX 9.0 (simplex and barrier methods) –Our algorithms are implemented using MATLAB 7.3

21. Running Time Versus Linear Program Size Problems with the Same Number of Variables and Constraints Number of variables (= number of constraints) Average seconds to solution

22. Average Seconds to Solve 10 Random Linear Programs with 100 Variables and Increasing Numbers of Constraints ConstraintsCPLEXNewton LPDLE 1,0000.1 10,0000.31.00.5 100,0003.013.05.5 1,000,00044.8173.570.9

23. VariablesCPLEXNewton LPDLE 1,0000.020.040.03 10,0000.050.200.90 100,0000.843.220.91 1,000,00017.929.19.3 Average Seconds to Solve 10 Random Linear Programs with 100 Constraints and Increasing Numbers of Variables

24. Conclusion Presented sufficient conditions for obtaining an exact solution to a primal linear program from a classical dual exterior penalty function Precise termination condition given for –Newton algorithm for linear programming (JMLR 2006) –Direct method based on solving the optimality condition of the convex penalty function Algorithms efficiently obtain optimal solutions using the precise termination condition

25. Future Work Deal with larger linear programs Application to real-world linear programs Direct methods for other optimization problems, e.g. linear complementarity problems Further improvements to performance and robustness

26. Links to Talk & Papers http://www.cs.wisc.edu/~olvi http://www.cs.wisc.edu/~wildt