Presentation is loading. Please wait.

Presentation is loading. Please wait.

On the role of Linear Algebra in the development of Interior Point algorithms and software Due Giorni di Algebra Lineare Numerica Bologna, 6-7 marzo, 2007.

Similar presentations


Presentation on theme: "On the role of Linear Algebra in the development of Interior Point algorithms and software Due Giorni di Algebra Lineare Numerica Bologna, 6-7 marzo, 2007."— Presentation transcript:

1 On the role of Linear Algebra in the development of Interior Point algorithms and software Due Giorni di Algebra Lineare Numerica Bologna, 6-7 marzo, 2007 D. di Serafino, Second University of Naples, Italy joint work with S. Cafieri, École Polytechnique, Paris M. DApuzzo, V. De Simone, Second University of Naples G. Toraldo, University of Naples Federico II

2 D. di SerafinoOn the role of Linear Algebra in the development of IP algorithms and software 2 Outline Motivations KKT system and IP methods Iterative solution of KKT system Constraint Preconditioner Adaptive termination control Numerical experiments

3 D. di SerafinoOn the role of Linear Algebra in the development of IP algorithms and software 3 Motivations ubiquitous in optimization algorithms critical issue in an effective implementation of optimization methods H symmetric, D diagonal semidefinite positive, J full rank symmetric indefinte Focus on IP methods KKT system

4 D. di SerafinoOn the role of Linear Algebra in the development of IP algorithms and software 4 Development of an IP-based software package for solving large-scale (convex) Quadratic Programming problems PRQP - Potential Reduction software for Quadratic Programming problems x, y,, t primal and dual variables, s,, z slack variables Motivations

5 D. di SerafinoOn the role of Linear Algebra in the development of IP algorithms and software 5 KKT system and IP methods type of problem different blocks formulation of the method in the KKT system large-scale problems sparse direct or iterative solvers local convexity of the problem KKT matrix inertia control accuracy requirements inexact solution of the KKT system adaptive stopping criteria increasing ill conditioning as the iterates approach the solution preconditioning techniques OPTIMIZATION LINEAR ALGEBRA

6 D. di SerafinoOn the role of Linear Algebra in the development of IP algorithms and software 6 Infeasible primal-dual PR framework reduce the infeasibility at least at the same rate as the duality gap 0, 0 corresponding to the initial point w 0 barrier functions potential function (Tanabe,1988; Todd & Ye, 1990) duality gap (Kojima, Mizuno & Todd, 1995) primal infeasibility dual infeasibility = 0 feasible version

7 D. di SerafinoOn the role of Linear Algebra in the development of IP algorithms and software 7 PR basic steps 1.Given the current interior iterate w=(x,y,s,z,,v,t), apply a Newton step to the perturbed KKT conditions 2.Update w with suitably chosen

8 D. di SerafinoOn the role of Linear Algebra in the development of IP algorithms and software 8 Newton system reduction: KKT system equality + bound constraints: inequality + bound constraints: ineq. + eq. + bound constraints: E accounts for bound constraints D i pd D pd bound constraints only ( J = I ): diagonal condensed system

9 D. di SerafinoOn the role of Linear Algebra in the development of IP algorithms and software 9 Inherent ill conditioning approaching the solution, some entries of D and E may become very large, producing an increasing ill conditioning in the matrix it is crucial to use preconditioning strategies

10 D. di SerafinoOn the role of Linear Algebra in the development of IP algorithms and software 10 KKT system solvers: direct vs iterative Direct solvers Widely used in well-established IP software (e.g. Mosek, PCx, Loqo, OOQP, KNITRO-Interior/Direct, IPOPT) Ill conditioning not a severe problem (S. Wright, 1997; M. Wright, 1998) Computational cost may become prohibitive for large-scale problems Iterative solvers Increasing attention by IP community in the last 15 years (implemented in KNITRO-Interior/CG, HOPDM, PRQP) Require suitable preconditioning techniques Adaptive termination criteria

11 D. di SerafinoOn the role of Linear Algebra in the development of IP algorithms and software 11 Indefinite preconditioner with the same structure as the KKT matrix M simple approximation to H such that is spd on ker(J e ). Common choice: M diagonal. CP variants investigated by many researchers (Axelsson, 1979; Golub & Wathen, 1998; Luksan & Vlcek, 1998; Keller et al., 2000; Rozloznik & Simoncini, 2002; Durazzi & Ruggiero, 2003; Gondzio et al., 2004; Dollar et al., ; di Serafino et al., 2007; Forsgren et al., 2007; …) Constraint Preconditioner (CP)

12 D. di SerafinoOn the role of Linear Algebra in the development of IP algorithms and software 12 P -1 K has at least m unit eigenvalues If rank(D)=p, then P -1 K has additional m-p unit eigenvalues The remaining eigenvalues are real positive (bounds are available) (Keller at al., 2000; Durazzi & Ruggiero, 2003; Bergamaschi et al., 2004; Dollar, 2007) When the iterate approaches the solution, if q entries of D get close to zero, then additional q eigenvalues tend to be clustered around 1 We expect that the preconditioner increases its effectiveness as the IP method progresses CP: spectral properties

13 D. di SerafinoOn the role of Linear Algebra in the development of IP algorithms and software 13 CP: use of Conjugate Gradient (CG) method (Gould, Hribar, Nocedal, 2001; Rozloznik, Simoncini, 2002; Cafieri, DApuzzo, De Simone, di Serafino, 2007; Dollar, 2007) No breakdown Convergence in at most n-m+p iterations, p=rank(D) Starting guess such that CP+CG applied to KKT system behaves as CG applied to a reduced system with matrix using as preconditioner ( Z basis of ker(J e ) ) CP+CG applied to KKT system behaves as CG applied to a reduced system with matrix using as preconditioner ( Z basis of ker(J e ) ) Preconditioned Projected CG

14 D. di SerafinoOn the role of Linear Algebra in the development of IP algorithms and software 14 Building CP Explicit LBL T factorization of P ( B block diagonal with 1x1 or 2x2 blocks) LBL T factorization of the Schur complement of -M Implicit Schilders factorization of P (Dollar et al., 2006) May still account for a large part of the computational cost! Requiring a factorization of P may still be considered a disadvantage, and methods which avoid this are urgently needed (Gould, Orban, Toint, Acta Numerica, 2005)

15 D. di SerafinoOn the role of Linear Algebra in the development of IP algorithms and software 15 Reducing the cost of building CP Always set the (2,2)-block to 0 ( Dollar, Gould, Schilders, Wathen, 2007) Approximate J by dropping away entries below a prescribed tolerance and outside a fixed band (case D=0 ) ( Bergamaschi, Gondzio, Venturin, Zilli, 2006) Approximate the Schur complement via incomplete Cholesky factorization (case D=0 ) ( Benzi, Simoncini, 2006) Reuse CP for some IP iterations (Cafieri, DApuzzo, De Simone, di Serafino, 2007b)

16 D. di SerafinoOn the role of Linear Algebra in the development of IP algorithms and software 16 Reusing CP The idea is not new (condensed system: Carpenter & Shanno, 1993; Karmakar & Ramakrishnan, 1991) Reusing CP use an approximate CP CG cannot be used, apply SQMR computed at a previous outer step

17 D. di SerafinoOn the role of Linear Algebra in the development of IP algorithms and software 17 Reusing CP: spectral properties (work in progress) has an eigenvalue at 1 with multiplicity at least 2m-p-j at most 2j eigenvalues with nonzero imaginary part The eigenvalues λ satify

18 D. di SerafinoOn the role of Linear Algebra in the development of IP algorithms and software 18 Reusing CP CP is reused until its effectiveness deteriorates the number of inner iterations must not increase too much the number of outer steps for which CP is reused must be bounded by taking into account the increasing ill conditioning and the accuracy requirements k = k+1 factorize P k apply to the KKT system SQMR with P k j = k; l = 0 while (iit(k) α iit(j) and l β l max ) do apply to the KKT system SQMR with P j k = k+1; l = l+1 endwhile l max = l k = k+1 factorize P k apply to the KKT system SQMR with P k j = k; l = 0 while (iit(k) α iit(j) and l β l max ) do apply to the KKT system SQMR with P j k = k+1; l = l+1 endwhile l max = l while (PR stopping criterion not satisfied) do … endwhile iit(k) = # inner iter. at outer step k = 2, = 1 (by numerical experiments)

19 D. di SerafinoOn the role of Linear Algebra in the development of IP algorithms and software 19 Adaptive stopping criteria, according to the quality of the IP iterate (to reduce the number of inner iterations) Basic idea: at early outer iterations low accuracy in solving the KKT system as the iterates approach the optimal solution the accuracy requirement grows up Regard the IP method as an inexact Newton method: residual of the KKT system Termination control perturbation of the KKT cond. of the original pb. KKT cond. of the original pb. condition for convergence

20 D. di SerafinoOn the role of Linear Algebra in the development of IP algorithms and software 20 Termination control polynomial convergence Cafieri, DApuzzo, De Simone, di Serafino, Toraldo, 2007c Inexact PR convergence

21 D. di SerafinoOn the role of Linear Algebra in the development of IP algorithms and software 21 Termination control: theory + comput. study CG/SQMR stopping criterion: Theory Computational study Further reduce the number of inner iterations Avoid slowdown in decreasing the infeasibility (Cafieri, DApuzzo, De Simone, di Serafino, 2007d)

22 D. di SerafinoOn the role of Linear Algebra in the development of IP algorithms and software 22 Software architecture of PRQP linear algebra kernels PRQP core CGSQMR MA27 (LBL T ) BLAS SIF, AMPL interfaces Pre/post-processing utilities CP ICFS

23 D. di SerafinoOn the role of Linear Algebra in the development of IP algorithms and software 23 Testing details PROBLEM nm1m1 m2m2 AUG3DQP * CVXQP CVXQP CVXQP GOULDQP LISWET5 * QPBAND CVXQP1-M CVXQP2-M CVXQP3-M Selected CUTEr problems sparsity of Hessian and constr. matrices 99% * = diagonal Hessian PR stopping criteria: δ = relative duality gap, σ = relative infeasibility # PR iterations 80 Computational environment 2.53 GHz Pentium IV, GB RAM, 128 KB L1 cache Linux Ubuntu 4.1.2, g and gcc compilers

24 D. di SerafinoOn the role of Linear Algebra in the development of IP algorithms and software 24 PRQP: selected results DirectCG + CP PROBLEMPR IT FACT TIME SOLVE TIME TOTAL TIME PR/GC IT CP FACT. TIME SOLVE TIME TOTAL TIME AUG3DQP * e+12.72e-13.11e+119/193.20e+17.28e-13.35e+1 CVXQP e+45.27e+01.16e+420/ e+13.84e+01.63e+1 CVXQP e+32.15e+03.03e+320/ e-11.10e+01.45e+0 CVXQP e+48.01e+02.63e+416/ e+15.76e+07.10e+1 GOULDQP e+08.35e-21.82e+021/905.47e-16.70e-11.79e+0 LISWET5 * e-13.82e-28.77e-125/256.27e-12.49e-11.13e+0 QPBAND141.37e+01.16e-12.32e+014/ e-11.74e+11.91e+1 CVXQP1-M262.79e+41.29e+12.79e+425/ e+14.30e+01.69e+1 CVXQP2-M244.32e+33.19e+04.32e+324/ e-11.19e+01.62e+0 CVXQP3-M304.73e+41.40e+14.72e+430/ e+21.36e+11.20e+2 * = diagonal Hessian

25 D. di SerafinoOn the role of Linear Algebra in the development of IP algorithms and software 25 CG + CPSQMR + reused CP PROBLEM PR/CG IT CP FACT. TIME SOLVE TIME TOTAL TIME PR/QMR IT CP FACT. TIME SOLVE TIME TOTAL TIME AUG3DQP * 19/193.20e+17.28e-13.35e+119/ e+17.02e+02.31e+1 CVXQP120/ e+13.84e+01.63e+120/ e+08.72e+01.26e+1 CVXQP220/ e-11.10e+01.45e+020/ e-22.15e+02.40e+0 CVXQP316/ e+15.76e+07.10e+116/ e+11.53e+13.42e+1 LISWET5 * 25/256.27e-12.49e-11.13e+025/ e-11.28e+01.81e+0 QPBAND14/ e-11.74e+11.91e+114/ e-13.81e+13.92e+1 CVXQP1-M25/ e+14.30e+01.69e+125/ e+06.81e+01.30e+1 CVXQP2-M24/ e-11.19e+01.62e+024/ e-22.20e+02.53e+0 CVXQP3-M30/ e+21.36e+11.20e+230/ e+11.96e+16.50e+1 PRQP: selected results * = diagonal Hessian

26 D. di SerafinoOn the role of Linear Algebra in the development of IP algorithms and software 26 Future work Devising and experimenting new strategies for CP approximation Applying and analysing CPs in the context of IP methods for nonconvex (quadratic) programming Developing and experimenting inertia-controlling iterative solvers Thanks for your attention!


Download ppt "On the role of Linear Algebra in the development of Interior Point algorithms and software Due Giorni di Algebra Lineare Numerica Bologna, 6-7 marzo, 2007."

Similar presentations


Ads by Google