Presentation is loading. Please wait.

Presentation is loading. Please wait.

Engineering Optimization – Concepts and Applications Engineering Optimization Concepts and Applications Fred van Keulen Matthijs Langelaar CLA H21.1

Similar presentations


Presentation on theme: "Engineering Optimization – Concepts and Applications Engineering Optimization Concepts and Applications Fred van Keulen Matthijs Langelaar CLA H21.1"— Presentation transcript:

1 Engineering Optimization – Concepts and Applications Engineering Optimization Concepts and Applications Fred van Keulen Matthijs Langelaar CLA H21.1

2 Engineering Optimization – Concepts and Applications Contents ● Constrained Optimization: Optimality conditions recap ● Constrained Optimization: Algorithms ● Linear programming

3 Engineering Optimization – Concepts and Applications Inequality constrained problems ● Consider problem with only inequality constraints: ● At optimum, only active constraints matter: ● Optimality conditions similar to equality constrained case g2g2 g1g1 x1x1 x2x2 ff g3g3

4 Engineering Optimization – Concepts and Applications Inequality constraints ● First order optimality: ● Consider feasible local variation around optimum: (boundary optimum) (feasible perturbation) Since

5 Engineering Optimization – Concepts and Applications Optimality condition ● Multipliers must be non-negative: g2g2 g1g1 x1x1 x2x2 ff -f-f ● Interpretation: negative gradient (descent direction) lies in cone spanned by positive constraint gradients -f-f

6 Engineering Optimization – Concepts and Applications Optimality condition (2) ● Feasible direction: ● Equivalent interpretation: no descent direction exists within the cone of feasible directions g2g2 g1g1 x1x1 x2x2 ff -f-f Feasible cone ● Descent direction:

7 Engineering Optimization – Concepts and Applications Karush-Kuhn-Tucker conditions ● First order optimality conditions for constrained problem: Lagrangian:

8 Engineering Optimization – Concepts and Applications Sufficiency ● KKT conditions are necessary conditions for local constrained minima ● For sufficiency, consider the sufficiency conditions based on the active constraints: on tangent subspace of h and active g. ● Special case: convex objective & convex feasible region: KKT conditions sufficient for global optimality

9 Engineering Optimization – Concepts and Applications Significance of multipliers ● Consider case where optimization problem depends on parameter a : Lagrangian: KKT: Looking for:

10 Engineering Optimization – Concepts and Applications Significance of multipliers (3) ● Lagrange multipliers describe the sensitivity of the objective to changes in the constraints: ● Similar equations can be derived for multiple constraints and inequalities ● Multipliers give “price of raising the constraint” ● Note, this makes it logical that at an optimum, multipliers of inequality constraints must be positive!

11 Engineering Optimization – Concepts and Applications Contents ● Constrained Optimization: Optimality Criteria ● Constrained Optimization: Algorithms ● Linear Programming

12 Engineering Optimization – Concepts and Applications Constrained optimization methods ● Approaches: – Transformation methods (penalty / barrier functions) plus unconstrained optimization algorithms – Random methods / Simplex-like methods – Feasible direction methods – Reduced gradient methods – Approximation methods (SLP, SQP) ● Note, constrained problems can also have interior optima!

13 Engineering Optimization – Concepts and Applications Augmented Lagrangian method ● Recall penalty method: ● Disadvantages: – High penalty factor needed for accurate results – High penalty factor causes ill-conditioning, slow convergence

14 Engineering Optimization – Concepts and Applications Augmented Lagrangian method ● Basic idea: – Add penalty term to Lagrangian – Use estimates and updates of multipliers ● Also possible for inequality constraints ● Multiplier update rules determine convergence ● Exact convergence for moderate values of p

15 Engineering Optimization – Concepts and Applications Contents ● Constrained Optimization: Optimality Criteria ● Constrained Optimization: Algorithms – Augmented Lagrangian – Feasible directions methods – Reduced gradient methods – Approximation methods – SQP ● Linear Programming

16 Engineering Optimization – Concepts and Applications Feasible direction methods ● Moving along the boundary – Rosen’s gradient projection method – Zoutendijk’s method of feasible directions ● Basic idea: – move along steepest descent direction until constraints are encountered – step direction obtained by projecting steepest descent direction on tangent plane – repeat until KKT point is found

17 Engineering Optimization – Concepts and Applications 1. Gradient projection method ● Iterations follow the constraint boundary: h = 0 x1x1 x2x2 x3x3 ● For nonlinear constraints, mapping back to the constraint surface is needed, in normal space ● For simplicity, consider linear equality constrained problem:

18 Engineering Optimization – Concepts and Applications Gradient projection method (2) ● Recall: – Tangent space: – Normal space: ● Projection: decompose in tangent/normal vector:

19 Engineering Optimization – Concepts and Applications Gradient projection method (3) ● Search direction in tangent space: Projection matrix ● Nonlinear case: ● Correction in normal space:

20 Engineering Optimization – Concepts and Applications Correction to constraint boundary ● Correction in normal subspace, e.g. using Newton iterations: x’ k+1 xkxk x k+1 sksk Iterations: First order Taylor approximation:

21 Engineering Optimization – Concepts and Applications Practical aspects ● How to deal with inequality constraints? ● Use active set strategy: – Keep set of active inequality constraints – Treat these as equality constraints – Update the set regularly (heuristic rules) ● In gradient projection method, if s = 0 : – Check multipliers: could be KKT point – If any  i < 0, this constraint is inactive and can be removed from the active set

22 Engineering Optimization – Concepts and Applications Slack variables ● Alternative way of dealing with inequality constraints: using slack variables: ● Disadvantages: all constraints considered all the time, + increased number of design variables

23 Engineering Optimization – Concepts and Applications 2. Zoutendijk’s feasible directions ● Basic idea: – move along steepest descent direction until constraints are encountered – at constraint surface, solve subproblem to find descending feasible direction – repeat until KKT point is found ● Subproblem: Descending: Feasible:

24 Engineering Optimization – Concepts and Applications Zoutendijk’s method ● Subproblem linear: efficiently solved ● Determine active set before solving subproblem! ● When  = 0 : KKT point found ● Method needs feasible starting point.

25 Engineering Optimization – Concepts and Applications Contents ● Constrained Optimization: Optimality Criteria ● Constrained Optimization: Algorithms – Augmented Lagrangian – Feasible directions methods – Reduced gradient methods – Approximation methods – SQP ● Linear Programming

26 Engineering Optimization – Concepts and Applications Reduced gradient methods ● Basic idea: – Choose set of n - m decision variables d – Use reduced gradient in unconstr. gradient-based method ● State variables s can be determined from: (iteratively for nonlinear constraints) ● Recall: reduced gradient

27 Engineering Optimization – Concepts and Applications Reduced gradient method ● Nonlinear constraints: Newton iterations to return to constraint surface (determine s ): ● Variants using 2 nd order information also exist ● Drawback: selection of decision variables (but some procedures exist) until convergence

28 Engineering Optimization – Concepts and Applications Contents ● Constrained Optimization: Optimality Criteria ● Constrained Optimization: Algorithms – Augmented Lagrangian – Feasible directions methods – Reduced gradient methods – Approximation methods – SQP ● Linear Programming

29 Engineering Optimization – Concepts and Applications Approximation methods ● SLP: Sequential Linear Programming ● Solving series of linear approximate problems ● Efficient methods for linear constrained problems available

30 Engineering Optimization – Concepts and Applications SLP 1-D illustration ● SLP iterations approach convex feasible domain from outside: x = 0.8 x = f g

31 Engineering Optimization – Concepts and Applications SLP points of attention ● Solves LP problem in every cycle: efficient only when analysis cost is relatively high ● Tendency to diverge – Solution: trust region (move limits) x1x1 x2x2

32 Engineering Optimization – Concepts and Applications SLP points of attention (2) ● Infeasible starting point can result in unsolvable LP problem – Solution: relaxing constraints in first cycles k sufficiently large to force solution into feasible region

33 Engineering Optimization – Concepts and Applications f SLP points of attention (3) ● Cycling can occur when optimum lies on curved constraint – Solution: move limit reduction strategy x1x1 x2x2

34 Engineering Optimization – Concepts and Applications Method of Moving Asymptotes ● First order method, by Svanberg (1987) ● Builds convex approximate problem, approximating responses using: ● Approximate problem solved efficiently ● Popular method in topology optimization

35 Engineering Optimization – Concepts and Applications Sequential Approximate Optimization ● Zeroth order method: 1. Determine initial trust region 2. Generate sampling points (design of experiments) 3. Build response surface (e.g. Least Squares, Kriging, …) 4. Optimize approximate problem 5. Check convergence, update trust region, repeate from 2 ● Many variants! ● See also Lecture 4

36 Engineering Optimization – Concepts and Applications Sequential Approximate Optimization Trust region ● Good approach for expensive models ● RS dampens noise ● Versatile Design domain Response surface Sub-optimal point Optimum

37 Engineering Optimization – Concepts and Applications Contents ● Constrained Optimization: Optimality Criteria ● Constrained Optimization: Algorithms – Augmented Lagrangian – Feasible directions methods – Reduced gradient methods – Approximation methods – SQP ● Linear Programming

38 Engineering Optimization – Concepts and Applications SQP ● SQP: Sequential Quadratic Programming ● Newton method to solve the KKT conditions KKT points: Newton:

39 Engineering Optimization – Concepts and Applications SQP (2) ● Newton:

40 Engineering Optimization – Concepts and Applications SQP (3) Note: KKT conditions of: Quadratic subproblem for finding search direction s k

41 Engineering Optimization – Concepts and Applications Quadratic subproblem ● Quadratic subproblem with linear constraints can be solved efficiently: – General case: – KKT condition: – Solution:

42 Engineering Optimization – Concepts and Applications Basic SQP algorithm 1. Choose initial point x 0 and initial multiplier estimates 0 2. Set up matrices for QP subproblem 3. Solve QP subproblem  s k, k+1 4. Set x k+1 = x k + s k 5. Check convergence criteriaFinished

43 Engineering Optimization – Concepts and Applications SQP refinements ● For convergence of Newton method, must be positive definite – Line search along s k improves robustness ● To avoid computation of Hessian information for, quasi-Newton approaches (DFP, BFGS) can be used (also ensure positive definiteness) ● For dealing with inequality constraints, various active set strategies exist

44 Engineering Optimization – Concepts and Applications Comparison MethodAugLagZoutendijkGRGSQP Feasible starting point?NoYesYesNo Nonlinear constraints?Yes Yes Yes Yes Equality constraints?YesHardYesYes SQP generally seen as best general-purpose method for constrained problems Uses active set?YesYesNoYes Iterates feasible?NoYesNoNo Derivatives needed? Yes Yes Yes Yes

45 Engineering Optimization – Concepts and Applications Contents ● Constrained Optimization: Optimality Criteria ● Constrained Optimization: Algorithms ● Linear programming

46 Engineering Optimization – Concepts and Applications Linear programming problem ● Linear objective and constraint functions:

47 Engineering Optimization – Concepts and Applications Feasible domain ● Linear constraints divide design space into two convex half-spaces: x1x1 x2x2 ● Feasible domain = intersection of convex half-spaces: ● Result: X = convex polyhedron

48 Engineering Optimization – Concepts and Applications Global optimality ● Convex objective function on convex feasible domain: KKT point = unique global optimum ● KKT conditions:


Download ppt "Engineering Optimization – Concepts and Applications Engineering Optimization Concepts and Applications Fred van Keulen Matthijs Langelaar CLA H21.1"

Similar presentations


Ads by Google