(iii) Lagrange Multipliers and Kuhn-tucker Conditions D Nagesh Kumar, IISc Introduction to Optimization Water Resources Systems Planning and Management: M2L3

Objectives D Nagesh Kumar, IISc 2  To study the optimization with multiple decision variables and equality constraint : Lagrange Multipliers.  To study the optimization with multiple decision variables and inequality constraint : Kuhn-Tucker (KT) conditions Water Resources Systems Planning and Management: M2L3

Constrained optimization with equality constraints D Nagesh Kumar, IISc 3 A function of multiple variables, f(x), is to be optimized subject to one or more equality constraints of many variables. These equality constraints, g j (x), may or may not be linear. The problem statement is as follows: Maximize (or minimize) f(X), subject to g j (X) = 0, j = 1, 2, …, m where (1) Water Resources Systems Planning and Management: M2L3

Constrained optimization… D Nagesh Kumar, IISc 4  With the condition that ; or else if m > n then the problem becomes an over defined one and there will be no solution. Of the many available methods, the method of constrained variation and the method of using Lagrange multipliers are discussed. Water Resources Systems Planning and Management: M2L3

Solution by method of Lagrange multipliers D Nagesh Kumar, IISc 5 Continuing with the same specific case of the optimization problem with n = 2 and m = 1 we define a quantity λ, called the Lagrange multiplier as (2) Using this in the constrained variation of f [ given in the previous lecture] And (2) written as (3) (4) Water Resources Systems Planning and Management: M2L3

Solution by method of Lagrange multipliers… D Nagesh Kumar, IISc 6 Also, the constraint equation has to be satisfied at the extreme point (5) Hence equations (2) to (5) represent the necessary conditions for the point [x 1 *, x 2 *] to be an extreme point. λ could be expressed in terms of as well has to be non- zero. These necessary conditions require that at least one of the partial derivatives of g(x 1, x 2 ) be non-zero at an extreme point. Water Resources Systems Planning and Management: M2L3

D Nagesh Kumar, IISc 7 The conditions given by equations (2) to (5) can also be generated by constructing a functions L, known as the Lagrangian function, as (6) Alternatively, treating L as a function of x 1,x 2 and, the necessary conditions for its extremum are given by (7) Solution by method of Lagrange multipliers… Water Resources Systems Planning and Management: M2L3

Necessary conditions for a general problem D Nagesh Kumar, IISc 8 For a general problem with n variables and m equality constraints the problem is defined as shown earlier Maximize (or minimize) f(X), subject to g j (X) = 0, j = 1, 2, …, m where In this case the Lagrange function, L, will have one Lagrange multiplier j for each constraint as (8) Water Resources Systems Planning and Management: M2L3

D Nagesh Kumar, IISc 9 L is now a function of n + m unknowns,, and the necessary conditions for the problem defined above are given by (9) which represent n + m equations in terms of the n + m unknowns, x i and j. The solution to this set of equations gives us and (10) Necessary conditions for a general problem… Water Resources Systems Planning and Management: M2L3

Sufficient conditions for a general problem D Nagesh Kumar, IISc 10 A sufficient condition for f(X) to have a relative minimum at X * is that each root of the polynomial in Є, defined by the following determinant equation be positive. (11) Water Resources Systems Planning and Management: M2L3

D Nagesh Kumar, IISc 11 where (12) Similarly, a sufficient condition for f(X) to have a relative maximum at X * is that each root of the polynomial in Є, defined by equation (11) be negative. If equation (11), on solving yields roots, some of which are positive and others negative, then the point X * is neither a maximum nor a minimum. Sufficient conditions for a general problem… Water Resources Systems Planning and Management: M2L3

Example D Nagesh Kumar, IISc 12 Minimize, Subject to Solution with n = 2 and m = 1 L = or Water Resources Systems Planning and Management: M2L3

Example… D Nagesh Kumar, IISc 13 Hence, Water Resources Systems Planning and Management: M2L3

Example… D Nagesh Kumar, IISc 14 The determinant becomes or Since is negative, X*, correspond to a maximum Water Resources Systems Planning and Management: M2L3

Kuhn – Tucker Conditions D Nagesh Kumar, IISc 15  KT condition: Both necessary and sufficient if the objective function is concave and each constraint is linear or each constraint function is concave, i.e., the problems belongs to a class called the convex programming problems. Water Resources Systems Planning and Management: M2L3

Kuhn-Tucker Conditions: Optimization Model Consider the following optimization problem Minimize f(X) subject to g j (X) ≤ 0 for j=1,2,…,p where the decision variable vector X=[x 1,x 2,…,x n ] D Nagesh Kumar, IISc Water Resources Systems Planning and Management: M2L3 16

Kuhn-Tucker Conditions Kuhn-Tucker conditions for X * = [x 1 *, x 2 *,... x n * ] to be a local minimum are D Nagesh Kumar, IISc Water Resources Systems Planning and Management: M2L3 17

Kuhn Tucker Conditions …  In case of minimization problems, if the constraints are of the form g j (X) ≥ 0, then λ j have to be non-positive  If the problem is one of maximization with the constraints in the form g j (X) ≥ 0, then λ j have to be nonnegative. D Nagesh Kumar, IISc Water Resources Systems Planning and Management: M2L3 18

Example 1 D Nagesh Kumar, IISc 19 Minimize subject to Water Resources Systems Planning and Management: M2L3

D Nagesh Kumar, IISc 20 Kuhn – Tucker Conditions Example 1… Water Resources Systems Planning and Management: M2L3

From (17) either = 0 or, Case 1: = 0  From (14), (15) and (16) we have x 1 = x 2 = and x 3 =  Using these in (18) we get  From (22),, therefore, =0,  Therefore, X * = [ 0, 0, 0 ] This solution set satisfies all of (18) to (21) Example 1… D Nagesh Kumar, IISc 21 Water Resources Systems Planning and Management: M2L3

Case 2:  Using (14), (15) and (16), we have or  But conditions (21) and (22) give us and simultaneously, which cannot be possible with Hence the solution set for this optimization problem is X * = [ ] Example 1… D Nagesh Kumar, IISc 22 Water Resources Systems Planning and Management: M2L3

Minimize subject to Example 2 D Nagesh Kumar, IISc 23 Water Resources Systems Planning and Management: M2L3

Kuhn – Tucker Conditions Example 2… D Nagesh Kumar, IISc 24 Water Resources Systems Planning and Management: M2L3

From (25) either = 0 or, Case 1  From (23) and (24) we have and  Using these in (26) we get  Considering, X * = [ 30, 0]. But this solution set violates (27) and (28)  For, X * = [ 45, 75]. But this solution set violates (27) Example 2… D Nagesh Kumar, IISc 25 Water Resources Systems Planning and Management: M2L3

Case 2:  Using in (23) and (24), we have  Substitute (31) in (26), we have  For this to be true, either Example 2… D Nagesh Kumar, IISc 26 Water Resources Systems Planning and Management: M2L3

 For,  This solution set violates (27) and (28)  For,  This solution set is satisfying all equations from (27) to (31) and hence the desired  Thus, the solution set for this optimization problem is X * = [ 80, 40 ] Example 2… D Nagesh Kumar, IISc 27 Water Resources Systems Planning and Management: M2L3

BIBLIOGRAPHY / FURTHER READING 1. Rao S.S., Engineering Optimization – Theory and Practice, Fourth Edition, John Wiley and Sons, Ravindran A., D.T. Phillips and J.J. Solberg, Operations Research – Principles and Practice, John Wiley & Sons, New York, Taha H.A., Operations Research – An Introduction, 8 th edition, Pearson Education India, Vedula S., and P.P. Mujumdar, Water Resources Systems: Modelling Techniques and Analysis, Tata McGraw Hill, New Delhi, D Nagesh Kumar, IISc 28 Water Resources Systems Planning and Management: M2L3

Thank You D Nagesh Kumar, IISc Water Resources Systems Planning and Management: M2L3