1. KKT Practice and Second Order Conditions from Nash and Sofer
NLP
KKT Practice and Second Order Conditions from Nash and Sofer

2. Unconstrained First Order Necessary Condition Second Order Necessary
Second Order Sufficient

3. Easiest Problem
Linear equality constraints

4. KKT Conditions Note for equality – multipliers are unconstrained
Complementarity not an issue

5. Null Space Representation
Let x* be a feasible point, Ax*=b.
Any other feasible point can be written as x=x*+p where Ap=0
The feasible region
{x : x*+p pN(A)}
where N(A) is null space of A

6. Null and Range Spaces
See Section 3.2 of Nash and Sofer for example

7. Orthogonality

8. Null Space Review

9. Constrained to Unconstrained
You can convert any linear equality constrained optimization problem to an equivalent unconstrained problem
Method 1 substitution
Method 2 using Null space representation and a feasible point.

10. Example
Solve by substitution
becomes

11. Null Space Method
x*= [4 0 0]’
x=x*+Zv
becomes

12. General Method There exists a Null Space Matrix
The feasible region is:
Equivalent “Reduced” Problem

13. Optimality Conditions
Assume feasible point and convert to null space formulation

14. Where is KKT? KKT implies null space Null Space implies KKT
Gradient is not in Null(A), thus it must be in Range(A’)

15. Lemma 14.1 Necessary Conditions
If x* is a local min of f over {x|Ax=b}, and Z is a null matrix
Or equivalently use KKT Conditions

16. Lemma 14.2 Sufficient Conditions
If x* satisfies (where Z is a basis matrix for Null(A))
then x* is a strict local minimizer

17. Lemma 14.2 Sufficient Conditions (KKT form)
If (x*,*) satisfies (where Z is a basis matrix for Null(A))
then x* is a strict local minimizer

18. Lagrangian Multiplier
* is called the Lagrangian Multiplier
It represents the sensitivity of solution to small perturbations of constraints

19. Optimality conditions
Consider min (x2+4y2)/2 s.t. x-y=10

20. Optimality conditions
Find KKT point Check SOSC

21. In Class Practice
Find a KKT point
Verify SONC and
SOSC

22. Linear Equality Constraints - I

23. Linear Equality Constraints - II

24. Linear Equality Constraints - III
so SOSC satisfied, and x* is a strict local minimum
Objective is convex, so KKT conditions are sufficient.

25. Next Easiest Problem Linear equality constraints
Constraints form a polyhedron

26. Close to Equality Case Equality FONC: x*
a2x = b
a2x = b
-a2
Polyhedron Ax>=b
a3x = b
a4x = b
-a1
a1x = b x*
contour set of function
unconstrained minimum
Which i are 0? What is the sign of I?

27. Close to Equality Case Equality FONC: x*
a2x = b
a2x = b
-a2
Polyhedron Ax>=b
a3x = b
a4x = b
-a1
a1x = b x*
Which i are 0? What is the sign of I?

28. Inequality Case Inequality FONC: x*
a2x = b
a2x = b
-a2
Polyhedron Ax>=b
a3x = b
a4x = b
-a1
a1x = b x*
Nonnegative Multipliers imply gradient points to the less than
Side of the constraint.

29. Lagrangian Multipliers

30. Lemma 14.3 Necessary Conditions
If x* is a local min of f over {x|Ax≤b}, and Z is a null-space matrix for active constraints then for some vector *

31. Lemma 14.5 Sufficient Conditions (KKT form)
If (x*,*) satisfies

32. Lemma 14.5 Sufficient Conditions (KKT form)
where Z+ is a basis matrix for Null(A +) and A + corresponds to nondegenerate active constraints)
i.e.

33. Sufficient Example
Find solution and verify SOSC

34. Linear Inequality Constraints - I

35. Linear Inequality Constraints - II

36. Linear Inequality Constraints - III

37. Linear Inequality Constraints - IV

38. Example
Problem

39. You Try
Solve the problem using above theorems:

40. Why Necessary and Sufficient?
Sufficient conditions are good for?
Way to confirm that a candidate point
is a minimum (local)
But…not every min satisifies any given SC
Necessary tells you:
If necessary conditions don’t hold then you know you don’t have a minimum.
Under appropriate assumptions, every point that is a min satisfies the necessary cond.
Good stopping criteria
Algorithms look for points that satisfy Necessary conditions

41. General Constraints

42. Lagrangian Function Optimality conditions expressed using
and Jacobian matrix
were each row is a gradient of a constraint

43. Theorem 14.2 Sufficient Conditions Equality (KKT form)
If (x*,*) satisfies

44. Theorem 14.4 Sufficient Conditions Inequality (KKT)
If (x*,*) satisfies

45. Lemma 14.4 Sufficient Conditions (KKT form)
where Z+ is a basis matrix for Null(A +) and A + corresponds to Jacobian of nondegenerate active constraints)
i.e.

46. Sufficient Example
Find solution and verify SOSC

47. Nonlinear Inequality Constraints - I

48. Nonlinear Inequality Constraints - II

49. Nonlinear Inequality Constraints - III

50. Sufficient Example
Find solution and verify SOSC

51. Nonlinear Inequality Constraints - V

52. Nonlinear Inequality Constraints - VI

53. Theorem 14.1 Necessary Conditions- Equality
If x* is a local min of f over {x|g(x)=0}, Z is a null-space matrix of the Jacobian g(x*)’, and x* is a regular point then

54. Theorem 14.3 Necessary Conditions
If x* is a local min of f over {x|g(x)>=0}, Z is a null-space matrix of the Jacobian g(x*)’, and x* is a regular point then

55. Regular point
If x* is a regular point with respect to the constraints g(x*) if the gradient of the active constraints are linearly independent.
For equality constraints, all constraints are active so
should have linearly independent rows.

56. Necessary Example Show optimal solution x*=[1,0]’
is regular and find KKT point

57. Constraint Qualifications
Regularity is an example of a constraint qualification CQ.
The KKT conditions are based on linearizations of the constraints.
CQ guarantees that this linearization is not getting us into trouble. Problem is
KKT point might not exist.
There are many other CQ,e.g., for inequalities Slater is there exists g(x)<0.
Note CQ not needed for linear constraints.

58. KKT Summary X* is global min Convex f Convex constraints
X* is local min CQ
SOSC
KKT Satisfied