1. Tutorial 7 Constrained Optimization Lagrange Multipliers
____________________________________________
Ordinary Differential equations
Math for CS
Tutorial 7
Tutorial 7

2. Constrained Optimization
Constrained optimization problem can be defined as following:
Minimize the function
,while searching among x, that satisfy the constraints:
Math for CS
Tutorial 7

3. Dimensionality Reduction
The straightforward method to solve constrained optimization problem is to reduce the number of free variables: If x=(x1,..xn) and there are k constraints g1(x)=0,…gk(x)=0, then, the k constraint equations can (sometimes) be solved to reduce the dimensionality of x from n to n-k:
Math for CS
Tutorial 7

4. Constrain for coordinate and relaxation for the gradient
We observe, that the more additional constraints are applied, the more restricted is the coordinate of the optimum (anywhere in R3 for k=0, on the surface for k=1, on the line for k=2), but the less restricted is the gradient of the function f(x) (zero, along the normal to the surface, within the plane, orthogonal to the line).
This requirement for the gradient to lie in the hyperspace, orthogonal to the intersection of the hypersurfaces, defined by the constraints can be summarized as:
(1)
Math for CS
Tutorial 7

5. Lagrange Multipliers
The second requirement is to satisfy the constraints:
The requirements (1-2) can be written together in the elegant form. Define the function:
Then, we can write to satisfy (1) and
to satisfy (2).
(2)
(2)
Math for CS
Tutorial 7

6. Lagrange Multipliers In other words, we have constructed a function
that depends on a variable , and we require
for that function.
(2)
Math for CS
Tutorial 7

7. Summary
The constants λi are called Lagrange Multipliers. We obtained that the solution (λ* , x*) of the constrained optimization problem
Satisfies the equations
,where
(2)
Math for CS
Tutorial 7

8. Problem 1
Minimizing f(x)=x+y under the constraint g(x)=x2+y2-1=0. The figure shows the circle defined by g(x)=0 and the lines of constant value of f(x).
g(x,y)= x2+y2-1=0 x*
Math for CS
Tutorial 7

9. Solution via Variable Elimination
From the constraint g(x)=x2+y2-1 we obtain:
Notice, that only negative y should correspond to the solution:
. Substituting this into the function we obtain the new problem: Find x, minimizing the function
Only negative value, is the solution.
Math for CS
Tutorial 7

10. Solution via Lagrange Function
(1)
(2)
(3)
From (1), we obtain Substituting in (2), we obtain:
. Substituting in (3) and solving it, we obtain , from which negative corresponds to minimum.
Math for CS
Tutorial 7

11. Ordinary Differential Equations
(2)
Linear Equations
Separable equations
Math for CS
Tutorial 7

12. First order differential equation
A first order linear differential equation has the following form:
To solve this equation, let us multiply both sides by :
(2)
Math for CS
Tutorial 7

13. Problem 2 Solve an equation:
This is the first order equation with p(x)=-2x
(2)
Math for CS
Tutorial 7

14. Separable Equations The differential equation of the form
is called separable, if f(x,y) = h(x)·g(y); that is,
In order to solve it, perform the following steps:
(1) Solve the equation g(y) = 0, which gives the constant solutions of (a);
(2) Rewrite the equation (a) as
(a)
(2)
Math for CS
Tutorial 7

15. Separable Equations (3) Now we can integrate to obtain (2)
(4) Now we can write down all the solutions, obtained in (1) and (2). If this is an IVP, we must use an initial to find a particular solution.
(2)
Math for CS
Tutorial 7

16. Problem 3
Solve the equation:
(2)
Math for CS
Tutorial 7

17. Problem 4
Solve the equation:
(2)
Math for CS
Tutorial 7