1. Introducción a la Optimización de procesos químicos. Curso 2005/2006 BASIC CONCEPTS IN OPTIMIZATION: PART II: Continuous & Unconstrained Important concepts for the optimization of systems with continuous variables and non-linear equations. Since we will limit the topic to unconstrained problems, we will concentrate on the OBJECTIVE FUNCTION. Optimality Conditions for Single Variable Optimality Conditions for Multivariable Variable Revisit Convexity and Its Importance

2. Introducción a la Optimización de procesos químicos. Curso 2005/2006 BASIC CONCEPTS IN OPTIMIZATION: PART II: Continuous & Unconstrained Wait a minute. No problem is unconstrained; so, why do we need to know this? Unconstrained problems - sometimes the solution doesn’t involve constraints Used in methods for constrained problems

3. Introducción a la Optimización de procesos químicos. Curso 2005/2006 BUILDING EXPERIENCE IN OPTIMIZATION CLASS EXERCISE: The reactor is isothermal and the reaction kinetics are first order. Is this system linear or non-linear? What must we define before defining an optimum? - The goal is to maximize C B in the effluent at S-S - You can adjust only the flow rate of feed This is an isothermal CFSTR with the reaction: A  B  C You can only adjust F

4. Introducción a la Optimización de procesos químicos. Curso 2005/2006 WHAT DEFINES THE LOCATION OF AN OPTIMUM? Optimum For LP, the optimum is at a corner point. For NLP the optimum is located …….?

5. Introducción a la Optimización de procesos químicos. Curso 2005/2006 WHAT DEFINES THE LOCATION OF AN OPTIMUM: Single variable? The general definition of a minimum of f(x) is x* is a minimum if f(x*)  f(x* +  x) for small  x We will start with a single-variable system and then generalize to multiple variable. We will not yet include constraints. We want to apply this concept, but we need to determine specific criteria that test for conformance to the statement in the box above.

6. Introducción a la Optimización de procesos químicos. Curso 2005/2006 WHAT DEFINES THE LOCATION OF AN OPTIMUM: Single variable? Necessary Condition for a single-variable system: [df(x)/dx] x* = 0 Let’s look at the definition of a derivative, which is continuous If this exists and f(x*)  (f(x*+  x), then Why isn’t this sufficient for a minimum?

7. Introducción a la Optimización de procesos químicos. Curso 2005/2006 WHAT DEFINES THE LOCATION OF AN OPTIMUM: Single variable? Necessary Condition for a single-variable system: [df(x)/dx] x* = 0 (a) (b) (c) (d) (e) Where is the derivative zero?

8. Introducción a la Optimización de procesos químicos. Curso 2005/2006 WHAT DEFINES THE LOCATION OF AN OPTIMUM: Single variable? Sufficient condition: A function with f’(x*)=0 has f’’(x*) = …= f n-1 (x*) = 0 (the next n-1 derivatives = zero) has for n = even f n (x*) > 0 (the nth derivative at x* > 0 ) Approximate the function with a Taylor Series. 0 Remainder ( 0  h  1) 0

9. Introducción a la Optimización de procesos químicos. Curso 2005/2006 WHAT DEFINES THE LOCATION OF AN OPTIMUM : Single variable? Sufficient condition: A function with f’(x*)=0 has f’’(x*) = …= f n-1 (x*) = 0 (2nd to n-1 derivatives = zero) has for n = even f n (x*) > 0 (the nth derivative at x* > 0 ) Rearrange the result. For n = even, (  x) n > 0; when n th derivative is positive, the condition for a minimum is satisfied!

10. Introducción a la Optimización de procesos químicos. Curso 2005/2006 WHAT DEFINES THE LOCATION OF AN OPTIMUM : Single variable? Necessary & Sufficient Conditions: [df(x)/dx] x* = 0 ; d 2 f(x*)/dx 2 > 0 (a) (b) (c) (d) (e) Which satisfy the necessary & sufficient?

11. Introducción a la Optimización de procesos químicos. Curso 2005/2006 WHAT DEFINES THE LOCATION OF AN OPTIMUM : Single variable? Let’s look at the following examples. f 1 = 3 + 2x + 5x 2 df 1 /dx = 2 + 10x = 0 : x = -.20 d 2 f 1 /dx 2 = 10 > 0 at x = -.20 Therefore, the function has a local minimum at x = x* = -.20 f 1 = 3 + 2x - 5x 2 df 1 /dx = 2 - 10x = 0 : x =.20 d 2 f 1 /dx 2 = -10 < 0 at x =.20 Therefore, the function has a local maximum at x = x* =.20

12. Introducción a la Optimización de procesos químicos. Curso 2005/2006 WHAT DEFINES THE LOCATION OF AN OPTIMUM: Single variable? Are these results consistent with the methods you have learned previously? What do we conclude if n = odd? What type of extremum occurs for f(x) = x 4 ? Necessary & Sufficient Conditions: [df(x)/dx] x* = 0 ; d 2 f(x*)/dx 2 > 0

13. Introducción a la Optimización de procesos químicos. Curso 2005/2006 WHAT DEFINES THE LOCATION OF AN OPTIMUM: Single variable? Are these results consistent with the methods you have learned previously? Hopefully, these are the rules that you learned in first-year calculus! What do we conclude if n = odd? The sign of the remainder depends on the sign of  x. This is not a local minimum. It is termed a saddle point. Necessary & Sufficient Conditions: [df(x)/dx] x* = 0 ; d 2 f(x*)/dx 2 > 0

14. Introducción a la Optimización de procesos químicos. Curso 2005/2006 WHAT DEFINES THE LOCATION OF AN OPTIMUM: Single variable? What type of extremum occurs for f(x) = x 4 ? Necessary & Sufficient Conditions: [df(x)/dx] x* = 0 ; d 2 f(x*)/dx 2 > 0 Therefore, the extreme point is a minimum!

15. Introducción a la Optimización de procesos químicos. Curso 2005/2006 WHAT DEFINES THE LOCATION OF AN OPTIMUM: Multivariable? Necessary: Let’s extend these results to multivariable systems, with x a vector of dimension n. Necessary condition: We call these equations the “stationarity conditions”. The proof is similar to the single-variable case.

16. Introducción a la Optimización de procesos químicos. Curso 2005/2006 WHAT DEFINES THE LOCATION OF AN OPTIMUM : Multivariable? The necessary condition for unconstrained optimization of a multivariable system is often stated as the following. The gradient equaling zero is the stationarity condition.

17. Introducción a la Optimización de procesos químicos. Curso 2005/2006 WHAT DEFINES THE LOCATION OF AN OPTIMUM : Multivariable? Sufficient: Let’s extend these results to multivariable systems, with x a vector of dimension n. We will restrict sufficient conditions to second derivatives. The first and second differential is defined as

18. Introducción a la Optimización de procesos químicos. Curso 2005/2006 WHAT DEFINES THE LOCATION OF AN OPTIMUM : Multivariable? These terms can be used in the expression for a Taylor series to determine the sufficient condition. 0 Remainder ( 0  h  1) The condition for a minimum is satisfied when the remainder is positive.

19. Introducción a la Optimización de procesos químicos. Curso 2005/2006 WHAT DEFINES THE LOCATION OF AN OPTIMUM : Multivariable? H = the Hessian of second derivatives It is symmetric.

20. Introducción a la Optimización de procesos químicos. Curso 2005/2006 WHAT DEFINES THE LOCATION OF AN OPTIMUM : Multivariable? For a minimum, the right hand side is positive for any non-zero values of the vector  x. How can we tell? We need to evaluate an infinite number of values of  x! Let’s try a little mathematics to improve the situation

21. Introducción a la Optimización de procesos químicos. Curso 2005/2006 WHAT DEFINES THE LOCATION OF AN OPTIMUM : Multivariable? We will consider a two-dimensional system. We start by defining a new vector of variables, w. x1x1 x2x2 w2w2 w1w1 Can we define the b’s to make the test for optimality easier?

22. Introducción a la Optimización de procesos químicos. Curso 2005/2006 WHAT DEFINES THE LOCATION OF AN OPTIMUM : Multivariable? The optimality test would be easy if the hessian were diagonal. Then, If, How can we determine the b’s to give this nice, diagonal hessian matrix?

23. Introducción a la Optimización de procesos químicos. Curso 2005/2006 WHAT DEFINES THE LOCATION OF AN OPTIMUM : Multivariable? The answer is determined from the eigenvalues and eigenvectors of the hessian matrix!!! When we prove that the function f(w) has a minimum at w* from 1 > 0 and 2 > 0 we also prove that the function f(x) has a minimum at x*!

24. Introducción a la Optimización de procesos químicos. Curso 2005/2006 WHAT DEFINES THE LOCATION OF AN OPTIMUM : Multivariable? A schematic of what we did. The coordinates are rotated to express the quadratic as the sum of variables squared times eigenvalues. Clearly, the remainder term must only increase if all i are positive. w1w1 w2w2

25. Introducción a la Optimización de procesos químicos. Curso 2005/2006 WHAT DEFINES THE LOCATION OF AN OPTIMUM : Multivariable? Positive Definite: A matrix is positive definite if all values of its eigenvalues ( ) are positive. Eigenvalues are the solution to the following equation, with H evaluated at x*. | H - I | = 0 What is the form of this equation? How many solutions are there?

26. Introducción a la Optimización de procesos químicos. Curso 2005/2006 WHAT DEFINES THE LOCATION OF AN OPTIMUM : Multivariable? The following two conditions are necessary & sufficient at x* The gradient is zero The Hessian is positive definite Some good news - We do not typically perform these calculations to test problems But, these concepts are used in many solution methods for non-linear optimization.

27. Introducción a la Optimización de procesos químicos. Curso 2005/2006 WHAT DEFINES THE LOCATION OF AN OPTIMUM : Multivariable? This is a “nice” objective function, which is convex and symmetric. Local derivative information will direct us toward the minimum. All eigenvalues are positive.

28. Introducción a la Optimización de procesos químicos. Curso 2005/2006 WHAT DEFINES THE LOCATION OF AN OPTIMUM : Multivariable? This is an objective function with a ridge. We will find the valley quickly; then, we will search the ridge with little success. One eigenvalue is near zero.

29. Introducción a la Optimización de procesos químicos. Curso 2005/2006 WHAT DEFINES THE LOCATION OF AN OPTIMUM : Multivariable? This objective function has a saddle point, which has a minimum in one direction and maximum in another direction. Derivative information will not direct us well. One eigenvalue is positive, and another is negative.

30. Introducción a la Optimización de procesos químicos. Curso 2005/2006 WHAT DEFINES THE LOCATION OF AN OPTIMUM : Multivariable? What’s going on here? What is the hessian for these stationary points 1 2 3 4

31. Introducción a la Optimización de procesos químicos. Curso 2005/2006 Convexity and the objective function. A function of x (a vector) is convex if the following is true. For points x 1 and x 2 and 0    1. f(x) x Is this function convex (over the region in the figure)? CONVEXITY: AN IMPORTANT PROPERTY IN OPTIMIZATION

32. Introducción a la Optimización de procesos químicos. Curso 2005/2006 Convexity and the objective function. A function of x (a vector) is convex over a region if the following is true over the region. Gradient Test: Hessian Test: The function is convex if its Hessian matrix is positive definite positive CONVEXITY: AN IMPORTANT PROPERTY IN OPTIMIZATION

33. Introducción a la Optimización de procesos químicos. Curso 2005/2006 Any local minimum of a convex function (over an unconstrained region) is a global minimum! CONVEXITY: AN IMPORTANT PROPERTY IN OPTIMIZATION

34. Introducción a la Optimización de procesos químicos. Curso 2005/2006 BASIC CONCEPTS IN OPTIMIZATION: PART II: Continuous & Unconstrained Conclusions on OBJECTIVE FUNCTION properties Opt. Conditions for Single Variable Opt. Conditions for Multivariable Variable Convexity and Its Importance When is local = global optimum? Basis of many optimization algorithms and tests for convergence We seek to formulate our models to yield a convex programming problem

35. Introducción a la Optimización de procesos químicos. Curso 2005/2006 OPTIMIZATION BASICS II - WORKSHOP #1 What is difference between suff. condition for optimality and convexity? Why is convexity important? We covered the conditions for optimality and convexity in this section. They seemed similar.

36. Introducción a la Optimización de procesos químicos. Curso 2005/2006 OPTIMIZATION BASICS II - WORKSHOP #2 Since convexity is important, let’s evaluate convexity for a very important function. Is the following function convex or concave? with c i constants

37. Introducción a la Optimización de procesos químicos. Curso 2005/2006 OPTIMIZATION BASICS II - WORKSHOP #3 Any local minimum of a convex function (over an unconstrained region) is a global minimum! The statement below is very important. Prove the statement. Hint: Consider directions of improvement for convex and non-convex functions.

38. Introducción a la Optimización de procesos químicos. Curso 2005/2006 OPTIMIZATION BASICS II - WORKSHOP #4 All convex functions have a unique minimum, i.e., they are unimodal. Determine whether all unimodal functions are convex x f(x)

39. Introducción a la Optimización de procesos químicos. Curso 2005/2006 OPTIMIZATION BASICS II - WORKSHOP #5 We seek a global, rather than a local, optimum. Define a global optimum in words Determine a mathematical test for the global optimum. Discuss how you would find a global optimum.

40. Introducción a la Optimización de procesos químicos. Curso 2005/2006 OPTIMIZATION BASICS II - WORKSHOP #6 The objective function is often the sum of several functions, for example, costs, revenues, taxes, and so forth. Determine if the following are a convex functions, when each term [g i (x)] is convex individually.

41. Introducción a la Optimización de procesos químicos. Curso 2005/2006 OPTIMIZATION BASICS II - WORKSHOP #7 A function is convex if its Hessian matrix is positive definite over the range of the variable x Positive definite One way to determine if a matrix (the hessian) is positive definite is to evaluate the determinants of its principle minors. If they are positive, the matrix is positive definite. The principle minors are the sub-matrices formed by eliminating n-k columns and rows, with k = 0 to n-1. Apply this approach to the following functions.

42. Introducción a la Optimización de procesos químicos. Curso 2005/2006 OPTIMIZATION BASICS II - WORKSHOP #7 A function is convex if its Hessian matrix is positive definite over the range of the variable x Positive definite

43. Introducción a la Optimización de procesos químicos. Curso 2005/2006 OPTIMIZATION BASICS II - WORKSHOP #7 SOLUTION Therefore, the function is convex

44. Introducción a la Optimización de procesos químicos. Curso 2005/2006 OPTIMIZATION BASICS II - WORKSHOP #7 SOLUTION Therefore, the function is not convex

45. Introducción a la Optimización de procesos químicos. Curso 2005/2006 OPTIMIZATION BASICS II - WORKSHOP #7 SOLUTION Therefore, the function is convex