Presentation is loading. Please wait.

Presentation is loading. Please wait.

Chapter 5 Algorithms to Seek Optimum

Similar presentations


Presentation on theme: "Chapter 5 Algorithms to Seek Optimum"— Presentation transcript:

1 Chapter 5 Algorithms to Seek Optimum
Chen Shenyan School of Astronautics (BUAA)

2 Flow chart of structural optimization
Giving X0, P=0 Structural analysis and sensitivity analysis Design variables linking Constraint evaluation and deletion Constraint/objective functions approximation Establishing Approximate problem (AP) Solution of approximate problem stop X= P=P+1

3 Implementation of Structural Optimization

4 Content 5.1 Optimization for Unconstrained Problems
5.1.1Minimization of function with one variable Fibonacci search Golden section(0.618) search 5.1.2 Optimization of functions with multi-variables Gradient method Newton method Conjugate gradient algorithm Variable metric (DFP) algorithm 5.2 Optimization for Constrained Problems 5.2.1 Penalty Function Methods 5.2.2 The Feasible Direction Method

5 5.1 Optimization for Unconstrained Problems

6 1. The meanings to study unconstrained optimization
There are some design problems that can be treated as unconstrained except very close to the final minimum point. Some of the most powerful and convenient methods of solving constrained minimization problems involve the transformation of the problem into one of unconstrained minimization. The study of the unconstrained minimization techniques provides the basic understanding necessary for the study of the constrained optimization techniques. These methods are emerging as powerful solution techniques for certain engineering analysis problems.

7 5.1.1 Minimization of function with one variable

8 Fig 1 The iterative process of optimization

9 One-dimensional minimization methods
Analytical methods(Differential calculus methods) Numerical methods Elimination methods Fibonacci method Golden section method Unrestricted search Exhaustive search Dichotomous search Interpolation methods Requiring no derivatives (Quadratic) Requiring derivatives

10 Unimodal Function A unimodal function is one that only one peak(Maximum or minimum) in a given interval. The assumption of unimodality is made in all the elimination techniques. If a function is known to be multimodal (i.e., having several valleys or peaks), the range of the function has to be subdivided into several parts and each part treated separately as a unimodal function. A unimodal function can be a nondifferentiable or even discontinuous function.

11 Unimodal Function(cont)
This can be stated mathematically as follows. A function is unimodal if (i) implies that , and (ii) implies that where is the optimum (minimum) and . A function of one variable is said to be unimodal if, given that two values of the variable are on the same side of optimum, then the one nearer the optimum gives the better functional value, i.e., the smaller value in the case of a minimization problem.

12 If implies that , or lies in [ , b] If implies that , or lies in [a, ]
If a function is known to be unimodal in a given range, the interval in which the minimum lies can be narrowed down provided the function values are known at two different points in the range. If implies that , or lies in [ , b] If implies that , or lies in [a, ]

13 To choose two points Fibonacci search Golden section(0.618) search
Requirements: (1) Two points chosen make the interval to be reduced quickly. (2) The length is predictable after given iterations. Problems need to solve: (1) How to choose the first two points? (2) How to choose a new test point? Fibonacci search Golden section(0.618) search

14 Fibonacci method Fibonacci numbers {Fn}
Yielding the sequence 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89…

15 Procedure For original interval , selecting two test points as

16 Limitations of Fibonacci method
The initial interval in which the optimum lies has to be known. The function being optimized has to be unimodal in the initial interval. The exact optimum cannot be located in this method. Only an interval, known as the final interval will be known. The final interval can be made as small as desired by making more computations. The number of function evaluations to be used in the search or the resolution required has to be specified beforehand.

17 Golden section method This method is same as the Fibonacci method except for one difference. The difference is that in Fibonacci method the total number of experiments to be conducted has to be specified before beginning the calculation, whereas, this is not required in the golden section method. Reduced ratio of interval is constant in iteration, which is

18 Historical background of 0.618
Ancient Greek architects believed that a building having the sides d and b satisfying the relation will be having the most pleasing properties. It is also found in Euclid’s geometry that the division of a line segment into two unequal parts so that the ratio of the larger part to the whole equal to the ratio of the smaller to the larger, being known as the golden section or golden mean.

19 Flow chart of method

20 6.1.2 Optimization of functions with multi-variables

21 General iterative scheme
All the minimization methods are iterative in nature and hence they start from an initial trial solution and proceed towards the minimum point in a sequential manner. Require an initial point X1 to start the iterative procedure Differ from one another only in the method of generating the new point Xi+1 (from Xi), and in testing the point Xi+1 for optimality.

22 Methods classification
Direct search methods直接法(Do not require the derivatives of the function) Random search method 随机搜索法 Univariate method 单变量法 Pattern search method模式搜索法 Rosenbrock’s method of rotating coordinates Rosenbrock法 Simplex method单纯形法 Descent methods间接法(Require the derivatives of the function)下降法(需要函数的导数信息) Gradient method梯度法(Steepest decent method最速下降法) Newton’s method牛顿法 Conjugate gradient method共轭梯度法 Variable metric method变尺度法(Davidon-Fletcher-Powell, DFP)

23 A point X* will be a relative minimum (局部最优) of f(X) if the necessary conditions(必要条件):
正定

24 Direct search method直接法
Univariate method In this method, we change only one variable at a time, and seek to produce a sequence of improved approximations to the minimum point. Since only one variable is changed, the problem becomes a one-dimensional minimization problem.

25 Univariate method(cont)
In fact, the search procedure is continued by taking each coordinate direction in turn. This procedure is continued until no further improvement is possible in the objective function in any of the n directions of a cycle. The efficiency of the method is quite low especially for the functions with large number of variable dimensions

26 Gradient of a function函数梯度
Descent methods下降法 Gradient of a function函数梯度 The partial derivatives 偏导数of a function , with respect to each of the n variables are collectively called the gradient of the function and is denoted by All the descent methods make use of the gradient vector, either directly or indirectly, in finding the search directions.

27 Property of the gradient
If we move along the gradient direction from any point in n-dimensional space, the function value increases at the fastest rate. Hence the gradient direction/梯度方向 is called the direction of steepest ascent/最速上升方向, and the negative of the gradient vector /负梯度方向denotes the direction of steepest descent/最速下降方向. Unfortunately, the direction of steepest ascent is a local property /局部特性and not a global one/非全局特性.

28 Descent methods Gradient method(Steepest decent method)梯度法
Newton’s method牛顿法 Conjugate gradient algorithm 共轭梯度法and Variable metric method变尺度法(Davidon-Fletcher-Powell, DFP)

29 (1) Gradient method/Steepest descent method(梯度法/最速下降法)
The use of the negative of the gradient vector /负梯度方向as a direction for minimization was first made by Cauchy in 1847. In this method, we start from an initial trial point X1 and iteratively move towards the optimum point according to the rule where is the optimal step length along the search direction寻优方向 .

30 Flow chart of gradient method梯度法框图

31 Convergence criteria收敛准则

32 Gradient method(cont)
The method of steepest descent may appear to be the best unconstrained minimization technique since each one-dimensional search starts in the “best” direction. However, owing to the fact that the steepest descent direction is a local property, the method is not really effective in most of the problems.

33 Property

34 Summary for Gradient Method
It can be seen that the path is a zig-zag(锯齿形) in much the same way as the univariate method. For functions with significant eccentricity, the method settles into a steady n-dimensional zig-zag and the process will be hopelessly slow. It often be combined with other methods, and act in the first several iterations.

35 (2) Newton Method All the local minima X* of a continuously differentiable function f satisfy the necessary conditions The oldest method for solving a set of nonlinear equations is Newton’s method.

36 Newton method Suppose is a point close to the minimum point of , and the function can be approximated with Taylor expansion/泰勒展开 with the first three terms

37 Newton method In general, the higher order terms in Taylor’s expansion are not negligible and hence an iterative procedure has to be used to find the improved approximations.

38 Newton method The sequence of points X1, X2, …, Xi+1 can be shown to converge to the actual solution X* from any point X1 sufficiently close to the solution X*, provided that is nonsingular 非奇异. The conditions are, however, very restrictive and the method frequently fails to converge.

39 Modified Newton Method
If f(X) is a quadratic二次式, we can find its minimum in a single step by using the above equation since the Taylor’s expansion is exact. If f(X) is a nonquadratic function, the Newton’s method may sometimes diverge发散, and it may converge to saddle points鞍点 and relative maxima.

40 Flow chart of Newton method

41 Modified Newton method
Advantages: It will find the minimum in lesser number of steps compared to the original method. It finds the minimum point in all the cases whereas the original method may not converge in some cases. It usually avoids convergence to a saddle point or a maximum.

42 Modified Newton method
In spite of these advantages, the method is not generally used in practice due to the following features of the method: It requires the storing of the n*n matrix , It becomes very difficult and sometimes, impossible to compute the elements of the matrix , It requires the inversion of the matrix逆矩阵 at each step, It requires the evaluation of the quality at each step. These features make the method impractical for problems involving a complicated objective function with large number of variables.

43 (3) Conjugate gradient algorithm and variable metric (DFP) algorithm
Definition: C Conjugate C--- symmetric matrix. A set of vectors If are said to be C-conjugate.

44 Conjugated vectors are linear independent
If are linear dependent, there must exist which let Both sides of the above Eq are multiplied , we have considering the definition, it comes and This conclusion is obviously conflict with the condition “linear dependent”.

45 Quadratically Convergent
Theorem: If a quadratic function is minimized sequentially, once along each direction of a set of n linearly independent, A-conjugate directions, the global minimum of Q will be located at or before the nth step regardless of the starting point.

46 Conjugate gradient algorithm
Since any general function can be approximated reasonably well by a quadratic near the optimum point, any quadratically convergent method is expected to find the optimum point in a finite number of iterations. Conjugate gradient algorithm takes conjugated vectors as search directions

47 Fletcher-Reeves method
Start with an arbitrary initial point Set the first direction Find the point according to the relation Set i=2 Find , and set Find the new point Test the optimality of the point …

48 Conjugate gradient algorithm
Since the directions used in this method are A-conjugate, the process should converge in n-cycles or less for a quadratic function. For ill-conditioned quadratics (whose contours are highly eccentric and distorted), the method may require much more than n-cycles for convergence. After every m=n+1 steps, equal to instead of the usual form.

49 Variable Metric Method变尺度法 (Davidon-Fletcher-Powell Method, DFP)
Significant developments have taken place in the area of decent techniques with the introduction of the variable metric method by Davidon in 1959. This method was extended by Fletcher and Powell in 1963. This method is the best general purpose unconstrained optimization technique making use of the derivatives that is currently available.

50 DFP The search direction of Newton method

51 DFP Iterative formula When

52 DFP Two problems need to solve: How to select ? How to construct ?

53 Characteristics of

54 should be a symmetrical positive definite matrix.
How to select ? √ should also satisfy the Quasi-Newton condition.

55 Quasi-Newton condition

56 Quasi-Newton condition
If is reversible, Let

57 Quasi-Newton condition
can be constructed by the information of gradient and vector

58 Iteration Formula of DFP

59 Iteration Formula of DFP
(6.11) Assuming (6.12) (6.13)

60 Iteration Formula of DFP
Substitute Eq.(6.12) into Eq.(6.11) (6.14) Compare both sides of Eq.(6.14) (6.15)

61 Iteration Formula of DFP
Substitute Eq.(6.13) into Eq.(6.15) (6.16)

62 Iteration Formula of DFP
substitute Eq.(6.16) into Eq.(6.12) How to construct ? √ (6.17)

63 Flow Chart of DFP

64 Characteristics of DFP
are is positive definite symmetric matrix for quadric function. --conjugate It should be pointed out that if after n times search or after search along , the convergence is still not reached, the iteration has to reset

65 5.2 Optimization for Constrained Problems

66 Feasible region / infeasible region

67 Active/critical constraints临界约束

68 Constrained Optimization Techniques
Indirect methods间接法 By the transformation of variables Penalty function method罚函数法 Interior penalty function method内点法 Exterior penalty function method外点法 Direct methods直接法 Heuristic search methods Heuristic搜索法 Constraint approximation methods约束近似法 Methods of feasible directions可行方向法 Rosen method Zoutendijk method Gellatly method

69 Penalty Function Methods罚函数法
The constrained problem is transformed into a sequence of unconstrained minimization problems such that the constrained minimum can be obtained by solving the sequence of unconstrained minimization problem. Interior/exterior penalty function method The sequence of unconstrained minima lie in the feasible/infeasible region. It converges to the constrained minimum from the interior/exterior of the feasible region.

70 Penalty Function Method
Penalty function method transform the basic optimization problem into alternative formulations such that numerical solutions are sought by solving a sequence of unconstrained minimization problems.

71 Penalty Function Method
-some function of the constraint -a positive constant known as penalty parameter罚因子 The second term on the right side of the equation is called the Penalty Term惩罚项.

72 Penalty Function Method
If the unconstrained minimization of function is repeated for a sequence of values of the penalty parameter , the solution may be brought to converge to that of the original problem. This is the reason why the penalty function methods are also known as sequential unconstrained minimization techniques(SUMT).

73 Penalty Function Method
Exterior penalty function method外罚函数法 and Interior penalty function method内罚函数法 and

74 Exterior Penalty Function Method
The sequence of unconstrained minima lie in the infeasible region. It converges to the constrained minimum from the exterior of the feasible region. and Constraint is violated Constraint is satisfied

75 Exterior Penalty Function Method

76 Algorithm of Exterior Penalty Function Method
Start from any design and a suitable value of , set k=1; Find the vector that minimizes the function Test whether the point satisfies all the constraints. If is feasible, it is desired optimum and hence terminate the procedure; Otherwise, go to step IV; Choose the next value of the penalty parameter which satisfies the relation and set the new value of k as original k plus one and go to step II.

77 Convergence criteria

78 Flow chart of exterior penalty function method

79 Exterior Penalty Function Method
It can deal with both equality and inequality constraints. The sequence of unconstrained minima lie in the infeasible region. It converges to the constrained minimum from the exterior of the feasible region. Since the can not be , the final still locate in infeasible region. This is an important property of exterior penalty function method.

80 Interior Penalty Function Method
The sequence of unconstrained minima lie in the feasible region. It converges to the constrained minimum from the interior of the feasible region.

81 Interior Penalty Function Method
and

82 Algorithm of Interior Penalty Function Method
Starting with an initial feasible point satisfying all the constraints with strict inequality sign, that is ,for , and an initial value of Set k=1. Minimize by using any of unconstrained minimization methods and obtain the solution Test whether is found to be optimum, terminate the process. Otherwise, go to next step. Find the value of the next penalty parameter Set the new value of k=k+1, take the new starting point as and go to step II.

83 Initial value of the penalty parameter

84 Interior Penalty Function Method
The points are all inside the feasible region. During the iterations, sometimes is not a optimum, it is still a relatively better solution which can be accepted. The initial point should located in the feasible region, that is all the constraints should be satisfied, which made it difficult to find an initial feasible design sometimes. The interior penalty function method can not deal with equality constraints.

85 Interior Penalty Function Method
The penalty term in interior penalty function method is not defined if X is infeasible. This introduces serious shortcoming while using the Eq.

86

87 Extended interior penalty
Quadric extension Linear extension

88 Extended interior penalty
Linear extension

89 Extended interior penalty
Quadric extension NEWSUMT

90

91

92 Summary of Penalty Function Methods
The penalty function methods transfer a constraint optimization problem into a sequence of unconstraint optimization problems, which are much easier to be solved. It does not like the direct method which need spent much computational time in analyzing the constraint functions, which will be more serious when there are many constraints.

93 Summary of Penalty Function Methods
The penalty function methods are widely applied in practical constraint optimization problems. The main reason is that there are no specific properties required for the objective function and constraint functions . They need to solve a series of unconstraint minimization problems which brought large calculation. When changing, the ill-condition of penalty functions will become more serious which made the unconstraint minimization problems are difficult to be solved.

94 Methods of feasible directions
To produce an improving succession of feasible vectors , by moving in a succession of usable feasible directions. A usable feasible direction is a feasible direction along which the objective function can be reduced at least by a small amount. Each iteration consists of two important steps: Finding a usable feasible direction at a specified point; Determining a proper step length along the usable feasible direction found in the first step.

95 Methods of Feasible Directions
The methods of feasible directions are based on the same philosophy as the methods of unconstrained minimization but are constructed to deal with inequality constraints. The basic idea is to choose a starting point satisfying all the constraints and to move to a better point according to the iterative scheme

96 Methods of Feasible Directions
is the starting point for the ith iteration is the direction of movement is the distance of movement(step length) is the final point obtained at the end of the ith iteration.

97 The search direction A small move in that direction violates no constraint. The value of the objective function can be reduced in that direction. The new point is taken as the starting point for the next iteration and the whole procedure is repeated several times until a point is obtained such that no direction satisfying both I and II can be found.

98 Methods of Feasible Directions
In general, such a point denotes the constrained local minimum of the problem. This local minimum need not be a global one unless the problem is a convex programming problem. A direction satisfying the property I is called feasible while a direction satisfying both the properties I and II is called a usable feasible direction. There are many ways of choosing usable feasible directions and hence there are many different methods of feasible directions.

99 The search direction A small move in that direction violates no constraint – Feasible condition (可行性条件) The value of the objective function can be reduced in that direction – Usable condition(适用性条件)

100 Feasible condition

101 Feasible condition The geometry meaning of the equation is that the vector S must make an obtuse angle with all the constraint normals except that, for the linear or concave constraints, the angle may go to as less as 90 °. Any feasible direction satisfying the strict inequality sign of the Eq. lies at least partly in the feasible region. By moving along such a direction from , one will be able to find another point which also lies in the feasible region.

102 Usable condition

103 A usable feasible direction
Feasible condition Usable condition

104 A usable feasible direction

105 To find the best The best usable feasible direction is a feasible direction of minimum objection function , i.e., the steepest decrease direction.

106 To find the best

107 To find the best

108 To find the best

109 To find the best

110 To find step length The optimal step length can be found by any of the one-dimensional minimization methods. The constraints will not be considered while finding with these methods. The new point may lie either in the interior of the feasible region, or on the boundary of the feasible region, or in the infeasible region.

111 To find step length in the interior of the feasible region
There are no active constraints and hence we proceed to the next iteration by setting the new usable feasible direction as

112 To find step length lie on the boundary of the feasible region.
We generate a new usable feasible direction by solving a new direction finding problem.

113 To detect active constraint
In order to detect that the point is lying on the constraint boundary, we have to find whether one or more are zero. Since the computations are done numerically, will we say that the constraint is active if A small margin has to be specified to detect an active constraint.

114 To detect active constraint
is a prescribed small number.

115 To find step length lie in the infeasible region.
The step length has to be reduced(corrected) so that the resulting point lies in the feasible region only.

116 To find step length still in feasible region. go to infeasible region.
Heuristic search(探索式寻优) still in feasible region. go to infeasible region.

117 Termination criteria How to judge a point lie in the boundary of constraints is optimum point or not?

118 Termination criteria

119 Kuhn-Tucher conditions

120 Termination criteria

121 Termination criteria If , then the point is not optimum point.
If is optimum, then and Thus, the above condition will be same as KT conditions. Therefore, the introduction of is helpful to convergence tests.

122 Termination criteria (1.a) (1.b)

123 Termination criteria (1.a) (1.b) Multiply to both sides of Eq.(1.a)

124 Termination criteria There will be three possibility: (1)
KT conditions are satisfied, the present point is optimum. (2) When one or several a certain and its corresponding should be removed. Then resolve the modified Eq.1, and will be obtained. (3) the present point is not optimum, it need to find new usable feasible direction

125 Flow chart of feasible direction method

126 Summary to feasible direction methods
To produce an improving succession of feasible vectors , by moving in a succession of usable feasible directions. When there are many nonlinear constraints, the feasible direction methods are usually not so effective. Equality constraints can not be dealt with by them.

127 Chapter 6 Algorithms to Seek Optimum
6.1 Optimization for Unconstrained Problems 6.1.1Minimization of function with one variable Fibonacci search Golden section(0.618) search 6.1.2 Optimization of functions with multi-variables Gradient method Newton method Conjugate gradient algorithm Variable metric (DFP) algorithm 6.2 Optimization for Constrained Problems 6.2.1 Penalty Function Methods 6.2.2 The Feasible Direction Method

128 5.1 Optimization for Unconstrained Problems
6.1.1Minimization of function with one variable

129 5.1.1Minimization of function with one variable
Fibonacci search Golden section(0.618) search

130 5.1.2 Optimization of functions with multi-variables

131 Gradient method/Steepest descent method(梯度法/最速下降法)
In this method, we start from an initial trial point X1 and iteratively move towards the optimum point according to the rule where is the optimal step length along the search direction

132 Newton method

133 DFP The search direction of Newton method

134 Iteration Formula of DFP
(6.17)

135 5.2 Optimization for Constrained Problems

136 Constrained Optimization Techniques
Indirect methods By the transformation of variables Penalty function method Interior penalty function method Exterior penalty function method Direct methods Heuristic search methods Constraint approximation methods Methods of feasible directions Rosen method Zoutendijk method Gellatly method

137 Flow chart of structural optimization
Giving X, P=0 Structural analysis and sensitivity analysis Design variables linking Constraint evaluation and deletion Constraint/objective functions approximation Establishing Approximate problem (AP) Solution of approximate problem stop X= P=P+1

138 Thank you!


Download ppt "Chapter 5 Algorithms to Seek Optimum"

Similar presentations


Ads by Google