1. A Novel of Step Size Selection Procedures for Steepest Descent Method
Applied Mathematical Sciences, Vol. 6, 2012, no. 51, 2507 – 2518
A Novel of Step Size Selection Procedures for Steepest Descent Method
1 Goh Khang Wen, 2Mustafa Mamat, 2Ismail bin Mohd, 3Yosza Dasril
1Department of Physical and Mathematical Science Faculty of Science, Universiti Tunku Abdul Rahman, Malaysia.
2Department of Mathematics
Universiti Malaysia Terengganu, Kuala Terengganu, Malaysia.
3Department of Industrial Electronics
Faculty Electronics and Computer Engineering, Universiti Teknikal Malaysia Melaka (UTeM), Malaysia.
Abstract: It is well known that the classical steepest descent method which uses the exact line search procedure in determining the step size is converge very slowly to the solution. By the way, several effective inexact line search procedures have been proposed to overcome the weakness. Beside that, Barzilai and Borwein (BB) have proposed two surprising non-line search procedures in determining the step size for steepest descent method which has been proved to be R-superlinearly convergent for convex quadratic in two-dimensional space. However, in order to give the greatest possible reduction to the objective function along the search direction, we have introduced a Newton-like exact line search procedure. First in this paper, we will elaborate several well known step size selection procedures in more detail. Then, a numerical performance comparison of all selected step size selection procedures has been done. The results showed the Newton-like exact line search has performed better than others well known procedures.

2. 1. Introduction 2508 Goh Khang Wen et al
The classical steepest descent method which design by Cauchy (1847) can be considered among the most important procedures for minimization of real-valued function defined on Rn . The steepest descent step size appeared in
xk+1 =xk +λkdk , (1.1) which the step size λ k is obtained using exact line search [4]
λk =arg λmin>0 f(x k +λdk ), (1.2) λk = min{ | (
λ f x k λ d k
λ >0
∇ + ) = 0} (1.3)
respectively where dk = −∇f (xk).
However, for solving some complicated optimization problems, it is difficult to compute the step size λk using (1.2) and (1.3) in practical computational and some
time even impossible to compute it [5,13,14]. Therefore, there are several inexact line search condition have been introduced, such as Armijo condition [1], Goldstein condition [7], or Wolfe condition [12] in determined the step size for steepest descent method. It is easy to show that the steepest descent method with those condition, is always convergent and theoretically the method will only terminate after a stationary point is found. According to the previous study of the inexact line search condition implemented to steepest descent method [2,9,13], we found that Armijo line search and Backtracking line search [10] are easier to apply and more effective compared to others.
Line search procedure is a useful and efficient technique for solving unconstrained optimization problems, especially for solving small and middle scale problems, such as determines a step size for steepest descent method. However, some researcher mentioned that if line search procedure is constructed every iteration of the approximation, it will lead a significant amount of computational cost. Therefore, in order to reduce the evaluations of objective functions and gradient, several researchers have tried to avoid using line search in their designed algorithms. In year 1988, Barzilai and Borwein (BB) have derived two-point step size for the steepest descent approximating the secant equation [3]. They have proposed two new step size formulae to compute the step size λk for steepest descent method which have been
proved to be R-superlinearly convergent for convex quadratic in two dimension space.

3. 2. Step Size Steepest descent method 2509
In year 1989, JM method [8] which improve the BB methods by takes the average value of both BB methods.
Even though there are several inexact line searches and non-line search procedures have been proposed, but in the previous studies on line search, we have found that only the exact line search can gives the greatest possible reduction to the objective function along the search direction. All the exact line search procedures which must satisfy the condition (1.2) are always computed by using (1.3). But the previous researchers have shown that the weakness of the (1.3) in computational for determining the step size A k [5,13,14]. Therefore, according to Newton-Raphson
method, we have introduced an alternative Newton-like exact line search [5,6] and we have proved that it is practically computational available in our preliminary study of the method.
This paper is organized as follows. In Section 2, we elaborate several mentioned procedure for determining the step size Ak for steepest descent method. In Section 3, we describe the Newton-like exact line search procedure and present the algorithm of steepest descent method. Section 4 contains numerical results of the testing examples for implementing the methods discuss in this paper. The conclusion which ends this paper is discussed in Section 5.
2. Step Size
How to select the suitable step size for steepest descent method? In the early studies of this method, there are a varieties of research have been done in determining a suitable step size for the algorithm in solving optimization problems. Several well known procedures which mentioned in previous section, have been selected and will be discussed in this Section.
2.1 Cauchy’s step size (exact line search)
The classical and oldest steepest descent step size Ak which was designed by Cauchy, is computed as
g g
A = , (2.1)
k k
k gAg 2
where g k = ∇ f ( x k ) and A = ∇ f ( xk ) . The equation (2.1) can be proved as follows.

4. Consider the quadratic model problem given by
2510 Goh Khang Wen et al
Consider the quadratic model problem given by
1 T
min f(x)= x Hkx−b x, n 2
T x ∈ R . (2.1.1)
Since gk = ∇f(xk), we have
gk =Hkxk −b. (2.1.2)
The step size is computed by using the exact line search in (3.2), to give
∇f(xk −λkgk)=Hk(xk −λkgk)−b= 0 (2.1.3) By introducing (2.1.2) into (2.1.3), we obtain
gk − λkHkgk = 0 . (2.1.4)
Since step size 1
λ k ∈ R , (2.1.4) can be written as
gk gk = λ k gk Hkgk
T T
(2.1.5)
from which the equation (2.1) can be obtained.
Even the formula of the step size (2.1) can be proved by using quadratic model problem, but since there are no any suggested formula in solving complicated optimization problem and it is very hard to compute the step size λk using (1.3), we
still using (2.1) as Cauchy’s step size formula in solving complicated optimization
problems as listed in Section 4.1.
2.2 Armijo Line Search (inexact line search)
Zhen (2006) notices that among several well known inexact line search procedures which introduced by previous researchers, Armijo line search [1] is one of the most useful and easy to be implemented in computational. The Armijo line search rule is
described as follows. Given
s>0, β ∈ (0,1), σ ∈ (0,1) and λk = max{s,sβ, sβ ,... }

5. Steepest descent method 2511
such that
f x k + l k d k − f x k £ sl k g k d k (2.2) T
where dk = −gk = −Ñf(xk).
( ) ( )
2.3 Backtracking Line Search (inexact line search)
Neculai [9] and Burachik [2] say that backtracking line search is more simple and effective than other inexact line search methods [10]. Actually backtracking line search is a method obtained from modification of Armijo line search, and its procedure is as follows.
Procedure Backtracking (aER1: Ak)
! This procedure computes the step size l k using Backtracking line search
Set t=1.
while (
f x k + td k > f x k + s tg k d k , do
) ( ) T
2.1 t = t/ 2.
lk =t.
2.4 Barzilai and Borwein (BB method)
Barzilai and Borwein [3] derive two-point step size for the steepest descent routine by approximating the secant equation underlying quasi-Newton method, in which they have determined two step sizes l k by minimizing || Lx−lLg || and it’s symmetry
|| lLg − Lx || with respect to l , where L x = x k − x k −1 and L g = g k − g k −1 , from which we 2
obtain
L x L g
l = (2.3)
k L L
g g
and
LxLx T
l = (2.4)
k L L
x g
2.4 Mohd and Jaafar (JM method)

6. 3. Newton-like exact line search
2512 Goh Khang Wen et al
Mohd and Jaafar [8] have improved the BB methods in determining step size λk for steepest descent by taking the average value of both BB methods with λk is given by
1 [ ΔxTTΔg= 2 Δg
Δx Δx T   .
(2.5)
Δ x T
Δ g
3. Newton-like exact line search
All the exact line search procedure which must satisfied the condition of (1.2) is always computed using (1.3). But the previous researchers have showed the weakness of the (1.3) in computational for determining the step size λk [5,13,14]. Therefore,
according to Newton-Raphson method, in this section an alternative Newton-like exact line search is suggested and we have proved it is available in practical computational in our preliminary study of the method. Thus, instead of using (1.3) to satisfied the condition of (1.2), we use the approximation of Newton-Raphson’s idea to obtained the step size λk through the routine
λ = λ
k + 1 k
ϕ λ
′ ( ) k
ϕ′(λk) ,
(3.1) where
(p,(Ak) df(xk+Akdk) ER1 and (p,,(Ak) d2f(xk+ 2Akdk) R1
dAk
d~k
respectively.
The procedure of this alternative exact line search can be described as follows.
procedure compute. 2.k(A0,EER1: 2.k)
! This procedure computes λk
i =0
c(7
A = A ) ϕ (λ k )
while || λi+1 −λi || ≥ ε , do
3.1 i = i+1
ϕ λ
′ ( ) k
3.2 1
λ λ
i + = − ′
iϕ(λk )
λk =λi+ 1 .
return.■

7. 4. Numerical Example Steepest descent method 2513
3.1 Algorithm of Steepest Descent Method Procedure ASDM(m ,n +
Î Z , x0ER , EER : xER ).
! This procedure compute xER implements the algorithm of steepest descent method where m is the
! maximum number of iteration, n is the number of variable, x0 is the initial point, e is a tolerance.
k =0
dk=−Ñf(xk)
converge = false
while | dk |³ e and k £ m and not converge do
4.1. Use any step size procedure to compute the step size Ak xk+1 =xk +Akdk
4.3. if ||xk+1 −xk ||£e, then
x:=xk+1 ! The minimizer converge = true else
k = k dk =−Ñf(xk)
return. ∎
4. Numerical Example
In this section, we report a comparison results between those selected procedures in Section 2 and the Newton-like exact line search applied into steepest descent method Testing Problems
f(x1 ,x2)=2 x1+3x2 , (5,5) x 0 = 2
f ( x 1 , x 2 ) = 4 x 1 + x 2 − x 1 x , (1,1) x 0 =
si x hump came back functi on x 6
f ( x , x ) = 4 x 2 − 2. 1 x + − x x − 4 x + 4 x , ( 1 . 6,0.9) 3
x 0 = −
Three hump came back function x 6
f x x = x 2 −
( , ) x + − x x + x , ( 1 . 6,0.9)
4 1 2 6
Booth function
f x x = x + x − + x + x −
( , ) ( 2 7) (2 5) , (0.4,1 .6)
x 0 =

8. Table 1: Iteration Number in Solving Problems
2514 Goh Khang Wen et al
4 2 2 1
x x
6. ( , ) , ( 1,1)
= − +
x 2
f x x 1 x x 0 = −
1 2 1
7. + − + 2
f ( x , x ) = (2 x x − x ) 6( x x x x 0 =
) , (1,1)
8. Goldstein-Price function f(x1,x2) = g(x)h(x)
( ) [1 ( x x 1) (
2 3 )]
g x x x 2
= − + − x + x x + x ,
( ) [30 (2 3 ) ( )]
h x x x x x 2
= + − − + + x − x x x , ( 1, 1)
x 0 = − − .
9. ( , ) = 0. 5 x 2
f x x (1 − cos( 2 x )) + x , (1,1)
10. The two dimension function
x 0 =
( , ) = [1 − 2 + sin( 4 ) ] [ 0.5 sin( 2 )]
f x x x c p x − x + x − p x , c = 0. 2 , (6, 2)
x 0 = −
11. ( , ) = [1 − 2 + sin( 4 ) ] [ 0.5 sin( 2 )]
f x x x c p x − x + x − p x , c = 0.5 , (0,0)
12. f ( x , x ) = [1 − 2 x + c sin( 4 p x ) −
x + − , c = 0.05 , ( 1,1)
2 2
1 ] [ x 0.5 sin( 2 p x )]
x 0 = −
2 1
4.2 Numerical Results
Each step size procedure which has been discussed in previous section has been implemented into steepest descent method and the algorithms have been programmed into visual C++ language. Our testing problems and the initial points used are shown in Section 4.1. For each problem, the limiting number of iteration is set to 100,000
and the tolerance e = . For Armijo line search procedure, we use s = 1 , b = 0.75 , 10−6
a = suggested by Zhen [14] and backtracking line search procedure, we use a = proposed by Neculai [9]. Beside that, Vrahatis [11] have noticed that, a “small” step size has to be chosen in order to avoid oscillation and to guarantee the steepest descent convergence. Therefore, we use the initial step size A 0 = for our Newton- like exact line search procedure. P
Cauchy
Armijo
Backtracking
BB1
BB2 JM
Newton 1 11 10 24 2 21 15
199 13 12 3 7 20 6 4 51 32 37 16 17 5 8 9 58 71 F 56 55 75 19
268 33 49
Table 1: Iteration Number in Solving Problems

9. 5. Conclusion Steepest descent method 2515
The number of iteration which used to solve each problems by all the selected step size procedures are shown in Table 1, where BB1, BB2, JM and Newton stand for (2.3), (2.4), (2.5) and Newton-like exact line search procedure respectively. The bolded numbers in the Table 1 represent the least of iteration number among all the procedures for each problem. The symbol ‘F’ stand for failure in which mean that the algorithm still cannot find the solution when the iteration number reach at 100,000 or the solution determined by the algorithm does not satisfied the minimization properties
f ( x ) f ( x + A d ) ≤ f ( x ) ≤ f ( x ) . (4.1) k + =
5. Conclusion
1 k k k k 0
Stephen and Areila [10] notice that although Newton’s method needs less iteration than steepest descent method for obtaining the solution, but labour cost of Newton’s method is higher than steepest descent since the Newton’s method requires more memory space and expensive in calculating the inverse of the Hessian matrix.
Since in our Newton-like exact line search, the first and second derivatives
with respect to A k are defined as ~~~~~~ ~ ~~~~~~~~~~~
~~~ and ~~~~~~~ ~
~~~~~~~~~~~~
~ ~~~ , therefore it does not have the weakness which mentioned above but
~~~
it does having the fast converge advantage of Newton’s method.
According to the comparison results shown in Section 4.2, we can say that the step size obtained by using the Newton-like exact line search is more effective compared to other step size procedures and even there are three problems it does not obtain the solution with the least iteration, but those number of iteration is not much more than the least one as seeing in Table 1. From the numerical results, we also found that the well known BB’s methods are easy to failure in solving complicated optimization problems.
Acknowledgement. The authors are grateful acknowledge financial support from the Government of Malaysia and Universiti Malaysia Terengganu through Fundamental Research Grant Scheme (Vot 59200).

10. References 2516 Goh Khang Wen et al
Armijo L., (1966). Minimization functions having Lipschitz continuous first partial derivative, Pacific J. Math 16, 1-3.
Burachik, R., Drummond, L. M. G., Iusem, A. N. and Svaiter, B. F. (1995). Full convergence of the steepest descent method with inexact line searches, Optimization
Barzilai J. and Borwein J.M., (1988). Two point step size gradient method. IMA J. Numer. Anal., 8,
Curry H.B., (1944). The method of steepest descent for nonlinear minimization problems, Quart. Appl. Math., 2,
Goh Khang Wen and Ismail Bin Mohd, (2006) Analyze steepest descent using maple, Computer Science and Mathematics Symposium (CSMS2006), Kolej Universiti Sains dan Teknologi Malaysia.
Goh, K. W. & Ismail, B. M. (2007). An Alternative Newton-like Exact Line Search for Steepest Descent Method. Proceedings of 3rd International Conference on Research and Education in Mathematics 2007, Universiti Putra Malaysia, Applied Mathematics and Mathematics Education, page
Goldstein A. A., (1965). On steepest descent, SIAM Journal of Control, 3,
Ismail Bin Mohd and Azmi Bin Jaafar, (1990). A modification of Two- point step size gradient method for unconstrained optimization, Sains Malaysiana 19(4),

11. Steepest descent method 2517
Neculai Andrei, (2004). A new descent gradient method for unconstrained optimization, Research report, Research Institute for Informatics, Romania.
Stephen G. Nash and Ariela Sofer, (1996). Linear and nonlinear programming, George Mason University, The McGraw-Hill Companies, Inc..
Vrahatis M.N., Androulakis G.S., Lambrinos J.N., Magoulas G.D., (2000). A class of gradient unconstrained minimization algorithms with adaptive stepsize, Journal of Computational and Applied Mathematics, 114,
Wolfe P., (1969). Convergence conditions for ascent methods, SIAM Rev. 11:
Zhen-Jun Shi and Jie Shen, (2006a). On step-size estimation of line search methods, Applied Mathematics and Computation, 173,
Zhen-Jun Shi and Jie Shen, (2006b). Convergence of descent method with new line search, Journal Applied Mathematics and Computing, 20(1-2),

12. 2518 Goh Khang Wen et al
Received: January, 2012