A STEP-LENGTH FORMULA FOR CONJUGATE GRADIENT METHODS

A newly step-length formula is proposed for implementing conjugate gradient methods’ algorithm to solve unconstrained optimization problems. The unified formula for obtaining step-length does not involve any matrix operation. Numerical results obtained are graphically illustrated using performance profiling software. This showed that the new formula performs efficiently in terms of computational efforts and execution time compared with some existing formulae for obtaining the step-length without line search procedures.


Introduction
We consider the conjugate gradient method (CGM) for the unconstrained optimization problem minf(x),x∈R n , where f is continuously differentiable and a real-valued function. The CGM constitutes an active choice for efficiently solving (1), particularly when the dimension n is very large, due to its simplicity analysis, very low memory requirement, fast convergence and its splendid numerical performance, for engineers and mathematicians (Jinhong & Genjiao, 2013). It uses the following iterative plot of the form: where, x (k) is the present iterative point, α k >0, the step-length which can be carried out by various step-length rules and d (k) is the search direction calculated by: where g (k) =∇f(x (k) ) and β k is the CG update parameter which decides the distinctive CGMs which in turn lead to very assorted computational efficiency and convergence results of the comparing methods (Zhang, 2010). The following are some well-known CG update parameters: Fletcher & Reeves (1964) [FR] (4) Polak & Ribiere (1969)[PRP] Hestenes & Stiefel (1952)[HS] Fletcher (1987)[CD] Dai & Yuan (2000)[DY] Liu & Storey (1992)[LS] (9) Bamigbola et al. (2010)[BAN] (10) Hager & Zhang (2005)[HZ], (11) where y (k) =(g (k+1) -g (k) ).
The success of an optimization method in the iterative scheme given in (2) depends largely on the accuracy of computing d (k) and α k .
In performing the iterative scheme in (2), various methods were applied to determine the step-length, one of such methods is the line search techniques. On the other hand, a fixed formula can also be used to obtain the step-length that is simply called step-length with no line search procedure (Sun & Zhang, 2001). But the line search schemes need many evaluations of some function and gradient expressions, thus reducing the numerical efficiency of CGMs in large-scale optimization problems (Zhang, 2010). Sun & Zhang (2001) developed a CGM where the step-length is computed by a formula in their method instead of a line search scheme. The following is their formula: where But the formula presented in (13) involves {Q (k) }, and this may take a toll extra memory space and execution time amid the numerical experiments for large-scale optimization issues. Wu (2011), later derived a formula for the step-length that is matrix-free by updating the formula in (13) using the quasi-Newton methods in Zhang et al., (1999), and the formula is presented as where , δ and γ are parameter satisfying, δ∈(0, K τ ) and γ≥0 if τ=K.
Ajimoti & Bamigbola (2016) derived a step-size formula that uses only gradient information to obtain the step-length in the iterative scheme (2) and the formula is given as In this work, the idea of enhancing the line search procedure is undertaking by prescribing a new step-length rule without a line search for CGMs involving both function and gradient information. The rest of the paper is constituted as follows. In Section 2, the derivation of the new formula is presented. In Section 3, numerical results illustrating the efficiency of the proposed formula and comparing its performance with two existing formulae using some CGMs were reported, thereafter, the conclusion was drawn in Section 4.

The New Step-length Formula
In this section, the new step-length formula is presented. The following assumptions which are commonly used in the literature on function f are required.

Assumption 2.1:
i. The level set L={x∈R n |f(x)≤f(x (1) ), with x (1) to be the current or starting point of the iterative method (2) is bounded.
ii. The objective function f(x) is Lipschitz continuous and differentiable function and strongly convex in ℜ , i.e., ∃ > 0 and ≥ 0 such that, and Lemma 2.1: Suppose that Assumption 2.1 hold. Then, holds for all k. Proof: ). Hence, the proof is straight forward by adopting (16) and (17).
Now we present the derivation of the step-length formula that uses available information on both function and gradient.
Using the Sun & Zhang (2001) approach, we have
Step 3: Compute step length α k by: Step 3a: Using a fixed formula in (14).

Numerical Comparison
The numerical experiments carried out in this work incorporated with fixed formulae in (14), and (25) into the CGM Algorithm 3.1. We aim to perform an experiment that aid in measuring the effectiveness of obtaining the step-length α k using a fixed formula rather than any line search procedures. The newly derived formula for finding step-length was compared with some existing fixed formulae by using nine kinds of conjugate gradient methods to solve thirty unconstrained optimization test functions obtained from the CUTE collection made available by Andrei (2008) and Jamil & Yang (2013) with standard starting points and each test function is given with two different dimensions (n=5,000 and n=10,000). The CGM Algorithm 3.1 was implemented via Matlab 8.0 version and run on a PC HP EliteBook 6930p with 2.00GB RAM, 2.20GHZ processor and 3.4 windows experience index operating system. In Figures 1-6, we adopt the performance profiles introduced by Dolan & More (2002) to evaluate and compare the performance of different kinds of CGM against various rules of obtaining the step size α k , to test the efficiency of these methods using optimization software based on the CPU time and the number of iteration where NSF (formula presented in (25)) represents new step-length formula, ESF1 (formula presented in (14)) represents existing step-length formula one and ESF2 (formula presented in (15)) represents existing step-length formula two. A table containing the test functions and their sources is presented in Table 1 and their numerical results are graphically illustrated in Figures 1-6. Linear Function -rank 1 (Andrei, 2008) 3 Quadratic Diagonal Perturbed Function (Andrei, 2008) 4 Perturbed Quadratic Function (Andrei, 2008) 5 Quadratic QF1 Function (Andrei, 2008) 6 ARGLINB(m=20) Function (Andrei, 2008) 7 Almost Perturbed Quadratic Function (Andrei, 2008) 8 Extended White & Holst Function (Andrei, 2008) 9 Raydan 1 Function (Andrei, 2008) 10 Raydan 2 Function (Andrei, 2008) 11 Extended Three Exponential Terms Function (Andrei, 2008) 12 Generalized Rosenbrock Function (Andrei, 2008) 13 Generalized White & Holst Function (Andrei, 2008) 14 Extended Block Diagonal BD1 Function (Andrei, 2008) 15 HimmelBG Function (Andrei, 2008) 16 Power Function (Andrei, 2008) 17 Extended Dixon and Price (Andrei, 2008) 18 Extended Booth Function (Andrei, 2008) 19 Extended Boh2 Function (Andrei, 2008) 20 Diagonal 3 Function (Jamil & Yang, 2013) 21 Hager Function (Jamil & Yang, 2013) 22 Extended Penalty Function (Andrei, 2008) 23 Extended Cliff & Roth Function (Andrei, 2008) 24 Extended Quadratic Penalty QP1 Function (Andrei, 2008) 25 Extended EP1 Function (Andrei, 2008) 26 ARWHEAD Function (Andrei, 2008) 27 Extended Freudenstein & Roth Function (Andrei, 2008) 28 Cube Function (Andrei, 2008) 29 Extended Goldstein & Price Function (Andrei, 2008) 30 Chebyquad Function (Andrei, 2008)

Discussion on Numerical Results
For better description and understanding of the Figures 1-6 above, the solvability measure considering the number of iteration and the execution time for the step-length formulae (SLF) used in this work are presented below in Table 2 based on the number of successes and failures in percentage recorded by each CGM:  Table 2 which measures the effectiveness of each CGM and the various SLFs presented in our work. The newly derived formula NSF displaced efficient performance over ESF1and ESF2 considering the number of iteration and the execution time taken in solving each of the thirty test functions using nine distinctive CGMs. It is observed that almost all the CG methods considered attained the highest level of efficiency using the newly derived formula, and it is evident to finalize that NSF showed better efficient performance displaced over ESF2 and ESF1 judged by the calculated average as presented in Table 2 above.

Conclusion
In this work, nine kinds of CG methods namely BAN, FR, PRP, HS, CD, DY, LS, HZ and GSC CG methods were employed in solving thirty unconstrained optimization test functions with two different dimensions (5000 and 10000) using the newly derived formula to obtain the step-length. To test the efficiency of the new step-length formula, it was assessed and compared with already existing formulae numerically. To better investigate our observations, performance profiling software is employed to plot the performance profiles of the different methods and afterward presented our report. It is also established that the BAN CGM and CD method exhibit better efficiency computationally when compared to the other CGMs mentioned in this research work.