A globally convergent hybrid conjugate gradient method with strong Wolfe conditions for unconstrained optimization

In this paper, we develop a new hybrid conjugate gradient method that inherits the features of the Liu and Storey (LS), Hestenes and Stiefel (HS), Dai and Yuan (DY) and Conjugate Descent (CD) conjugate gradient methods. The new method generates a descent direction independently of any line search and possesses good convergence properties under the strong Wolfe line search conditions. Numerical results show that the proposed method is robust and efficient.


Introduction
In this paper, we consider solving the unconstrained optimization problem where x ∈ ℝ n is an n-dimensional real vector and f ∶ ℝ n → ℝ is a smooth function, using a nonlinear conjugate gradient method. Optimization problems arise naturally in problems from many scientific and operational applications (see e.g. [12, 19-22, 35, 36], among others).
To solve problem (1), a nonlinear conjugate gradient method starts with an initial guess, x 0 ∈ ℝ n , and generates a sequence {x k } ∞ k=0 using the recurrence where the step size k is a positive parameter and d k is the search direction defined by The scalar k is the conjugate gradient update coefficient and g k = ∇f (x k ) is the gradient of f at x k . In finding the step size k , the inexpensive line searches such as the weak Wolfe line search the strong Wolfe line search or the generalized Wolfe conditions where 0 < < < 1 and 1 ≥ 0 are constants, are often used. Generally, conjugate gradient methods differ by the choice of the coefficient k . Well-known formulas for k can be divided into two categories. The first category includes Fletcher and Reeves (FR) [11], Dai and Yuan (DY) [6] and Conjugate Descent (CD) [10]: where ‖ ⋅ ‖ denotes the Euclidean norm and y k−1 = g k − g k−1 .
These methods have strong convergence properties. However, since they are very often susceptible to jamming, they tend to have poor numerical performance. The other (1) min f (x), (2) x k+1 = x k + k d k , category includes Hestenes and Stiefel (HS) [16], Polak-Ribière-Polyak (PRP) [28,29] and Liu and Storey (LS) [26]: Although these methods may fail to converge, they have an in-built automatic restart feature which helps them avoid jamming and hence makes them numerically efficient [5].
In view of the above stated drawbacks and advantages, many researchers have proposed hybrid conjugate gradient methods that combine different k coefficients so as to limit the drawbacks and maximize in the advantages of the original respective conjugate gradient methods. For instance, Touati-Ahmed and Storey [31] suggested one of the first hybrid method where the coefficient k is given by The authors proved that TS k has good convergence properties and numerically outperforms both the FR k and PRP k methods. Alhawarat et al. [3] introduced a hybrid conjugate gradient method in which the conjugate gradient update coefficient is computed as where k is defined as The authors proved that the method possesses global convergence property when weak Wolfe line search is employed. Moreover, numerical results demonstrate that the proposed method outperforms both the CG-Descent 6.8 [14] and CG-Descent 5.3 [13] methods on a number of benchmark test problems.
In this paper, we suggest another new hybrid conjugate gradient method that inherits good computational efforts of LS k and HS k methods and also nice convergence properties of DY k and CD k methods. This proposed method is presented in the next section, and the rest of the paper is structured as follows. In Sect. 3, we show that the proposed method satisfies the descent condition for any line search and also present its global convergence analysis under the strong Wolfe line search. Numerical comparison with respect to performance profiles of Dolan-Morè [7] and conclusion is presented in Sects. 4 and 5, respectively.

A new hybrid conjugate gradient method
In [32], a variant of the PRP k method is proposed, where the coefficient k is computed as This method inherits the good numerical performance of the PRP method. Moreover, Huang et al. [17] proved that the WYL k method satisfies the sufficient descent property and established that the method is globally convergent under the strong Wolfe line search if the parameter in (5) (7) and (8), respectively, we present our hybrid conjugate gradient algorithm below.

Global convergence of the proposed method
The following standard assumptions which have been used extensively in the literature are necessary to analyse the global convergence properties of our hybrid method.

Assumption 3.1 The level set
is bounded, where x 0 ∈ ℝ n is the initial guess of the iterative method (2). Proof If k = 0 or �g T k g k−1 � ≥ 0.2‖g k ‖ 2 , then the search direction d k is given by This gives Otherwise, the search direction d k is given by Proof From (5) and (10), it follows that and since 0 < < 1, we have Also, by descent condition (10), we get implying Therefore, from (7), (13) and (14), it is clear that PKT Squaring both sides gives Now, dividing by (g T k d k ) 2 and applying the descent condition , we obtain Noting that and using (17)   This contradicts the Zoutendjik condition (12), concluding the proof. ◻

Numerical results
In this section, we analyse the numerical efficiency of our proposed PKT k method, herein denoted PKT, by comparing its performance to that of Jian et al. [18], herein denoted N, and that of Alhawarat et al. [3], herein denoted AZPRP, on a set of 55 unconstrained test problems selected from [4]. We stop the iterations if either ‖g k ‖ ≤ 10 −5 or a maximum of 10,000 iterations is reached. All the algorithms are coded   Table 1, where "Function" denotes name of test problem, "Dim" denotes dimension of test problem,"NI" denotes number of iterations, "FE" denotes number of function evaluations, "GE" denotes number of gradient evaluations, "CPU" denotes CPU time in seconds and "-" means that the method failed to solve the problem within 10,000 iterations. The bolded figures show the best performer for each problem. From Table 1, we observe that the PKT and AZPRP methods successfully solved all the problems, whereas the N method failed to solve one problem within 10,000 iterations. Moreover, the numerical results in the table indicate that the new PKT method is competitive as it is the best performer for a significant number of problems.
To further illustrate the performance of the three methods, we adopted the performance profile tool proposed by Dolan and Morè [7]. This tool evaluates and compares the performance of n s solvers running on a set of n p problems. The comparison between the solvers is based on the performance ratio (19) where f p,s denotes either number of functions (gradient) evaluations, number of iterations or CPU time required by solver s to solve problem p. The overall evaluation of the performance of the solvers is then given by the performance profile function where ≥ 0. If solver s fails to solve a problem p, we set the ratio r p,s to some sufficiently large number. The corresponding profiles are plotted in Figs. 1, 2, 3 and 4, where Fig. 1 shows the performance profile of number of iterations, Fig. 2 shows the performance profile of number of gradient evaluations, Fig. 3 shows the performance profile of function evaluations and Fig. 4 shows the performance profile of CPU time. The figures illustrate that the new method outperforms the AZPRP and N conjugate gradient methods.

Conclusion
In this paper, we developed a new hybrid conjugate gradient method that inherits the features of the famous Liu and Storey (LS), Hestenes and Stiefel (HS), Dai and Yuan (DY) and Conjugate Descent (CD) conjugate gradient methods. The global convergence of the proposed method was established under the strong Wolfe line search conditions. We compared the performance of our method with those of Jian et al. [18] and Alhawarat et al. [3] on a number of benchmark unconstrained optimization problems. Evaluation of performance based on the tool of Dolan-Moré [7] showed that the proposed method is both efficient and effective.
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creat iveco mmons .org/licen ses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.