Abstract
This paper focuses on the problem of convex constraint nonlinear equations involving monotone operators in Euclidean space. A Fletcher and Reeves type derivative-free conjugate gradient method is proposed. The proposed method is designed to ensure the descent property of the search direction at each iteration. Furthermore, the convergence of the proposed method is proved under the assumption that the underlying operator is monotone and Lipschitz continuous. The numerical results show that the method is efficient for the given test problems.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
1 Introduction
In this paper, we deal with the following problem:
where the mapping \(\vartheta \) is from the Euclidean space \({\mathbb {R}}^n\) onto itself. Also, \(\vartheta \) is monotone and continuous and the set B is a nonempty closed and convex subset of \({\mathbb {R}}^n\). The problem (1) is called a convex constraint nonlinear equation and we denote the solution set of it by \(Sol (B,\vartheta )\).
The convex constraint nonlinear problem is motivated by several applications in various fields such as financial forecasting problems [9], learning constrained neural networks [8], economic equilibrium problems [14], nonlinear compressed sensing [6], chemical equilibrium systems [27], phase retrieval [7], power flow equations [33], and non-negative matrix factorisation [4, 24]. The fast local superlinear convergence property of the Newton method, quasi-Newton method, Levenberge–Marquardt method, as well as a variety of their variants [10,11,12, 30] have made them appealing for solving (1). However, to apply these methods, a linear system of equations must be computed using a Jacobian matrix or its approximation. To overcome this drawback, many authors have suggested derivative-free methods that does not require or approximate the Jacobian matrix [18]. Our focus in this present research will be on derivative-free methods based on one of the first-order optimization methods. The method of interest is the conjugate gradient (CG) method which is well known for solving large-scale unconstrained optimization problems due to its simplicity and low storage.
In recent years, motivated by the projection scheme proposed by Solodov and Svaiter [31], several derivative-free methods have been proposed for solving (1). For example, Liu and Feng [25] introduced a derivative-free projection method which converges to the solution of the convex constraint problem (1). The proposed scheme involves only one projection per iteration with a monotone and Lipschitz continuity assumption imposed on the underlying mapping. Their proposed method can be viewed as a modification of the well-known Dai–Yuan CG method for unconstrained optimization. Also, Ibrahim et al. [19] proposed a derivative-free projection method for solving the nonlinear equation (1). The method combines the projection technique and the LS–FR CG method proposed by Djordjević [15]. At each iteration, the proposed method does not store any matrix. With the aid of the projection scheme, several other derivative-free methods have been developed. Interested readers can refer to [1,2,3, 20,21,22, 28] as well as references therein.
In this present work, based on the Fletcher and Reeves (FR) CG method for unconstrained optimization, we introduced a derivative-free type iterative method for solving the constraint problem (1). The proposed method is designed to ensure the descent property of the search direction at each iteration. Furthermore, the convergence of the proposed method is proved under the assumption that the underlying operator is monotone and Lipschitz continuous. The numerical results show that the method is efficient for the given test problems.
Our paper is organized as follows: in Sect. 2, we review some definitions which are used in the sequel. Also, the proposed search direction is presented and its global convergence result is established. Section 3 is devoted for the numerical experiment where numerical results are reported for several examples. In the last section, conclusions and discussions are given.
Notation. Throughout this article except stated otherwise, \(\Vert \cdot \Vert \) stands for Euclidean norm on \({\mathbb {R}}^n\). In addition, for a nonempty closed and convex set \(B\in {\mathbb {R}}^n\), \(P_B[\cdot ]\) is the projection mapping from \({\mathbb {R}}^n\) onto B given by \(P_{B}[w]=\arg \min \{\Vert w-y\Vert \ : y\in B\}.\)
2 Algorithm and convergence analysis
Let \(\vartheta : {\mathbb {R}}^n \rightarrow {\mathbb {R}}^n\) be a monotone and continuous nonlinear function, and let B be a nonempty closed and convex subset of \({\mathbb {R}}^n\). Recall that \(\vartheta \) is said to be:
-
1.
monotone if:
$$\begin{aligned} \forall y_1,y_2\in {\mathbb {R}}^n, ~~(\vartheta (y_1)-\vartheta (y_2))^T(y_1-y_2) \ge 0. \end{aligned}$$(2) -
2.
L-Lipschitz continuous, with \(L>0\) if:
$$\begin{aligned} \forall y_1,y_2\in {\mathbb {R}}^n, ~~\Vert \vartheta (y_1)-\vartheta (y_2)\Vert \le L\Vert y_1-y_2\Vert . \end{aligned}$$(3)
Next, we propose an algorithm based on a modified FR CG method. The modification is done on the FR CG parameter and by extension, the direction. Generally, a CG method for solving (1) generates a sequence of iterates from an initial guess \(w_0\) via the formula:
where \(\alpha _k\) is the stepsize computed using a suitable line search procedure and \(d_k\) is the search direction defined as:
\(\beta _k\) is called the CG parameter and \(\vartheta (w_k)\) is the function evaluation of \(\vartheta \) at \(w_k\). One of the properties required to establish the convergence of an algorithm for finding approximate solution to (1) is the sufficient descent property of the direction. We say that a direction is sufficiently descent if:
In this paper, to solve the problem (1), we consider the following FR-type direction given by :
where:
It can be observed that as \(\Vert \vartheta (w_{k-1})\Vert \rightarrow 0\), the FR parameter may fail to be defined. To maintain well-defineness of the parameter, we replace \(\Vert \vartheta (w_{k-1})\Vert \) with \(\mu \Vert \vartheta (w_{k})\Vert \Vert d_{k-1}\Vert +\Vert \vartheta (w_{k-1})\Vert ^2\) and define a modified FR parameter \( \beta _k^{MFR}\) as:
To make the direction (6) with the parameter defined in (8) descent, we introduce a new term to the direction as follows:
where:
Remark 2.1
Note that the term \(-\varrho _k \vartheta (w_k)\) was specifically introduced, so that the direction defined by (9) satisfies the sufficient descent condition (see Lemma 2.5).
In what follows, we give a systematic description of the proposed algorithm for finding approximate solutions to problem (1).
In what follows, we show that the sequence generated by the algorithm converges. In this case, we will require the following conditions and results from the Lemmas.
Condition 2.2
The constraint set \(B \subseteq {\mathbb {R}}^n\) is nonempty, closed, and convex set.
Condition 2.3
The mapping \(\vartheta \) is monotone and L-Lipschitz continuous on \({\mathbb {R}}^n\).
Condition 2.4
The solution set of the convex constraint problem is nonempty, that is \(Sol(B, \vartheta ) \ne \emptyset \).
Lemma 2.5
The search direction defined by (8)–(10) satisfies the sufficient descent condition.
Proof
For \(k=0\), we have:
For \(k\ge 1\), then by (9) and (10):
\(\square \)
Lemma 2.6
Let Conditions 2.2, 2.3, and 2.4 be fulfilled. If \(\{\varphi _k\}\) and \(\{w_k\}\) are given by (11) and (13) in Algorithm 1, then:
Proof
If \(\alpha _k\ne \varpi \), then \(\alpha _k^{'}=\frac{\alpha _k}{\rho }\) does not satisfy (12), that is:
Now, from Condition 2.3, (5), and (15):
Therefore:
\(\square \)
Lemma 2.7
Let Condition 2.2, 2.3, and 2.4 be fulfilled, then the sequences \(\{\varphi _k\}\) and \(\{w_k\}\) defined by (11) and (13) are bounded. Furthermore:
and
Proof
Let \({\tilde{w}}\in Sol(B,\vartheta )\), and then, by Condition 2.3:
Likewise from (12) and \(t>0\):
Now:
We can conclude from (20) that:
-
the sequence \(\{\Vert w_k - {\tilde{w}}\Vert \}\) is convergent.
-
for all k:
$$\begin{aligned} \Vert w_{k}- {\tilde{w}}\Vert ^2 \le \Vert w_0 - {\tilde{w}}\Vert ^2. \end{aligned}$$(21) -
for all k, \(\{w_k\}\) is bounded. That is, there is a \(c_1>0\), such that:
$$\begin{aligned} \Vert w_k\Vert \le c_1. \end{aligned}$$(22) -
From Condition 2.3, (21) and letting \(\ell _1=L\Vert w_{0}- {\tilde{w}}\Vert :\)
$$\begin{aligned} \Vert \vartheta (w_k)\Vert =\Vert \vartheta (w_k)- \vartheta ({\tilde{w}})\Vert \le L\Vert w_{k}- {{\tilde{w}}}\Vert \le L\Vert w_{0}- {\tilde{w}}\Vert =\ell _1. \end{aligned}$$(23)
Now, by monotonicity of \(\vartheta \), we have that:
From (11), (19), (24), and the Cauchy-Schwartz inequality:
Thus, we have:
From (25) and the reverse triangle inequality:
Combining the above inequality with (22) and (23):
Hence, \(\{\varphi _k\}\) is bounded and so is \(\{\varphi _k-{\tilde{w}}\}\). That is, there is an \(\epsilon _1>0\), such that:
By Condition 2.3:
Also, by (20):
and
Inequality (26) implies that:
However, using (13) and the Cauchy–Schwartz inequality:
Therefore:
\(\square \)
Lemma 2.8
Let the search direction \(d_k\) be defined by (9)–(10), and then, for all \(k\ge 0:\)
Proof
It is obvious that for \(k=0\),
Now, for \(k\ge 1,\) from (9), (10), (22), and (23):
Letting \(N_1:= \ell _1 + 2\frac{\ell _1}{\mu }\), (27) is obtained. \(\square \)
Theorem 2.9
Let Condition 2.2, 2.3, and 2.4 be fulfilled. If \(\{w_k\}\) is a sequence defined by (13), then:
Proof
Suppose that \(\displaystyle \liminf _{k\rightarrow \infty }\Vert \vartheta (w_k)\Vert \ne 0,\) and then, we can find a positive constant \(\nu _1\) for which \(\Vert \vartheta (w_k)\Vert \ge \nu _1.\) By the sufficient descent condition (5) and the Cauchy–Schwartz inequality:
Combining (16) and (30), we have:
However, by (14):
which contradicts (31). Therefore: \(\displaystyle \liminf _{k\rightarrow \infty }\Vert \vartheta (w_k)\Vert =0\). \(\square \)
3 Numerical experiments on monotone operator equations
In this section, we give some numerical illustrations of Algorithm 1 called modified FR derivative-free (MFRDF) method by solving nonlinear monotone operator equations. We consider numerical comparison of MFRDF method with MPCGM method proposed in [32] and Algorithm 2.1 proposed in [17]. For the numerical illustrations, we use the proposed method and the compared methods to solve the test problems given in Table 1. For the control parameters, we choose \(\varpi = 1, \eta = 1.8\), \(\rho = 0.8\) \(\mu =1.3\), and \(t=10^{-4}\) for the MFRDF Algorithm, and for the compared methods, we set the same values of parameters as it appears in their respective papers. Additionally, for each algorithm, we take the stopping criteria to be \(\Vert \vartheta (w_k) \Vert \le 10^{-5}.\)
In the numerical experiments, we consider six different initial points \(x_1=(0.1, 0.1, \dots , 0.1)^T, x_2=(0.2, 0.2, \dots , 0.2)^T, x_3=(0.5, 0.5, \dots , 0.5)^T, x_4=(1.2, 1.2, \dots , 1.2)^T, x_5=(1.5, 1.5, \dots 1.5)^T, x_6=(2, 2, \dots , 2)^T\) and five dimensions ranging from 1000 to 100, 000. Therefore, we performed 300 cases for the experiments. The results of the experiments in terms of number of functions evaluations, CPU elapsed time, and number of iterations can be seen at https://documentcloud.adobe.com/link/review?uri=urn:aaid:scds:US:c6829e5f-3f54-46d2-ab38-e10a2a1dd2f7.
It can be seen from the results in the table of the compared algorithms that the proposed MFRDF method performs better than the compared methods in terms of number of iterations, elapsed CPU time, and number of function evaluations. Moreover, to visualize the extent of performance of the proposed method in comparison with the MPCGM and Algorithm 2.1, we adopt the performance profiles from [16]. It can be seen, respectively, from Figs. 1, 2, and 3 that MFRDF has more than \(75 \%\) performance in terms of number of iterations, about \(72 \%\) in terms of CPU time, and \(55 \%\) success in terms of number of function evaluations.
4 Conclusion
This paper proposed an FR-like derivative-free algorithm combined with the projection technique for solving nonlinear monotone operator equations. The proposed search direction is bounded and satisfies the sufficient descent condition. Useful assumptions were considered to establish the global convergence. The results obtained from the numerical experiments illustrate the strength and efficiency of the proposed algorithm in contrast with the existing algorithms.
References
Abubakar, A.B.; Ibrahim, A.H.; Muhammad, A.B.; Tammer, C.: A modified descent dai-yuan conjugate gradient method for constraint nonlinear monotone operator equations. Appl. Anal. Optim. 4(1), 1–24 (2020)
Abubakar, A.B.; Kumam, P.; Mohammad, H.: A note on the spectral gradient projection method for nonlinear monotone equations with applications. Comput. Appl. Math. 39, 129 (2020)
Abubakar, A.B.; Rilwan, J.; Yimer, S.E.; Ibrahim, A.H.; Ahmed, I.: Spectral three-term conjugate descent method for solving nonlinear monotone equations with convex constraints. Thai J. Math. 18(1), 501–517 (2020)
Berry, M.W.; Browne, M.; Langville, A.N.; Paul Pauca, V.; Plemmons, R.J.: Algorithms and applications for approximate nonnegative matrix factorization. Comput. Stat. Data Anal. 52(1), 155–173 (2007)
Bing, Y.; Lin, G.: An efficient implementation of merrill’s method for sparse or partially separable systems of nonlinear equations. SIAM J. Optim. 1(2), 206–221 (1991)
Blumensath, T.: Compressed sensing with nonlinear observations and related nonlinear optimization problems. IEEE Trans. Inf. Theory 59(6), 3466–3474 (2013)
Candes, E.J.; Li, X.; Soltanolkotabi, M.: Phase retrieval via wirtinger flow: theory and algorithms. IEEE Trans. Inf. Theory 61(4), 1985–2007 (2015)
Chorowski, J.; Zurada, J.M.: Learning understandable neural networks with nonnegative weight constraints. IEEE Trans. Neural Netw. Learn. Syst. 26(1), 62–69 (2014)
Dai, Z., Dong, X., Kang, J., Hong, L.: Forecasting stock market returns: New technical indicators and two-step economic constraint method. N. Am. J. Econ. Financ. page 101216 (2020)
Dennis, J.E.; Moré, J.J.: A characterization of superlinear convergence and its application to quasi-newton methods. Math. Comput. 28(126), 549–560 (1974)
Dennis Jr., J.E.; Moré, J.J.: Quasi-newton methods, motivation and theory. SIAM Rev. 19(1), 46–89 (1977)
Dennis, J.E., Jr., Schnabel, R.: Numerical methods for unconstrained optimization and nonlinear equations (1983)
Ding, Y.; Xiao, Y.H.; Li, J.: A class of conjugate gradient methods for convex constrained monotone equations. Optimization 66(12), 2309–2328 (2017)
Dirkse, S.P.; Ferris, M.C.: Mcplib: A collection of nonlinear mixed complementarity problems. Optim. Methods Softw. 5(4), 319–345 (1995)
Djordjević, S.S.: New hybrid conjugate gradient method as a convex combination of ls and fr methods. Acta Math. Sci. 39(1), 214–228 (2019)
Dolan, E.D.; Moré, J.J.: Benchmarking optimization software with performance profiles. Math. Program. 91(2), 201–213 (2002)
Gao, P.; He, C.: An efficient three-term conjugate gradient method for nonlinear monotone equations with convex constraints. Calcolo 55(4), 53 (2018)
Huang, N.; Ma, C.; Xie, Y.: The derivative-free double newton step methods for solving system of nonlinear equations. Mediterr. J. Math. 13(4), 2253–2270 (2016)
Ibrahim, A.H.; Kumam, P.; Abubakar, A.B.; Jirakitpuwapat, W.; Abubakar, J.: A hybrid conjugate gradient algorithm for constrained monotone equations with application in compressive sensing. Heliyon 6(3), e03466 (2020)
Ibrahim, A.H.; Kumam, P.; Abubakar, A.B.; Yusuf, U.B.; Rilwan, J.: Derivative-free conjugate residual algorithms for convex constraints nonlinear monotone equations and signal recovery. J. Nonlinear Convex Anal. 21(9), 1959–1972 (2020)
Ibrahim, A.H.; Kumam, P.; Abubakar, A.B.; Yusuf, U.B.; Yimer, S.E.; Aremu, K.O.: An efficient gradient-free projection algorithm for constrained nonlinear equations and image restoration. AIMS Math. 6(1), 235 (2020)
Ibrahim, A.H., Kumam, P., Kumam, W.: A family of derivative-free conjugate gradient methods for constrained nonlinear equations and image restoration. IEEE Access 8 (2020)
La Cruz, W.; Martínez, J.; Raydan, M.: Spectral residual method without gradient information for solving large-scale nonlinear systems of equations. Math. Comput. 75(255), 1429–1448 (2006)
Lee, D.D., Sebastian Seung, H.: Algorithms for non-negative matrix factorization. In Advances in neural information processing systems, pp. 556–562 (2001)
Liu, J.K., Feng, Y.: A derivative-free iterative method for nonlinear monotone equations with convex constraints. Numer. Algorithms, pp. 1–18 (2018)
Lukšan, Ladislav, Vlcek, Jan: Test problems for unconstrained optimization. Academy of Sciences of the Czech Republic, Institute of Computer Science, Technical Report, 897, (2003)
Meintjes, K.; Morgan, A.P.: A methodology for solving chemical equilibrium systems. Appl. Math. Comput. 22(4), 333–361 (1987)
Mohammad, H.; Abubakar, A.B.: A descent derivative-free algorithm for nonlinear monotone equations with convex constraints. RAIRO Oper. Res. 54(2), 489–505 (2020)
Moré, J.J.; Garbow, B.S.; Hillstrom, K.E.: Testing unconstrained optimization software. ACM Trans. Math. Softw. (TOMS) 7(1), 17–41 (1981)
Qi, L.; Sun, J.: A nonsmooth version of newton’s method. Math. Program. 58(1–3), 353–367 (1993)
Solodov, M.V., Svaiter, B.F: A globally convergent inexact newton method for systems of monotone equations. In: Reformulation: Nonsmooth, Piecewise Smooth, Semismooth and Smoothing Methods, pages 355–369. Springer, New York (1998)
Sun, M., Liu, J., Wang, Y.: Two improved conjugate gradient methods with application in compressive sensing and motion control. Math. Probl. Eng. (2020)
Wood, A.J.; Wollenberg, B.F.; Sheblé, G.B.: Power generation, operation, and control. Wiley, Amsterdam (2013)
Zhensheng, Yu; Lin, J.; Sun, J.; Xiao, Y.; Liu, L.; Li, Z.: Spectral gradient projection method for monotone nonlinear equations with convex constraints. Appl. Numer. Math. 59(10), 2416–2423 (2009)
Zhou, W.; Li, D.H.: Limited memory BFGS method for nonlinear monotone equations. J. Comput. Math. 25(1), 89–96 (2007)
Acknowledgements
The first author acknowledges with thanks, the Department of Mathematics and Applied Mathematics at the Sefako Makgatho Health Sciences University. The second author was financially supported by Rajamangala University of Technology Phra Nakhon (RMUTP) Research Scholarship.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Abubakar, A.B., Muangchoo, K., Ibrahim, A.H. et al. FR-type algorithm for finding approximate solutions to nonlinear monotone operator equations . Arab. J. Math. 10, 261–270 (2021). https://doi.org/10.1007/s40065-021-00313-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40065-021-00313-5