Skip to main content
SpringerLink
Log in

Two efficient nonlinear conjugate gradient methods with restart procedures and their applications in image restoration

  • Original Paper
  • Published:
Nonlinear Dynamics Aims and scope Submit manuscript

Abstract

Nonlinear conjugate gradient method (CGM) is one of the most efficient iterative methods for dealing with large-scale optimization problems. In this paper, based on the Fletcher–Reeves and Dai–Yuan CGMs, two restart CGMs with different restart procedures are proposed for unconstrained optimization, in which their restart conditions are designed according to their conjugate parameters with the aim of ensuring that their search directions are sufficient descent. Under usual assumptions and using the weak Wolfe line search to yield their steplengths, the proposed methods are proved to be global convergent. To test the validity of the proposed methods, we choose four restart directions for each method and perform large-scale numerical experiments for unconstrained optimization and image restoration problems. Moreover, we report their detailed numerical results and performance profiles, which show that the encouraging efficiency and applicability of the proposed methods even compared with the current well-accepted methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability

The datasets generated during the current study are available from the corresponding author on reasonable request.

Notes

  1. Their codes can be downloaded at https://github.com/jhyin-optim/FHTTCGMs_with_applications.

References

  1. Chen, F.Y., Ding, F., Li, J.H.: Maximum likelihood gradient-based iterative estimation algorithm for a class of input nonlinear controlled autoregressive ARMA systems. Nonlinear Dyn. 79, 927–936 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  2. Balaram, B., Narayanan, M.D., Rajendrakumar, P.K.: Optimal design of multi-parametric nonlinear systems using a parametric continuation based genetic algorithm approach. Nonlinear Dyn. 67, 2759–2777 (2012)

    Article  MathSciNet  Google Scholar 

  3. Zhang, L.M., Gao, H.T., Chen, Z.Q., et al.: Multi-objective global optimal parafoil homing trajectory optimization via Gauss pseudospectral method. Nonlinear Dyn. 72, 1–8 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  4. Hestenes, M.R., Stiefel, E.: Method of conjugate gradient for solving linear systems. J. Res. Natl. Bur. Stand. 49(6), 409–436 (1952)

    Article  MathSciNet  MATH  Google Scholar 

  5. Fletcher, R., Reeves, C.M.: Function minimization by conjugate gradients. Comput. J. 7(2), 149–154 (1964)

    Article  MathSciNet  MATH  Google Scholar 

  6. Polak, E., Ribiere, G.: Note sur la convergence de methods de directions conjugées. Rev. Fr. Informat Rech. Operationelle 3e Anneè. 16(3), 35–43 (1969)

    MATH  Google Scholar 

  7. Polyak, B.T.: The conjugate gradient method in extreme problems. USSR Comput. Math. Math. Phys. 9(4), 94–112 (1969)

    Article  MATH  Google Scholar 

  8. Dai, Y.H., Yuan, Y.X.: A nonlinear conjugate gradient method with a strong global convergence property. SIAM J. Optim. 10(1), 177–182 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  9. Hager, W.W., Zhang, H.C.: A new conjugate gradient method with guaranteed descent and an efficient line search. SIAM J. Optim. 16(1), 170–192 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  10. Hager, W.W., Zhang, H.C.: A survey of nonlinear conjugate gradient methods. Pac. J. Optim. 2(1), 35–58 (2006)

    MathSciNet  MATH  Google Scholar 

  11. Dai, Y.H., Kou, C.X.: A nonlinear conjugate gradient algorithm with an optimal property and an improved Wolfe line search. SIAM J. Optim. 23(1), 296–320 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  12. Crowder, H., Wolfe, P.: Linear convergence of the conjugate gradient method. IBM J. Res. Dev. 16(4), 431–433 (1972)

    Article  MathSciNet  MATH  Google Scholar 

  13. Powell, M.J.D.: Some convergence properties of the conjugate gradient method. Math. Program. 11, 42–49 (1976)

    Article  MathSciNet  MATH  Google Scholar 

  14. Andrei, N.: Open problems in nonlinear conjugate gradient algorithms for unconstrained optimization. Bull. Malays. Math. Sci. Soc. 34(2), 319–330 (2011)

    MathSciNet  MATH  Google Scholar 

  15. Powell, M.J.D.: Restart procedures for the conjugate gradient method. Math. Program. 12, 241–254 (1977)

    Article  MathSciNet  MATH  Google Scholar 

  16. Xiao, Y.H., Hu, Q.J., Wei, Z.X.: Modified active set projected spectral gradient method for bound constrained optimization. Appl. Math. Modell. 35(7), 3117–3127 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  17. Cheng, W.Y., Li, D.H.: An active set modified Polak–Ribiére–Polyak method for large-scale nonlinear bound constrained optimization. J. Optim. Theory Appl. 155, 1084–1094 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  18. Xiao, Y.H., Zhu, H.: A conjugate gradient method to solve convex constrained monotone equations with applications in compressive sensing. J. Math. Anal. Appl. 405(1), 310–319 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  19. Sun, M., Liu, J.: Three derivative-free projection methods for nonlinear equations with convex constraints. J. Appl. Math. Comput. 47, 265–276 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  20. Burke, J.V., Lewis, A.S., Overton, M.L.: A robust gradient sampling algorithm for nonsmooth, nonconvex optimization. SIAM J. Optim. 15(3), 751–779 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  21. Li, Q.: Conjugate gradient type methods for the nondifferentiable convex minimization. Optim. Lett. 7(3), 533–545 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  22. Liu, J.K., Du, S.Q., Chen, Y.Y.: A sufficient descent nonlinear conjugate gradient method for solving M-tensor equations. J. Comput. Appl. Math. 371, 112709 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  23. Ziadi, R., Ellaia, R., Bencherif-Madani, A.: Global optimization through a stochastic perturbation of the Polak–Ribière conjugate gradient method. J. Comput. Appl. Math. 317, 672–684 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  24. Zhu, X.J.: A Riemannian conjugate gradient method for optimization on the Stiefel manifold. Comput. Optim. Appl. 67, 73–110 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  25. Liu, J.K., Du, X.L.: A gradient projection method for the sparse signal reconstruction in compressive sensing. Appl. Anal. 97(12), 2122–2131 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  26. Chen, X.J., Zhou, W.J.: Smoothing nonlinear conjugate gradient method for image restoration using nonsmooth nonconvex minimization. SIAM J. Imaging Sci. 3(4), 765–790 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  27. Yin, J.H., Jian, J.B., Jiang, X.Z., Liu, M.X., Wang, L.Z.: A hybrid three-term conjugate gradient projection method for constrained nonlinear monotone equations with applications. Numer. Algorithms. 88(1), 389–418 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  28. Liu, Y.F., Zhu, Z.B., Zhang, B.X.: Two sufficient descent three-term conjugate gradient methods for unconstrained optimization problems with applications in compressive sensing. J. Appl. Math. Comput. 68, 1787–1816 (2022)

    Article  MathSciNet  MATH  Google Scholar 

  29. Liu, P.J., Shao, H., Wang, Y., Wu, X.Y.: A three-term CGPM-based algorithm without Lipschitz continuity for constrained nonlinear monotone equations with applications. Appl. Numer. Math. 175, 98–107 (2022)

    Article  MathSciNet  MATH  Google Scholar 

  30. Birgin, E.G., Martínez, J.M.: A spectral conjugate gradient method for unconstrained optimization. Appl. Math. Optim. 43, 117–128 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  31. Beale, E.M.L.: A derivative of conjugate gradients. Numer. Methods Nonlinear Optim. 39–43 (1972)

  32. McGuire, M.F., Wolfe, P.: Evaluating a restart procedure for conjugate gradients. IBM Thomas J, Watson Research Division (1973)

    Google Scholar 

  33. Kou, C.X., Dai, Y.H.: A modified self-scaling memoryless Broyden–Fletcher–Goldfarb–Shanno method for unconstrained optimization. J. Optim. Theory Appl. 165, 209–224 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  34. Jiang, X.Z., Jian, J.B., Song, D., et al.: An improved Polak–Ribiére–Polyak conjugate gradient method with an efficient restart direction. Comput. Appl. Math. 40, 1–24 (2021)

    Article  MATH  Google Scholar 

  35. Zoutendijk, G.: Nonlinear Programming Computational Methods, in J Abadie(ED), pp. 37–86. Integer and Nonlinear Programming, North-Holland, Amsterdam (1970)

    Google Scholar 

  36. Jiang, X.Z., Liao, W., Yin, J.H., Jian, J.B.: A new family of hybrid three-term conjugate gradient methods with applications in image restoration. Numer. Algor. 91, 161–191 (2022)

    Article  MathSciNet  MATH  Google Scholar 

  37. Andrei, N.: Hybrid conjugate gradient algorithm for unconstrained optimization. J. Optim. Theory Appl. 141, 249–264 (2009)

  38. Gould, N.I.M., Orban, D., Toint, P.L.: CUTEr and SifDec: a constrained and unconstrained testing environment, revisited. ACM Trans. Math. Softw. (TOMS) 29(4), 373–394 (2003)

    Article  MATH  Google Scholar 

  39. Andrei, N.: An unconstrained optimization test functions collection. Adv. Model. Optim. 10(1), 147–161 (2008)

    MathSciNet  MATH  Google Scholar 

  40. Moré, J.J., Garbow, B.S., Hillstrome, K.E.: Testing unconstrained optimization software. ACM Trans. Math. Softw. 7(1), 17–41 (1981)

    Article  MathSciNet  MATH  Google Scholar 

  41. Dolan, E.D., Moré, J.J.: Benchmarking optimization software with performance profiles. Math. Program. 91, 201–213 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  42. Chan, R.H., Ho, C.W., Nikolova, M.: Salt-and-pepper noise removal by median-type noise detectors and detail-preserving regularization. IEEE Trans. Image Process. 14(10), 1479–1485 (2005)

    Article  Google Scholar 

  43. Cai, J.F., Chan, R., Morini, B.: Minimization of an edge-preserving regularization functional by conjugate gradient type methods. Image Processing Based on Partial Differential Equations, pp. 109–122. Springer, Berlin, Heidelberg (2007)

    Google Scholar 

  44. Hwang, H., Haddad, R.A.: Adaptive median filters: new algorithms and results. IEEE Trans. Image Process. 4(4), 499–502 (1995)

    Article  Google Scholar 

  45. Bovik, A.C.: Handbook of Image and Video Processing, 2nd edn. Academic Press, San Diego (2010)

    MATH  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Grant No. 12171106), the Natural Science Foundation of Guangxi Province (Grant Nos. 2020GXNSFDA238017, 2016GXNSFAA380028), and the Research Project of Guangxi Minzu University (Grant No. 2018KJQD02).

Author information

Authors and Affiliations

  1. Center for Applied Mathematics of Guangxi, Guangxi Key Laboratory of Hybrid Computation and IC Design Analysis, College of Mathematics and Physics, Guangxi Minzu University, Nanning, 530006, Guangxi, People’s Republic of China

    Xian-Zhen Jiang & Jin-Bao Jian

  2. College of Mathematics and Physics, Guangxi Minzu University, Nanning, 530006, Guangxi, People’s Republic of China

    Yi-Han Zhu

Authors
  1. Xian-Zhen Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yi-Han Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jin-Bao Jian
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jin-Bao Jian.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jiang, XZ., Zhu, YH. & Jian, JB. Two efficient nonlinear conjugate gradient methods with restart procedures and their applications in image restoration. Nonlinear Dyn 111, 5469–5498 (2023). https://doi.org/10.1007/s11071-022-08013-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11071-022-08013-1

Keywords

Search

Navigation