Skip to main content
SpringerLink
Log in

Greedy randomized sampling nonlinear Kaczmarz methods

  • Published:
Calcolo Aims and scope Submit manuscript

Abstract

The nonlinear Kaczmarz method was recently proposed to solve the system of nonlinear equations. In this paper, we first discuss two greedy selection rules, i.e., the maximum residual and maximum distance rules, for the nonlinear Kaczmarz iteration. Then, based on them, two kinds of greedy randomized sampling methods are presented. Furthermore, we also devise four corresponding greedy randomized block methods, i.e., the multiple samples-based methods. The linear convergence in expectation of all the proposed methods is proved. Numerical results show that, in some applications, including brown almost linear function and generalized linear model, the greedy selection rules give faster convergence rates than the existing ones, and the block methods outperform the single sample-based ones.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Algorithm 1
Algorithm 2
Algorithm 3
Algorithm 4
Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Availability of supporting data

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Notes

  1. We note that at almost the same time the NSKM method, which is the same as our MR-SNK method, was presented by Zhang et al. [39]. However, the convergence factor of the method obtained in this paper is tighter than the one given in [39].

References

  1. Björck, Å.: Numerical Methods for Least Squares Problems. SIAM, Philadelphia (1996)

    Book  Google Scholar 

  2. Chen, Q., Hao, W.: A homotopy training algorithm for fully connected neural networks. Proc. R. Soc. Lond. Ser. A Math. Phys. Eng. Sci. 475, 20190662 (2019)

  3. Dennis, J.E., Jr., Schnabel, R.B.: Numerical Methods for Unconstrained Optimization and Nonlinear Equations. SIAM, Philadelphia (1996)

    Book  Google Scholar 

  4. Hao, W., Hauenstein, J.D., Hu, B., Sommese, A.J.: A bootstrapping approach for computing multiple solutions of differential equations. J. Comput. Appl. Math. 258, 181–190 (2014)

    Article  MathSciNet  Google Scholar 

  5. Atkinson, K.E.: A survey of numerical methods for solving nonlinear integral equations. J. Integral Equations Appl. 4, 15–46 (1992)

    Article  MathSciNet  Google Scholar 

  6. Hao, W., Harlim, J.: An equation-by-equation method for solving the multidimensional moment constrained maximum entropy problem. Commun. Appl. Math. Comput. Sci. 13, 189–214 (2018)

    Article  MathSciNet  Google Scholar 

  7. Hao, W.: A homotopy method for parameter estimation of nonlinear differential equations with multiple optima. J. Sci. Comput. 74, 1314–1324 (2018)

    Article  MathSciNet  Google Scholar 

  8. Kelley, C.T.: Iterative Methods for Optimization. SIAM, Philadelphia (1999)

    Book  Google Scholar 

  9. Brewster, M.E., Kannan, R.: Nonlinear successive over-relaxation. Numer. Math. 44, 309–315 (1984)

    Article  MathSciNet  Google Scholar 

  10. Brewster, M.E., Kannan, R.: A computational process for choosing the relaxation parameter in nonlinear SOR. Computing 37, 19–29 (1986)

    Article  MathSciNet  Google Scholar 

  11. Yamashita, N., Fukushima, M.: On the rate of convergence of the Levenberg–Marquardt method. In: Topics in Numerical Analysis, pp. 239–249. Springer, Wien (2001)

  12. Dennis, J.E.J., Moré, J.J.: Quasi-Newton methods, motivation and theory. SIAM review 19, 46–89 (1977)

    Article  MathSciNet  Google Scholar 

  13. Ortega, J.M., Rheinboldt, W.C.: Iterative Solution of Nonlinear Equations in Several Variables. SIAM, Philadelphia (2000)

    Book  Google Scholar 

  14. Yuan, R., Lazaric, A., Gower, R.M.: Sketched Newton-Raphson. SIAM J. Optim. 32, 1555–1583 (2022)

    Article  MathSciNet  Google Scholar 

  15. Zeng, W., Ye, J.: Successive projection for solving systems of nonlinear equations/inequalities (2020). arXiv:2012.07555

  16. Wang, Q., Li, W., Bao, W., Gao, X.: Nonlinear Kaczmarz algorithms and their convergence. J. Comput. Appl. Math. 399, 113720 (2022)

    Article  MathSciNet  Google Scholar 

  17. Polyak, B.T.: Introduction to Optimization. Optimization Software. New York 1, 32 (1987)

    Google Scholar 

  18. Prazeres, M., Oberman, A.M.: Stochastic gradient descent with polyak’s learning rate. J. Sci. Comput. 89, 1–16 (2021)

    Article  MathSciNet  Google Scholar 

  19. Loizou, N., Vaswani, S., Laradji, I.H., Lacoste-Julien, S.: Stochastic polyak step-size for sgd: An adaptive learning rate for fast convergence. In: International Conference on Artificial Intelligence and Statistics, pp. 1306–1314 (2021). PMLR

  20. Zhang, J.H., Wang, Y.Q., Zhao, J.: On maximum residual nonlinear Kaczmarz-type algorithms for large nonlinear systems of equations. J. Computat. Appl. Math. 425, 115065 (2023)

    Article  MathSciNet  Google Scholar 

  21. Liu, L., Li, W.G., Xing, L.L., Bao, W.D.: Nonlinear greedy relaxed randomized Kaczmarz method. Res. Appl. Math. 16, 100340 (2022)

    Article  MathSciNet  Google Scholar 

  22. Gower, R.M., Lorenz, D.A., Winkler, M.: A Bregman–Kaczmarz method for nonlinear systems of equations (2023). arXiv:2303.08549

  23. Gao, Y., Chen, C.: Convergence analysis of nonlinear Kaczmarz method for systems of nonlinear equations with component–wise convex mapping (2023). arXiv:2309.15003

  24. Kaczmarz, S.: Angenäherte auflösung von systemen linearer gleichungen. Bull. Int. Acad. Pol. Sci. Lett. A 35, 355–357 (1937)

  25. Strohmer, T., Vershynin, R.: A randomized Kaczmarz algorithm with exponential convergence. J. Fourier Anal. Appl. 15, 262–278 (2009)

    Article  MathSciNet  Google Scholar 

  26. Griebel, M., Oswald, P.: Greedy and randomized versions of the multiplicative schwarz method. Linear Algebra Appl. 437, 1596–1610 (2012)

    Article  MathSciNet  Google Scholar 

  27. Nutini, J., Sepehry, B., Virani, A., Laradji, I., Schmidt, M., Koepke, H.: Convergence rates for greedy Kaczmarz algorithms. In: 32nd Conference on Uncertainty in Artificial Intelligence (2016). AUAI Press

  28. Du, K., Gao, H.: A new theoretical estimate for the convergence rate of the maximal weighted residual Kaczmarz algorithm. Numer. Math. Theor. Meth. Appl. 12, 627–639 (2019)

    Article  MathSciNet  Google Scholar 

  29. Eldar, Y., Needell, D.: Acceleration of randomized Kaczmarz method via the Johnson-Lindenstrauss lemma. Numer. Algor. 58, 163–177 (2011)

    Article  MathSciNet  Google Scholar 

  30. Bai, Z., Wu, W.: On greedy randomized Kaczmarz method for solving large sparse linear systems. SIAM J. Sci. Comput. 40, 592–606 (2018)

    Article  MathSciNet  Google Scholar 

  31. Zhang, Y., Li, H.: A count sketch maximal weighted residual Kaczmarz method for solving highly overdetermined linear systems. Appl. Math. Comput. 410, 126486 (2021)

    MathSciNet  Google Scholar 

  32. Zhang, Y., Li, H.: Greedy Motzkin-Kaczmarz methods for solving linear systems. Numer. Linear Algebra Appl. 29, 2429 (2022)

    Article  MathSciNet  Google Scholar 

  33. De Loera, J.A., Haddock, J., Needell, D.: A sampling Kaczmarz-Motzkin algorithm for linear feasibility. SIAM J. Sci. Comput. 39, 66–87 (2017)

    Article  MathSciNet  Google Scholar 

  34. Haddock, J., Ma, A.: Greed works: An improved analysis of sampling Kaczmarz-Motzkin. SIAM J. Math. Data Sci. 3, 342–368 (2021)

    Article  MathSciNet  Google Scholar 

  35. Haltmeier, M., Kowar, R., Leitão A., Scherzer, O.: Kaczmarz methods for regularizing nonlinear ill-posed equations ii: Applications. Inverse Probl Imaging 1, 507–523 (2007)

  36. Golub, G.H., Van Loan, C.F.: Matrix Computations. JHU press, Baltimore (2013)

    Book  Google Scholar 

  37. Agamon, S.: The relaxation method for linear inequalities. Canad. J. Math. 6, 382–392 (1954)

    Article  MathSciNet  Google Scholar 

  38. Motzkin, T.S., Schoenberg, I.J.: The relaxation method for linear inequalities. Canad. J. Math. 6, 393–404 (1954)

    Article  MathSciNet  Google Scholar 

  39. Wang, Q., Li, W., Bao, W.: On sampling Kaczmarz-Motzkin methods for solving large-scale nonlinear systems (2022). arXiv:2209.04195

  40. Strohmer, T., Vershynin, R.: A randomized Kaczmarz algorithm with exponential convergence. J. Fourier Anal. Appl. 15, 262–278 (2009)

    Article  MathSciNet  Google Scholar 

  41. Zhang, Y., Li, H.: Block sampling Kaczmarz-Motzkin methods for consistent linear systems. Calcolo 58, 1–20 (2021)

    Article  MathSciNet  Google Scholar 

  42. Moré, J.J., Garbow, B.S., Hillstrom, K.E.: Testing unconstrained optimization software. ACM Trans. Math. Softw. (TOMS) 7, 17–41 (1981)

    Article  MathSciNet  Google Scholar 

  43. Chang, C.C., Lin, C.J.: Libsvm: a library for support vector machines. ACM Trans. Intell. Syst. Technol. (TIST) 2, 1–27 (2011)

    Article  Google Scholar 

  44. Yuan, R., Lazaric, A., Gower, R.M.: Sketched Newton-Raphson (2020). arXiv:2006.12120

  45. Niu, Y., Zheng, B.: A greedy block Kaczmarz algorithm for solving large-scale linear systems. Appl. Math. Lett. 104, 106294 (2020)

    Article  MathSciNet  Google Scholar 

  46. Gower, R.M., Molitor, D., Moorman, J., Needell, D.: On adaptive sketch-and-project for solving linear systems. SIAM J. Matrix Anal. Appl. 42(2), 954–989 (2021)

    Article  MathSciNet  Google Scholar 

  47. Gower, R.M., Richtárik, P.: Randomized iterative methods for linear systems. SIAM J. Matrix Anal. Appl. 36, 1660–1690 (2015)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

Not Applicable

Funding

This work was supported by the National Natural Science Foundation of China (No. 11671060) and the Natural Science Foundation Project of CQ CSTC (No. cstc2019jcyj-msxmX0267)

Author information

Authors and Affiliations

  1. Department of Statistics and Data Science, Chongqing University of Technology, Chongqing, 400054, People’s Republic of China

    Yanjun Zhang

  2. College of Mathematics and Statistics, Chongqing University, Chongqing, 401331, People’s Republic of China

    Hanyu Li

  3. School of Sciences, Southwest Petroleum University, Chengdu, Sichuan, 610500, People’s Republic of China

    Ling Tang

Authors
  1. Yanjun Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hanyu Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ling Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All the authors contributed to the study conception and design, and read and approved the final manuscript.

Corresponding author

Correspondence to Yanjun Zhang.

Ethics declarations

Ethical Approval

Not Applicable

Competing Interests

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Y., Li, H. & Tang, L. Greedy randomized sampling nonlinear Kaczmarz methods. Calcolo 61, 25 (2024). https://doi.org/10.1007/s10092-024-00577-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10092-024-00577-1

Keywords

Mathematics Subject Classification (2010)

Search

Navigation