Skip to main content
SpringerLink
Log in

On pseudoinverse-free block maximum residual nonlinear Kaczmarz method for solving large-scale nonlinear system of equations

  • Original Paper
  • Published:
Japan Journal of Industrial and Applied Mathematics Aims and scope Submit manuscript

Abstract

Recently, motivated by works arising from large-scale linear systems, some stochastic and deterministic nonlinear Kaczmarz (NK) methods have been developed to solve large-scale nonlinear system of equations. In this paper, based on row sample with an approximate maximum residual control criterion, we propose a pseudoinverse-free block maximum residual nonlinear Kaczmarz (FBMRNK) method for solving large-scale nonlinear system of equations. Then we show that FBMRNK is a variant of the sketched Newton–Raphson (SNR) method with an adaptive sketching matrix. Furthermore, using this connection we establish the global convergence theory of FBMRNK with \(\mu\)-strongly quasi-convexity or star-convexity. Finally, numerical results are provided to demonstrate superior performance of FBMRNK to the stochastic or deterministic nonlinear Kaczmarz-type methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Algorithm 1
Algorithm 2
Algorithm 3
Algorithm 4
Algorithm 5
Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data availability

All data generated or analysed during this study are included in this article.

References

  1. Cands, E.J., Li, X., Soltanolkotabi, M.: Phase retrieval via Wirtinger flow: theory and algorithms. IEEE Trans. Inf. Theory. 61, 1985–2007 (2015)

    MathSciNet  Google Scholar 

  2. Ortega, J.M., Rheinboldt, W.C.: Iterative solution of nonlinear equations in several variables. In: Classics Appl. Math., vol. 30. SIAM, Philadelphia (2000)

    Google Scholar 

  3. Torres, G.L., Quintana, V.H.: Optimal power flow by a nonlinear complementarity method. IEEE Trans. Power Syst. 15, 1028–1033 (2000)

    Google Scholar 

  4. Boyd, S., Boyd, S.P., Vandenberghe, L.: Convex Optimization. Cambridge University Press, Cambridge (2004)

    Google Scholar 

  5. Wright, S., Nocedal, J.: Numerical Optimization, 2nd edn. Springer Science and Business Media, Berlin (2006)

    Google Scholar 

  6. Kawaguchi, K.: Deep learning without poor local minima. Adv. Neural Inf. Process. Syst. 29, 586–594 (2016)

    Google Scholar 

  7. Brewster, M.E., Kannan, R.: Nonlinear successive over-relaxation. Numer. Math. 44, 309–315 (1984)

    MathSciNet  Google Scholar 

  8. Brewster, M.E., Kannan, R.: A computational process for choosing the relaxation parameter in nonlinear SOR. Computing 37, 19–29 (1986)

    MathSciNet  Google Scholar 

  9. Ecker, A., Gross, D.: A system of simultaneous non-linear equations in three-thousand variables. J. Comput. Phys. 64, 246–252 (1986)

    MathSciNet  Google Scholar 

  10. Agarwal, N., Bullins, B., Hazan, E.: Second-order stochastic optimization for machine learning in linear time. J. Mach Learn Res. 18, 1–40 (2017)

    MathSciNet  Google Scholar 

  11. Bollapragada, R., Byrd, R.H., Nocedal, J.: Exact and inexact subsampled Newton methods for optimization. IMA J. Numer Anal. 39, 545–578 (2018)

    MathSciNet  Google Scholar 

  12. Gower, R., Koralev, D., Lieder, F., Richtárik, P.: RSN: randomized subspace Newton. arXiv preprint arXiv:1905.10874 (2019)

  13. Khorasani, F.R., Mahoney, M.W.: Sub-sampled Newton methods I: globally convergent algorithms. arXiv preprint arXiv:1601.04737 (2016)

  14. Kovalev, D., Mishchenko, K., Richtárik, P.: Stochastic Newton and cubic Newton methods with simple local linear-quadratic rates. arXiv preprint arxiv:1912.01597 (2019)

  15. Dai, Y.-H.: Convergence properties of the BFGS algorithm. SIAM J. Optim. 13, 693–701 (2002)

    MathSciNet  Google Scholar 

  16. Byrd, R.H., Hansen, S.L., Nocedal, J., Singer, Y.: A stochastic quasi-Newton method for large-scale optimization. SIAM J. Optim. 26, 1008–1031 (2016)

    MathSciNet  Google Scholar 

  17. Schraudolph, N.N., Yu, J., Gunter, S.: A stochastic quasi-Newton method for online convex optimization. In: International Conference on Artificial Intelligence and Statistics, Society for Artificial Intelligence and Statistics. pp 436–443 (2007)

  18. Kantorovitch, L.: The method of successive approximation for functional equations. Acta Math. 71, 63–97 (1939)

    MathSciNet  Google Scholar 

  19. Deuflhard, P.: Newton methods for nonlinear problems: affine invariance and adaptive algorithms. In: Springer Series in Computational Mathematics, vol. 35. Springer, Berlin (2004)

    Google Scholar 

  20. Yuan, R., Lazaric, A., Gower, R.M.: Sketched Newton–Raphson. SIAM J. Optim. 32, 1555–1583 (2022)

    MathSciNet  Google Scholar 

  21. Gower, R.M., Richtárik, P.: Randomized iterative methods for linear systems. SIAM J. Matrix Anal. Appl. 36, 1660–1690 (2015)

    MathSciNet  Google Scholar 

  22. Wang, Q.-F., Li, W.-G., Bao, W.-D.: Nonlinear Kaczmarz algorithms and their convergence. J. Comput. Appl. Math. 399, 113720 (2022)

    MathSciNet  Google Scholar 

  23. Jin, B.T., Zhou, Z.-H., Zou, J.: On the convergence of stochastic gradient descent for nonlinear ill-posed problems. SIAM J. Optim. 30, 1421–1450 (2020)

    MathSciNet  Google Scholar 

  24. Kaczmarz, S.: Angenaherte auflosung von systemen linearer glei-chungen. Bull. Intern. Acad. Pol. Sic. Lett., Cl. Sci. Math. Nat. 355–357 (1937)

  25. Strohmer, T., Vershynin, R.: A randomized Kaczmarz algorithm with exponential convergence. J. Fourier Anal. Appl. 15, 262–278 (2009)

    MathSciNet  Google Scholar 

  26. Eldar, Y.C., Needell, D.: Acceleration of randomized Kaczmarz method via the Johnson–Lindenstrauss lemma. Numer. Algorithms 58, 163–177 (2011)

    MathSciNet  Google Scholar 

  27. Leventhal, D., Lewis, A.S.: Randomized methods for linear constraints: convergence rates and conditioning. Math. Oper. Res. 35, 641–654 (2010)

    MathSciNet  Google Scholar 

  28. Liu, J., Wright, S.: An accelerated randomized Kaczmarz algorithm. Math. Comput. 85, 153–178 (2016)

    MathSciNet  Google Scholar 

  29. Ma, A., Needell, D., Ramdas, A.: Convergence properties of the randomized extended Gauss–Seidel and Kaczmarz methods. SIAM J. Matrix Anal. Appl. 36, 1590–1604 (2015)

    MathSciNet  Google Scholar 

  30. Needell, D.: Randomized Kaczmarz solver for noisy linear systems. BIT 50, 395–403 (2010)

    MathSciNet  Google Scholar 

  31. Elfving, T.: Block-iterative methods for consistent and inconsistent linear equations. Numer. Math. 35, 1–12 (1980)

    MathSciNet  Google Scholar 

  32. Eggermont, G.T., Herman, P.P.B., Lent, A.: Iterative algorithms for large partitioned linear systems. Linear Algebra Appl. 40, 37–67 (1981)

    MathSciNet  Google Scholar 

  33. Needell, D., Tropp, J.-A.: Paved with good intentions: analysis of a randomized block Kaczmarz method. Linear Algebra Appl. 441, 199–221 (2014)

    MathSciNet  Google Scholar 

  34. Niu, Y.-Q., Zheng, B.: A greedy block Kaczmarz algorithm for solving large-scale linear systems. Appl. Math. Lett. 104, 106294 (2020)

    MathSciNet  Google Scholar 

  35. Moorman, J.D., Tu, T.K., Molitor, D., Needell, D.: Randomized Kaczmarz with averaging. BIT 61, 337–359 (2021)

    MathSciNet  Google Scholar 

  36. Necoara, I.: Faster randomized block Kaczmarz algorithm. SIAM J. Matrix Anal. Appl. 40, 1425–1452 (2019)

    MathSciNet  Google Scholar 

  37. Du, K., Si, W.-T., Sun, X.-H.: Randomized extended average block Kaczmarz for solving least squares. SIAM J. Sci. Comput. 42, A3541–A3559 (2020)

    MathSciNet  Google Scholar 

  38. Du, K., Sun, X.-H.: A doubly stochastic block Gauss-Seidel algorithm for solving linear equations. Appl. Math. Comput. 408, 126373 (2021)

    MathSciNet  Google Scholar 

  39. Chen, J.-Q., Huang, Z.-D.: On a fast deterministic block Kaczmarz method for solving large-scale linear systems. Numer. Algorithms 89, 1007–1029 (2022)

    MathSciNet  Google Scholar 

  40. Motzkin, T.S., Schoenberg, I.J.: The relaxation method for linear inequalities. Can. J. Math. 6, 393–404 (1954)

    MathSciNet  Google Scholar 

  41. Bai, Z.-Z., Wu, W.-T.: On greedy randomized Kaczmarz method for solving large sparse linear systems. SIAM J. Sci. Comput. 40, A592–A606 (2018)

    MathSciNet  Google Scholar 

  42. Nutini, J.: Greed is good: greedy optimization methods for large-scale structured problems. PhD thesis, University of British Columbia (2018)

  43. De Loera, J.A., Haddock, J., Needell, D.: A sampling Kaczmarz–Motzkin algorithm for linear feasibility. SIAM J. Sci. Comput. 39, S66–S87 (2017)

    MathSciNet  Google Scholar 

  44. Morshed, M.S., Islam, M.S., Noor-E-Alam, M.: Accelerated sampling Kaczmarz Motzkin algorithm for the linear feasibility problem. J. Glob. Optim. 77, 361–382 (2020)

    MathSciNet  Google Scholar 

  45. Morshed, M.S., Islam, M.S., Noor-E-Alam, M.: Sampling Kaczmarz Motzkin method for linear feasibility problems: generalization and acceleration. Math. Progr. 194, 719–779 (2022)

    MathSciNet  Google Scholar 

  46. Liu, Y., Gu, C.-Q.: On greedy randomized block Kaczmarz method for consistent linear systems. Linear Algebra Appl. 616, 178–200 (2021)

    MathSciNet  Google Scholar 

  47. Zhang, Y.-J., Li, H.-Y.: Block sampling Kaczmarz–Motzkin methods for consistent linear systems. Calcolo 58, 39 (2021)

    MathSciNet  Google Scholar 

  48. Kelley, C.T.: Solving Nonlinear Equations with Newton’s Method. SIAM, Philadelphia (2003)

    Google Scholar 

  49. More, J.J., Garbow, B.S., Hillstrom, K.E.: Testing unconstrained optimization software. ACM Trans. Math. Softw. 7, 17–41 (1981)

    MathSciNet  Google Scholar 

  50. Gomes-Ruggiero, M.A., Martänez, J.M., Moretti, A.C.: Comparing algorithms for solving sparse nonlinear systems of equations. SIAM J. Sci. Stat. Comput. 13, 459–483 (1992)

    MathSciNet  Google Scholar 

  51. Lukšan, L.: Inexact trust region method for large sparse nonlinear least squares. Kybern. Praha 29, 305–324 (1993)

    MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Science, East China University of Technology, Nanchang, 330013, Jiangxi, China

    Jianhua Zhang, Yuqing Wang & Jing Zhao

Authors
  1. Jianhua Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuqing Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jing Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jianhua Zhang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The work is supported by the National Natural Science Foundation of China under Grant no. 12061009. The work is supported by Jiangxi Provincial Natural Science Foundation under Grant no. 20202BAB201002.

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, J., Wang, Y. & Zhao, J. On pseudoinverse-free block maximum residual nonlinear Kaczmarz method for solving large-scale nonlinear system of equations. Japan J. Indust. Appl. Math. 41, 637–657 (2024). https://doi.org/10.1007/s13160-023-00620-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13160-023-00620-8

Keywords

Mathematics Subject Classification

Search

Navigation