Iterative Sampled Methods for Massive and Separable Nonlinear Inverse Problems

  • Julianne ChungEmail author
  • Matthias Chung
  • J. Tanner Slagel
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11603)


In this paper, we consider iterative methods based on sampling for computing solutions to separable nonlinear inverse problems where the entire dataset cannot be accessed or is not available all-at-once. In such scenarios (e.g., when massive amounts of data exceed memory capabilities or when data is being streamed), solving inverse problems, especially nonlinear ones, can be very challenging. We focus on separable nonlinear problems, where the objective function is nonlinear in one (typically small) set of parameters and linear in another (larger) set of parameters. For the linear problem, we describe a limited-memory sampled Tikhonov method, and for the nonlinear problem, we describe an approach to integrate the limited-memory sampled Tikhonov method within a nonlinear optimization framework. The proposed method is computationally efficient in that it only uses available data at any iteration to update both sets of parameters. Numerical experiments applied to massive super-resolution image reconstruction problems show the power of these methods.


Tikhonov regularization Sampled methods Variable projection Kaczmarz methods Super-resolution Medical imaging and other applications 



We gratefully acknowledge support by the National Science Foundation under grants NSF DMS 1723005 and NSF DMS 1654175.


  1. 1.
    Andersen, M.S., Hansen, P.C.: Generalized row-action methods for tomographic imaging. Numer. Algorithms 67(1), 121–144 (2014)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Berisha, S., Nagy, J.G., Plemmons, R.J.: Estimation of atmospheric PSF parameters for hyperspectral imaging. Numer. Linear Algebra Appl. (2015)Google Scholar
  3. 3.
    Björck, A.: Numerical Methods for Least Squares Problems. SIAM (1996)Google Scholar
  4. 4.
    Chung, J., Chung, M., Slagel, J.T., Tenorio, L.: Stochastic Newton and quasi-Newton methods for large linear least-squares problems. arXiv preprint arXiv:1702.07367 (2017)
  5. 5.
    Chung, J., Haber, E., Nagy, J.G.: Numerical methods for coupled super-resolution. Inverse Prob. 22, 1261–1272 (2006)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Chung, J., Nagy, J.G.: An efficient iterative approach for large-scale separable nonlinear inverse problems. SIAM J. Sci. Comput. 31(6), 4654–4674 (2010)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Cornelio, A., Piccolomini, E.L., Nagy, J.G.: Constrained variable projection method for blind deconvolution. J. Phys. Conf. Ser. 386, 012005 (2012)CrossRefGoogle Scholar
  8. 8.
    Escalante, R., Raydan, M.: Alternating Projection Methods, vol. 8. SIAM (2011)Google Scholar
  9. 9.
    Golub, G., Pereyra, V.: Separable nonlinear least squares: the variable projection method and its applications. Inverse Prob. 19, R1–R26 (2003)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Gower, R.M., Richtárik, P.: Randomized iterative methods for linear systems. SIAM J. Matrix Anal. Appl. 36(4), 1660–1690 (2015)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Hansen, P.C.: Discrete Inverse Problems: Insight and Algorithms. SIAM (2010)Google Scholar
  12. 12.
    Herring, J., Nagy, J., Ruthotto, L.: LAP: a linearize and project method for solving inverse problems with coupled variables. Sampling Theor. Sign. Image Process. 17(2), 127–151 (2018)MathSciNetGoogle Scholar
  13. 13.
    Kaczmarz, S.: Angenäherte Auflösung linearer Gleichungssysteme. Bulletin International de l’Académie Polonaise des Sciences et des Lettres. Classe des Sciences Mathématiques et Naturelles. Série A, Sciences Mathématiques, pp. 355–357 (1937)Google Scholar
  14. 14.
    Marchesini, S., et al.: SHARP: a distributed GPU-based ptychographic solver. J. Appl. Crystallogr. 49(4), 1245–1252 (2016)CrossRefGoogle Scholar
  15. 15.
    NASA: Images from NASA webpage. Accessed 10 Jan 2019
  16. 16.
    Needell, D., Tropp, J.A.: Paved with good intentions: analysis of a randomized block Kaczmarz method. Linear Algebra Appl. 441, 199–221 (2014)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Needell, D., Zhao, R., Zouzias, A.: Randomized block Kaczmarz method with projection for solving least squares. Linear Algebra Appl. 484, 322–343 (2015)MathSciNetCrossRefGoogle Scholar
  18. 18.
    Nocedal, J., Wright, S.J.: Numerical Optimization, 2nd edn. Springer, New York (2006). Scholar
  19. 19.
    O’Leary, D.P., Rust, B.W.: Variable projection for nonlinear least squares problems. Comput. Optim. Appl. 54(3), 579–593 (2013)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Paige, C.C., Saunders, M.A.: LSQR: an algorithm for sparse linear equations and sparse least squares. ACM Trans. Math. Softw. 8(1), 43–71 (1982)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Parkinson, D.Y., et al.: Machine learning for micro-tomography. In: Developments in X-Ray Tomography XI, vol. 10391, p. 103910J. International Society for Optics and Photonics (2017)Google Scholar
  22. 22.
    Slagel, J.T., Chung, J., Chung, M., Kozak, D., Tenorio, L.: Sampled Tikhonov regularization for large linear inverse problems. In: Inverse Problems (2019, to appear)Google Scholar
  23. 23.
    Strohmer, T., Vershynin, R.: A randomized Kaczmarz algorithm with exponential convergence. J. Fourier Anal. Appl. 15(2), 262–278 (2009)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Zouzias, A., Freris, N.M.: Randomized extended Kaczmarz for solving least squares. SIAM J. Matrix Anal. Appl. 34(2), 773–793 (2013)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Julianne Chung
    • 1
    Email author
  • Matthias Chung
    • 1
  • J. Tanner Slagel
    • 1
  1. 1.Department of MathematicsVirginia TechBlacksburgUSA

Personalised recommendations