Skip to main content
SpringerLink
Log in

On FISTA with a relative error rule

  • Published:
Computational Optimization and Applications Aims and scope Submit manuscript

Abstract

The fast iterative shrinkage/thresholding algorithm (FISTA) is one of the most popular first-order iterations for minimizing the sum of two convex functions. FISTA is known to improve the complexity of the classical proximal gradient method (PGM) from \(O(k^{-1})\) to the optimal complexity \(O(k^{-2})\) in terms of the sequence of the functional values. When the evaluation of the proximal operator is hard, inexact versions of FISTA might be used to solve the problem. In this paper, we proposed an inexact version of FISTA by solving the proximal subproblem inexactly using a relative error criterion instead of exogenous and diminishing error rules. The introduced relative error rule in the FISTA iteration is related to the progress of the algorithm at each step and does not increase the computational burden per iteration. Moreover, the proposed algorithm recovers the same optimal convergence rate as FISTA. Some numerical experiments are also reported to illustrate the numerical behavior of the relative inexact method when compared with FISTA under an absolute error criterion.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data Availibility Statement

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. Adona, V.A., Gonçalves, M.L.N., Melo, J.G.: A partially inexact proximal alternating direction method of multipliers and its iteration-complexity analysis. J. Optim. Theory Appl. 182(2), 640–666 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  2. Adona, V.A., Gonçalves, M.L.N., Melo, J.G.: An inexact proximal generalized alternating direction method of multipliers. Comput. Optim. Appl. 76(3), 621–647 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  3. Alves, M.M., Eckstein, J., Geremia, M., Melo, J.G.: Relative-error inertial-relaxed inexact versions of Douglas-Rachford and ADMM splitting algorithms. Comput. Optim. Appl. 75(2), 389–422 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  4. Anderson, E., Bai, Z., Bischof, C., Blackford, S., Demmel, J., Dongarra, J., Du Croz, J., Greenbaum, A., Hammarling, S., McKenney, A., Sorensen, D.: LAPACK Users’ Guide, 3rd edn. Society for Industrial and Applied Mathematics, Philadelphia, PA (1999)

    Book  MATH  Google Scholar 

  5. Attouch, H., Cabot, A.: Convergence rates of inertial forward-backward algorithms. SIAM J. Optim. 28(1), 849–874 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  6. Attouch, H., Cabot, A., Chbani, Z., Riahi, H.: Inertial forward-backward algorithms with perturbations: application to Tikhonov regularization. J. Optim. Theory Appl. 179(1), 1–36 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  7. Attouch, H., Chbani, Z., Peypouquet, J., Redont, P.: Fast convergence of inertial dynamics and algorithms with asymptotic vanishing viscosity. Math. Program. 168(1–2), 123–175 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  8. Attouch, H., Peypouquet, J.: The rate of convergence of Nesterov’s accelerated forward-backward method is actually faster than \(1/k^2\). SIAM J. Optim. 26(3), 1824–1834 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  9. Aujol, J.F., Dossal, C.: Stability of over-relaxations for the forward-backward algorithm, application to FISTA. SIAM J. Optim. 25(4), 2408–2433 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  10. Bauschke, H.H., Bui, M., Wang, X.: Applying FISTA to optimization problems (with or) without minimizers. Math. Program. 192, 1–20 (2019)

    MATH  Google Scholar 

  11. Bauschke, H.H., Combettes, P.L.: Convex analysis and monotone operator theory in Hilbert Spaces, 2nd edn. CMS Books in Mathematics. Springer International Publishing, Cham (2017)

    Book  MATH  Google Scholar 

  12. Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imag. Sci. 2(1), 183–202 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  13. Beck, A., Teboulle, M.: Gradient-based algorithms with applications to signal-recovery problems. In: Convex optimization in signal processing and communications, pp. 42–88. Cambridge Univ. Press, Cambridge (2010)

  14. Bello Cruz, J.Y.: On proximal subgradient splitting method for minimizing the sum of two nonsmooth convex functions. Set-Valued Var. Anal. 25(2), 245–263 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  15. Bello Cruz, J.Y., Nghia, T.A.: On the convergence of the forward-backward splitting method with linesearches. Optim. Methods Softw. 31(6), 1209–1238 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  16. Bezanson, J., Edelman, A., Karpinski, S., Shah, V.: Julia: A fresh approach to numerical computing. SIAM Review 59(1), 65–98 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  17. Borsdorf, R., Higham, N.J.: A preconditioned Newton algorithm for the nearest correlation matrix. IMA J. Numer. Anal. 30, 94–107 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  18. Chambolle, A., Dossal, C.: On the convergence of the iterates of the “fast iterative shrinkage/thresholding algorithm’’. J. Optim. Theory Appl. 166(3), 968–982 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  19. Dolan, E.D., Moré, J.J.: Benchmarking optimization software with performance profiles. Math. Program. 91, 201–213 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  20. Eckstein, J., Silva, P.J.S.: A practical relative error criterion for augmented Lagrangians. Math. Programming 141(1), 319–348 (2013). https://doi.org/10.1007/s10107-012-0528-9

    Article  MathSciNet  MATH  Google Scholar 

  21. Eckstein, J., Yao, W.: Approximate ADMM algorithms derived from Lagrangian splitting. Comput. Optim. Appl. 68(2), 363–405 (2017). https://doi.org/10.1007/s10589-017-9911-z

    Article  MathSciNet  MATH  Google Scholar 

  22. Eckstein, J., Yao, W.: Relative-error approximate versions of Douglas-Rachford splitting and special cases of the ADMM. Math. Programming 170(2), 417–444 (2018). https://doi.org/10.1007/s10107-017-1160-5

    Article  MathSciNet  MATH  Google Scholar 

  23. Gould, N., Scott, J.: A note on performance profiles for benchmarking software. ACM Trans. Math. Softw. 43(2), 1–5 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  24. Hale, E.T., Yin, W., Zhang, Y.: Fixed-point continuation for \(l_1\)-minimization: methodology and convergence. SIAM J. Optim. 19(3), 1107–1130 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  25. Jiang, K., Sun, D., Toh, K.C.: An inexact accelerated proximal gradient method for large scale linearly constrained convex SDP. SIAM J. Optim. 22(3), 1042–1064 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  26. Lewandowski, D., Kurowicka, D., Joe, H.: Generating random correlation matrices based on vines and extended onion method. J. Multivar. Anal. 100(9), 1989–2001 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  27. Millán, R.D., Machado, M.P.: Inexact proximal \(\epsilon \)-subgradient methods for composite convex optimization problems. J. Global Optim. 75(4), 1029–1060 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  28. Monteiro, R.D.C., Svaiter, B.F.: On the complexity of the hybrid proximal extragradient method for the iterates and the ergodic mean. SIAM J. Optim. 20(6), 2755–2787 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  29. Monteiro, R.D.C., Svaiter, B.F.: An accelerated hybrid proximal extragradient method for convex optimization and its implications to second-order methods SIAM. J. Optim. 23(2), 1092–1125 (2013)

    MathSciNet  MATH  Google Scholar 

  30. Morales, J.L., Nocedal, J.: Remark on “Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound constrained optimization’’. ACM Trans. Math. Softw. 38(1), 1–4 (2011)

    Article  MATH  Google Scholar 

  31. Moré, J.J., Wild, S.M.: Benchmarking derivative-free optimization algorithms. SIAM J. Optim. 20(1), 172–191 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  32. Nesterov, Y.: A method for solving the convex programming problem with convergence rate \(O(1/k^{2})\). Dokl. Akad. Nauk SSSR 269(3), 543–547 (1983)

    MathSciNet  Google Scholar 

  33. Nesterov, Y.: An approach to constructing optimal methods for minimization of smooth convex functions. Èkonom. i Mat. Metody 24(3), 509–517 (1988)

    MathSciNet  MATH  Google Scholar 

  34. Nesterov, Y.: Smooth minimization of non-smooth functions. Math. Program. 103(1), 127–152 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  35. Nesterov, Y.: Gradient methods for minimizing composite functions. Math. Program. 140(1), 125–161 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  36. Pastell, M.: Weave.jl: Scientific reports using Julia. J. Open Source Softw. 2(11), 204 (2017). https://doi.org/10.21105/joss.00204

    Article  Google Scholar 

  37. Qi, H., Sun, D.: A quadratically convergent Newton method for computing the nearest correlation matrix. SIAM J. Matrix Anal. Appl. 28(2), 360–385 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  38. Qi, H., Sun, D., Gao, Y.: CorNewton3.m: A Matlab code for computing the nearest correlation matrix with fixed diagonal and off diagonal elements. https://www.polyu.edu.hk/ama/profile/dfsun/CorNewton3.m (2009)

  39. Salzo, S., Villa, S.: Inexact and accelerated proximal point algorithms. J. Convex Anal. 19(4), 1167–1192 (2012)

    MathSciNet  MATH  Google Scholar 

  40. Schmidt, M., Roux, N.L., Bach, F.R.: Convergence rates of inexact proximal-gradient methods for convex optimization. In: Advances in Neural Information Processing Systems 24, pp. 1458–1466. Curran Associates, Inc. (2011)

  41. Solodov, M.V., Svaiter, B.F.: A hybrid approximate extragradient-proximal point algorithm using the enlargement of a maximal monotone operator. Set-Valued Anal. 7(4), 323–345 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  42. Su, W., Boyd, S., Candès, E.J.: A differential equation for modeling Nesterov’s accelerated gradient method: theory and insights. J. Mach. Learn. Res. 17, 153 (2016)

    MathSciNet  MATH  Google Scholar 

  43. Tropp, J.A.: Just relax: convex programming methods for identifying sparse signals in noise. IEEE Trans. Inform. Theory 52(3), 1030–1051 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  44. Villa, S., Salzo, S., Baldassarre, L., Verri, A.: Accelerated and inexact forward-backward algorithms. SIAM J. Optim. 23(3), 1607–1633 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  45. Zhu, C., Byrd, R.H., Lu, P., Nocedal, J.: Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization. ACM Trans. Math. Softw. 23(4), 550–560 (1997)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

YBC was partially supported by the National Science Foundation (NSF), Grant DMS – 1816449 and by internal funds at NIU. MLNG was partially supported by the Brazilian Agency Conselho Nacional de Pesquisa (CNPq), Grants 304133/2021-3 and 405349/2021-1. The authors would like to thank the two anonymous referees for their valuable suggestions which improved this manuscript.

Author information

Authors and Affiliations

  1. Department of Mathematical Sciences, Northern Illinois University, DeKalb, IL, 60115, USA

    Yunier Bello-Cruz & Nathan Krislock

  2. IME, Universidade Federal de Goiás, Goiânia, GO, 74001-970, Brazil

    Max L. N. Gonçalves

Authors
  1. Yunier Bello-Cruz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Max L. N. Gonçalves
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nathan Krislock
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yunier Bello-Cruz.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (TGZ 151427 kb)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bello-Cruz, Y., Gonçalves, M.L.N. & Krislock, N. On FISTA with a relative error rule. Comput Optim Appl 84, 295–318 (2023). https://doi.org/10.1007/s10589-022-00421-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10589-022-00421-8

Keywords

Mathematics Subject Classification

Search

Navigation