Advertisement

Accelerated Randomized Coordinate Descent Algorithms for Stochastic Optimization and Online Learning

  • Akshita BhandariEmail author
  • Chandramani Singh
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11353)

Abstract

We propose accelerated randomized coordinate descent algorithms for stochastic optimization and online learning. Our algorithms have significantly less per-iteration complexity than the known accelerated gradient algorithms. The proposed algorithms for online learning have better regret performance than the known randomized online coordinate descent algorithms. Furthermore, the proposed algorithms for stochastic optimization exhibit as good convergence rates as the best known randomized coordinate descent algorithms. We also show simulation results to demonstrate performance of the proposed algorithms.

Notes

Acknowledgments

The second author acknowledges support of INSPIRE Faculty Research Grant (DSTO-1363).

References

  1. 1.
    Nesterov, Y.: Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM J. Optim. 22, 341–362 (2012)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Nemirovski, A., Juditsky, A., Lan, G., Shapiro, A.: Robust stochastic approximation approach to stochastic programming. SIAM J. Optim. 19, 1574–1609 (2009)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Zinkevich, M.: Online convex programming and generalized infinitesimal gradient ascent. In: Proceedings of the 20th International Conference on Machine Learning (ICML-03), pp. 928–936 (2003)Google Scholar
  4. 4.
    Hu, C., Pan, W., Kwok, J.: Accelerated gradient methods for stochastic optimization and online learning. Advances in Neural Information Processing Systems, pp. 781–789 (2009)Google Scholar
  5. 5.
    Le Roux, N., Schmidt, M., Bach, F.: A stochastic gradient method with an exponential convergence rate for finite training sets. In: Neural Information Processing Systems (2012)Google Scholar
  6. 6.
    Johnson, R., Zhang, T.: Accelerating stochastic gradient descent using predictive variance reduction. In: Advances in Neural Information Processing Systems, pp. 315–323 (2013)Google Scholar
  7. 7.
    Langford, J., Smola, A., Zinkevich, M.: Slow learners are fast. Adv. Neural Inf. Process. Syst. 22, 2331–2339 (2009)Google Scholar
  8. 8.
    McMahan, B., Streeter, M.: Delay-tolerant algorithms for asynchronous distributed online learning. In: Advances in Neural Information Processing Systems, pp. 2915–2923 (2014)Google Scholar
  9. 9.
    Luo, Z.-Q., Tseng, P.: On the convergence of the coordinate descent method for convex differentiable minimization. J. Optim. Theory Appl. 72, 735 (2002)MathSciNetGoogle Scholar
  10. 10.
    Tseng, P.: Convergence of a block coordinate descent method for non differentiable minimization. J. Optim. Theory Appl. 109, 475–494 (2001)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Fercoq, O., Richtarik, P.: Accelerated, parallel, and proximal coordinate descent. SIAM J. Optim. 25, 1997–2023 (2015)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Singh, C., Nedic, A., Srikant, R.: Random block-coordinate gradient projection algorithms. In: Decision and Control (CDC), pp. 185–190. IEEE (2014)Google Scholar
  13. 13.
    Allen-Zhu, Z., Qu, Z., Richtarik, P., Yuan, Y.: Even faster accelerated coordinate descent using non-uniform sampling. In: International Conference on Machine Learning, pp. 1110-1119 (2016)Google Scholar
  14. 14.
    Deng, Q., Lan, G., Rangarajan, A.: Randomized block subgradient methods for convex nonsmooth and stochastic optimization (2015)Google Scholar
  15. 15.
    Wang, H., Banerjee, A.: Randomized block coordinate descent for online and stochastic optimization (2014)Google Scholar
  16. 16.
    Hua, X., Kadomoto, S., Yamshita, N.: Regret analysis of block coordinate gradient methods for online convex programming (2015)Google Scholar
  17. 17.
    Zhao, T., Yu, M., Wang, Y., Arora, R., Liu, H.: Accelerated mini-batch randomized block coordinate descent method. In: Advances in neural information processing systems, pp. 3329–3337 (2014)Google Scholar
  18. 18.
    Zhang, A., Gu, Q.: Accelerated stochastic block coordinate descent with optimal sampling. In: KDD, pp. 2035–2044 (2016)Google Scholar
  19. 19.
    Nathan, A., Klabjan, D.: Optimization for large-scale machine learning with distributed features and observations. In: Perner, P. (ed.) MLDM 2017. LNCS (LNAI), vol. 10358, pp. 132–146. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-62416-7_10CrossRefGoogle Scholar
  20. 20.
    Konecny, J., McMahan, H., Ramage, D., Richtarik, P.: Federated optimization: distributed machine learning for on-device intelligence (2016)Google Scholar
  21. 21.
    Bhandari, A., Singh, C.: Accelerated randomized coordinate descent algorithms for stochastic optimization and online learning (2018). arXiv:1806.01600

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Department of ESEIndian Institute of Science BangaloreBangaloreIndia

Personalised recommendations