Skip to main content
SpringerLink
Log in
Book cover

International Conference on Optimization and Applications

OPTIMA 2021: Optimization and Applications pp 246–257Cite as

Near-Optimal Decentralized Algorithms for Saddle Point Problems over Time-Varying Networks

Part of the Lecture Notes in Computer Science book series (LNTCS,volume 13078)

Abstract

Decentralized optimization methods have been in the focus of optimization community due to their scalability, increasing popularity of parallel algorithms and many applications. In this work, we study saddle point problems of sum type, where the summands are held by separate computational entities connected by a network. The network topology may change from time to time, which models real-world network malfunctions. We obtain lower complexity bounds for algorithms in this setup and develop near-optimal methods which meet the lower bounds.

Keywords

  • Saddle-point problem
  • Distributed optimization
  • Decentralized optimization
  • Time-varying network
  • Lower and upper bounds

The research of A. Beznosikov, A. Rogozin and A. Gasnikov was supported by Russian Science Foundation (project No. 21-71-30005).

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   64.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   84.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Abadeh, S., Esfahani, P., Kuhn, D.: Distributionally robust logistic regression. In: Advances in Neural Information Processing Systems (NeurIPS), pp. 1576–1584 (2015)

    Google Scholar 

  2. Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: Proceedings of the 34th International Conference on Machine Learning (ICML), vol. 70, no. 1, pp. 214–223 (2017)

    Google Scholar 

  3. Bertsekas, D.P., Tsitsiklis, J.N.: Parallel and Distributed Computation: Numerical Methods, vol. 23. Prentice Hall Englewood Cliffs (1989)

    Google Scholar 

  4. Beznosikov, A., Dvurechensky, P., Koloskova, A., Samokhin, V., Stich, S.U., Gasnikov, A.: Decentralized local stochastic extra-gradient for variational inequalities. arXiv preprint arXiv:2106.08315 (2021)

  5. Beznosikov, A., Samokhin, V., Gasnikov, A.: Local SGD for saddle-point problems. arXiv preprint arXiv:2010.13112 (2020)

  6. Beznosikov, A., Scutari, G., Rogozin, A., Gasnikov, A.: Distributed saddle-point problems under similarity. arXiv preprint arXiv:2107.10706 (2021)

  7. Boyd, S., Ghosh, A., Prabhakar, B., Shah, D.: Randomized gossip algorithms. IEEE Trans. Inf. Theory 52(6), 2508–2530 (2006)

    Google Scholar 

  8. Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40(1), 120–145 (2011)

    Google Scholar 

  9. Facchinei, F., Pang, J.: Finite-Dimensional Variational Inequalities and Complementarity Problems. Springer Series in Operations Research and Financial Engineering, Springer, New York (2007). https://books.google.ru/books?id=lX_7Rce3_Q0C. https://doi.org/10.1007/b97543

  10. Goodfellow, I.J., et al.: Generative adversarial networks (2014)

    Google Scholar 

  11. Jin, Y., Sidford, A.: Efficiently solving MDPs with stochastic mirror descent. In: III, H.D., Singh, A. (eds.) Proceedings of the 37th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 119, pp. 4890–4900. PMLR, 13–18 July 2020

    Google Scholar 

  12. Juditsky, A., Nemirovskii, A.S., Tauvel, C.: Solving variational inequalities with Stochastic Mirror-Prox algorithm (2008)

    Google Scholar 

  13. Kempe, D., Dobra, A., Gehrke, J.: Gossip-based computation of aggregate information. In: 44th Annual IEEE Symposium on Foundations of Computer Science, 2003. Proceedings, pp. 482–491. IEEE (2003)

    Google Scholar 

  14. Koloskova, A., Loizou, N., Boreiri, S., Jaggi, M., Stich, S.U.: A unified theory of decentralized SGD with changing topology and local updates. arXiv preprint arXiv:2003.10422 (2020)

  15. Kovalev, D., Gasanov, E., Richtárik, P., Gasnikov, A.: Lower bounds and optimal algorithms for smooth and strongly convex decentralized optimization over time-varying networks. arXiv preprint arXiv:2106.04469 (2021)

  16. Kovalev, D., Shulgin, E., Richtárik, P., Rogozin, A., Gasnikov, A.: ADOM: accelerated decentralized optimization method for time-varying networks. arXiv preprint arXiv:2102.09234 (2021)

  17. Li, H., Lin, Z.: Accelerated gradient tracking over time-varying graphs for decentralized optimization. arXiv preprint arXiv:2104.02596 (2021)

  18. Liu, M., et al.: A decentralized parallel algorithm for training generative adversarial nets. arXiv preprint arXiv:1910.12999 (2019)

  19. Liu, W., Mokhtari, A., Ozdaglar, A., Pattathil, S., Shen, Z., Zheng, N.: A decentralized proximal point-type method for saddle point problems. arXiv preprint arXiv:1910.14380 (2019)

  20. Maros, M., Jaldén, J.: PANDA: a dual linearly converging method for distributed optimization over time-varying undirected graphs. In: 2018 IEEE Conference on Decision and Control (CDC), pp. 6520–6525 (2018)

    Google Scholar 

  21. McDonald, R., Hall, K., Mann, G.: Distributed training strategies for the structured perceptron. In: Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pp. 456–464 (2010)

    Google Scholar 

  22. McMahan, B., Moore, E., Ramage, D., Hampson, S., Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Artificial Intelligence and Statistics, pp. 1273–1282. PMLR (2017)

    Google Scholar 

  23. Nedić, A., Olshevsky, A., Shi, W.: Achieving geometric convergence for distributed optimization over time-varying graphs. SIAM J. Optim. 27(4), 2597–2633 (2017)

    Google Scholar 

  24. Nedic, A., Ozdaglar, A.: Distributed subgradient methods for multi-agent optimization. IEEE Trans. Autom. Control 54(1), 48–61 (2009)

    Google Scholar 

  25. Nemirovski, A.: Prox-method with rate of convergence o(1/t) for variational inequalities with Lipschitz continuous monotone operators and smooth convex-concave saddle point problems. SIAM J. Optim. 15, 229–251 (2004). https://doi.org/10.1137/S1052623403425629

  26. von Neumann, J., Morgenstern, O., Kuhn, H.: Theory of Games and Economic Behavior (Commemorative Edition). Princeton University Press (2007)

    Google Scholar 

  27. Rogozin, A., Beznosikov, A., Dvinskikh, D., Kovalev, D., Dvurechensky, P., Gasnikov, A.: Decentralized distributed optimization for saddle point problems. arXiv preprint arXiv:2102.07758 (2021)

  28. Rogozin, A., Gasnikov, A.: Projected gradient method for decentralized optimization over time-varying networks. arXiv preprint arXiv:1911.08527 (2019)

  29. Rogozin, A., Uribe, C.A., Gasnikov, A.V., Malkovsky, N., Nedić, A.: Optimal distributed convex optimization on slowly time-varying graphs. IEEE Trans. Control Network Syst. 7(2), 829–841 (2019)

    Google Scholar 

  30. Scaman, K., Bach, F., Bubeck, S., Lee, Y.T., Massoulié, L.: Optimal algorithms for smooth and strongly convex distributed optimization in networks. arXiv preprint arXiv:1702.08704 (2017)

  31. Shalev-Shwartz, S., Ben-David, S.: Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press (2014)

    Google Scholar 

  32. Ye, H., Luo, L., Zhou, Z., Zhang, T.: Multi-consensus decentralized accelerated gradient descent. arXiv preprint arXiv:2005.00797 (2020)

Download references

Author information

Authors and Affiliations

  1. Moscow Institute of Physics and Technology, Moscow, Russia

    Aleksandr Beznosikov, Alexander Rogozin & Alexander Gasnikov

  2. Higher School of Economics, Moscow, Russia

    Aleksandr Beznosikov & Alexander Rogozin

  3. Institute for Information Transmission Problems RAS, Moscow, Russia

    Alexander Gasnikov

  4. Caucasus Mathematical Center, Adyghe State University, Maykop, Russia

    Alexander Gasnikov

  5. King Abdullah University of Science and Technology, Thuwal, Saudi Arabia

    Dmitry Kovalev

Authors
  1. Aleksandr Beznosikov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alexander Rogozin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dmitry Kovalev
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alexander Gasnikov
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Aleksandr Beznosikov .

Editor information

Editors and Affiliations

  1. Dorodnicyn Computing Centre, FRC CSC RAS, Moscow, Russia

    Nicholas N. Olenev

  2. Dorodnicyn Computing Centre, FRC CSC RAS, Moscow, Russia

    Yuri G. Evtushenko

  3. University of Montenegro, Podgorica, Montenegro

    Milojica Jaćimović

  4. Krasovsky Institute of Mathematics and Mechanics, Ekaterinburg, Russia

    Prof. Michael Khachay

  5. Dorodnicyn Computing Centre, FRC CSC RAS, Moscow, Russia

    Vlasta Malkova

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Beznosikov, A., Rogozin, A., Kovalev, D., Gasnikov, A. (2021). Near-Optimal Decentralized Algorithms for Saddle Point Problems over Time-Varying Networks. In: Olenev, N.N., Evtushenko, Y.G., Jaćimović, M., Khachay, M., Malkova, V. (eds) Optimization and Applications. OPTIMA 2021. Lecture Notes in Computer Science(), vol 13078. Springer, Cham. https://doi.org/10.1007/978-3-030-91059-4_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-91059-4_18

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-91058-7

  • Online ISBN: 978-3-030-91059-4

  • eBook Packages: Computer ScienceComputer Science (R0)

search

Navigation