Skip to main content
SpringerLink
Log in

Convergence and Applications of ADMM on the Multi-convex Problems

  • Conference paper
  • First Online:
Advances in Knowledge Discovery and Data Mining (PAKDD 2022)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13281))

Included in the following conference series:

Abstract

In recent years, although the Alternating Direction Method of Multipliers (ADMM) has been empirically applied widely to many multi-convex applications, delivering an impressive performance in areas such as nonnegative matrix factorization and sparse dictionary learning, there remains a dearth of generic work on proposed ADMM with a convergence guarantee under mild conditions. In this paper, we propose a generic ADMM framework with multiple coupled variables in both objective and constraints. Convergence to a Nash point is proven with a sublinear convergence rate o(1/k). Two important applications are discussed as special cases under our proposed ADMM framework. Extensive experiments on ten real-world datasets demonstrate the proposed framework’s effectiveness, scalability, and convergence properties. We have released our code at https://github.com/xianggebenben/miADMM.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The supplementary materials are available at https://github.com/xianggebenben/miADMM/blob/main/multi_convex_ADMM-13-18.pdf.

References

  1. Ando, R.K., Zhang, T.: A framework for learning predictive structures from multiple tasks and unlabeled data. J. Mach. Learn. Res. 6, 1817–1853 (2005)

    MathSciNet  MATH  Google Scholar 

  2. Argyriou, A., Evgeniou, T., Pontil, M.: Multi-task feature learning. In: Advances in Neural Information Processing Systems, pp. 41–48 (2007)

    Google Scholar 

  3. Bai, J., Li, J., Xu, F., Zhang, H.: Generalized symmetric ADMM for separable convex optimization. Comput. Optim. Appl. 70(1), 129–170 (2017). https://doi.org/10.1007/s10589-017-9971-0

    Article  MathSciNet  MATH  Google Scholar 

  4. Boţ, R.I., Nguyen, D.-K.: The proximal alternating direction method of multipliers in the nonconvex setting: convergence analysis and rates. Math. Oper. Res. 45(2), 682–712 (2020)

    Google Scholar 

  5. Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations Trends® Mach. Learn. 3(1), 1–122 (2011)

    Google Scholar 

  6. Carrington, P.J., Scott, J., Wasserman, S.: Models and Methods in Social Network Analysis, vol. 28. Cambridge University Press, Cambridge (2005)

    Google Scholar 

  7. Cavalletti, F., Rajala, T.: Tangent lines and Lipschitz differentiability spaces. Anal. Geom. Metr. Spaces 4(1), 85–103 (2016)

    MathSciNet  MATH  Google Scholar 

  8. Chao, M.T., Zhang, Y., Jian, J.B.: An inertial proximal alternating direction method of multipliers for nonconvex optimization. Int. J. Comput. Math. 98(6), 1199–1217 (2021)

    Article  MathSciNet  Google Scholar 

  9. Chartrand, R., Wohlberg, B.: A nonconvex ADMM algorithm for group sparsity with sparse groups. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6009–6013 (2013)

    Google Scholar 

  10. Chen, J., Zhou, J., Ye, J.: Integrating low-rank and group-sparse structures for robust multi-task learning. In: Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 42–50 (2011)

    Google Scholar 

  11. Deng, W., Lai, M.-J., Peng, Z., Yin, W.: Parallel multi-block ADMM with o (1/k) convergence. J. Sci. Comput. 71(2), 712–736 (2017)

    Article  MathSciNet  Google Scholar 

  12. Gao, W., Goldfarb, D., Curtis, F.E.: ADMM for multiaffine constrained optimization. Optim. Methods Softw. 35(2), 257–303 (2020)

    Article  MathSciNet  Google Scholar 

  13. Goldstein, T., O’Donoghue, B., Setzer, S., Baraniuk, R.: Fast alternating direction optimization methods. SIAM J. Imag. Sci. 7(3), 1588–1623 (2014)

    Article  MathSciNet  Google Scholar 

  14. Gu, Y., Jiang, B., Han, D.: A semi-proximal-based strictly contractive Peaceman-Rachford splitting method. arXiv preprint arXiv:1506.02221, pp. 1–20 (2015)

  15. Hassibi, A., How, J., Boyd, S.: A path-following method for solving BMI problems in control. In: Proceedings of the 1999 American Control Conference, vol. 2, pp. 1385–1389 (1999)

    Google Scholar 

  16. He, B., Yuan, X.: On the o(1/n) convergence rate of the Douglas-Rachford alternating direction method. SIAM J. Numer. Anal. 50(2), 700–709 (2012)

    Article  MathSciNet  Google Scholar 

  17. Kadkhodaie, M., Christakopoulou, K., Sanjabi, M., Banerjee, A.: Accelerated alternating direction method of multipliers. In: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 497–506 (2015)

    Google Scholar 

  18. Lee, D.D., Seung, H.S.: Algorithms for non-negative matrix factorization. In: Advances in Neural Information Processing Systems, pp. 556–562 (2001)

    Google Scholar 

  19. Li, G., Pong, T.K.: Global convergence of splitting methods for nonconvex composite optimization. SIAM J. Optim. 25(4), 2434–2460 (2015)

    Google Scholar 

  20. Li, Y., Tian, X., Liu, T., Tao, D.: Multi-task model and feature joint learning. In: International Joint Conference on Artificial Intelligence, pp. 3643–3649 (2015)

    Google Scholar 

  21. Lin, T.-Y., Ma, S.-Q., Zhang, S.-Z.: On the sublinear convergence rate of multi-block ADMM. J. Oper. Res. Soc. China 3(3), 251–274 (2015)

    Article  MathSciNet  Google Scholar 

  22. Ouyang, H., He, N., Tran, L., Gray, A.: Stochastic alternating direction method of multipliers. In: International Conference on Machine Learning, pp. 80–88 (2013)

    Google Scholar 

  23. Reed, T.: Open source indicators project (2017). https://doi.org/10.7910/DVN/EN8FUW

  24. Shen, X., Diamond, S., Udell, M., Gu, Y., Boyd, S.: Disciplined multi-convex programming. In: 2017 29th Chinese Control and Decision Conference (CCDC), pp. 895–900 (2017)

    Google Scholar 

  25. Tao, M., Yuan, X.: Convergence analysis of the direct extension of ADMM for multiple-block separable convex minimization. Adv. Comput. Math. 44(3), 773–813 (2017). https://doi.org/10.1007/s10444-017-9560-x

    Article  MathSciNet  MATH  Google Scholar 

  26. Tseng, P.: Dual coordinate ascent methods for non-strictly convex minimization. Math. Program. 59(1–3), 231–247 (1993)

    Article  MathSciNet  Google Scholar 

  27. Tseng, P.: Convergence of a block coordinate descent method for nondifferentiable minimization. J. Optim. Theory Appl. 109(3), 475–494 (2001)

    Article  MathSciNet  Google Scholar 

  28. Wang, H., Banerjee, A.: Bregman alternating direction method of multipliers. In: Advances in Neural Information Processing Systems, pp. 2816–2824 (2014)

    Google Scholar 

  29. Wang, J., Chai, Z., Cheng, Y., Zhao, L.: Toward model parallelism for deep neural network based on gradient-free ADMM framework. In: 2020 IEEE International Conference on Data Mining (ICDM), pp. 591–600 (2020)

    Google Scholar 

  30. Wang, J., Gao, Y., Züfle, A., Yang, J., Zhao, L.: Incomplete label uncertainty estimation for petition victory prediction with dynamic features. In: 2018 IEEE International Conference on Data Mining (ICDM), pp. 537–546 (2018)

    Google Scholar 

  31. Wang, J., Li, H., Chai, Z., Wang, Y., Cheng, Y., Zhao, L.: Towards quantized model parallelism for graph-augmented MLPS based on gradient-free ADMM framework. arXiv preprint arXiv:2105.09837 (2021)

  32. Wang, J., Yu, F., Chen, X., Zhao, L.: ADMM for efficient deep learning with global convergence. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 111–119 (2019)

    Google Scholar 

  33. Wang, J., Zhao, L.: Nonconvex generalization of alternating direction method of multipliers for nonlinear equality constrained problems. In: Results in Control and Optimization, p. 100009 (2019)

    Google Scholar 

  34. Wang, L., Li, Y., Zhou, J., Zhu, D., Ye, J.: Multi-task survival analysis. In: 2017 IEEE International Conference on Data Mining (ICDM), pp. 485–494 (2017)

    Google Scholar 

  35. Wang, Yu., Yin, W., Zeng, J.: Global convergence of ADMM in nonconvex nonsmooth optimization. J. Sci. Comput. 78(1), 29–63 (2019)

    Article  MathSciNet  Google Scholar 

  36. Warga, J.: Minimizing certain convex functions. J. Soc. Ind. Appl. Math. 11(3), 588–593 (1963)

    Article  MathSciNet  Google Scholar 

  37. Yangyang, X., Yin, W.: A block coordinate descent method for regularized multiconvex optimization with applications to nonnegative tensor factorization and completion. SIAM J. Imag. Sci. 6(3), 1758–1789 (2013)

    Article  MathSciNet  Google Scholar 

  38. Xu, Z., De, S., Figueiredo, M., Studer, C., Goldstein, T.: An empirical study of ADMM for nonconvex problems. In: NIPS 2016 Workshop on Nonconvex Optimization for Machine Learning: Theory and Practice (2016)

    Google Scholar 

  39. Zhao, L., Wang, J., Guo, X.: Distant-supervision of heterogeneous multitask learning for social event forecasting with multilingual indicators. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32(1), April 2018

    Google Scholar 

  40. Zhou, J., Chen, J., Ye, J.: Clustered multi-task learning via alternating structure optimization. In: Advances in Neural Information Processing Systems, pp. 702–710 (2011)

    Google Scholar 

  41. Zhou, J., Chen, J., Ye, J.: MALSAR: multi-task learning via structural regularization. Arizona State University, vol. 21 (2011)

    Google Scholar 

Download references

Acknowledgement

This work was supported by the National Science Foundation (NSF) Grant No. 1755850, No. 1841520, No. 2007716, No. 2007976, No. 1942594, No. 1907805, a Jeffress Memorial Trust Award, Amazon Research Award, NVIDIA GPU Grant, and Design Knowledge Company (subcontract No: 10827.002.120.04).

Author information

Authors and Affiliations

  1. Emory University, 201 Dowman Drive, Atlanta, GA, USA

    Junxiang Wang & Liang Zhao

Authors
  1. Junxiang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Liang Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Junxiang Wang .

Editor information

Editors and Affiliations

  1. Laboratory of Artificial Intelligence and Decision Support, University of Porto, Porto, Portugal

    João Gama

  2. School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu, China

    Tianrui Li

  3. National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China

    Yang Yu

  4. School of Computer Science and Technology, University of Science and Technology of China, Hefei, China

    Enhong Chen

  5. JD iCity, JD Technology & JD Intelligent Cities Research, Beijing, China

    Yu Zheng

  6. School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu, China

    Fei Teng

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wang, J., Zhao, L. (2022). Convergence and Applications of ADMM on the Multi-convex Problems. In: Gama, J., Li, T., Yu, Y., Chen, E., Zheng, Y., Teng, F. (eds) Advances in Knowledge Discovery and Data Mining. PAKDD 2022. Lecture Notes in Computer Science(), vol 13281. Springer, Cham. https://doi.org/10.1007/978-3-031-05936-0_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-05936-0_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-05935-3

  • Online ISBN: 978-3-031-05936-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation