Skip to main content
SpringerLink
Log in

BSPADMM: block splitting proximal ADMM for sparse representation with strong scalability

  • Regular Paper
  • Published:
CCF Transactions on High Performance Computing Aims and scope Submit manuscript

Abstract

Sparse representation (SR) is a fundamental component of linear representation techniques and plays a crucial role in signal processing, machine learning, and computer vision. Most parallel methods for solving sparse representations rely on the alternating direction method of multipliers (ADMM). However, the classical 2-block ADMM or N-block ADMM often suffer from three problems: (1) solving the subproblem requires solving a linear system, (2) unsuitable sparse data structure for parallelization, and (3) unsatisfactory parallel efficiency and scalability performance. In this paper, we propose a parallel ADMM-based algorithm called block splitting proximal ADMM (BSPADMM). First, BSPADMM organizes the sparse signals in the compressed sparse columns (CSC) format, and each processor deals with them independently. Second, BSPADMM designs the proximal term that avoids solving a linear system of the subproblem during iterations. Its advantage is that the BSPADMM computes the subproblem by using sparse matrix–vector multiplication, without communication between processors. Third, each processor updates the size asynchronously, which eliminates the synchronization effort of adjusting the step size between processes. Thus, the communication overhead can be naturally reduced. Our experimental results on three datasets of varying scales show that BSPADMM outperforms state-of-the-art ADMM techniques, including the adaptive relaxed ADMM (ARADMM) and N-block ADMM, in terms of computing time and parallel efficiency. BSPADMM runs 1.64 times faster than the N-block ADMM, and the ratio grows to 8.27 times as the dataset size doubles. More importantly, the parallel efficiency of BSPADMM remains above \(70\%\) as the number of processors grows to 10,000, demonstrating strong scalability.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  • Noise modeling and representation based classification methods for face recognition. Neurocomputing. 148 , 420–429 (2015)

  • Edoardo Amaldi, A., Viggo Kann, B.: On the approximability of minimizing nonzero variables or unsatisfied relations in linear systems. Theoretical Computer Science. 209(1–2), 237–260 (1998)

  • Bao, C., Ji, H., Quan, Y., Shen, Z.: L0 Norm based dictionary learning by proximal methods with global convergence. In 2014 IEEE Conference on Computer Vision and Pattern Recognition. pp. 3858–3865 (2014)

  • Bao, C., Wu, Y., Ling, H., Ji, H.: Real time robust L1 tracker using accelerated proximal gradient approach. In 2012 IEEE Conference on Computer Vision and Pattern Recognition. pp. 1830–1837 (2012)

  • Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and trends in machine learning. 3, (1): 1–122 (2011)

  • Cai, Xingju, Han, Deren, Yuan, Xiaoming: On the convergence of the direct extension of ADMM for three-block separable convex minimization models with one strongly convex function. Comput Optim Appl. 66(1), 39–73 (2017)

    Article  MathSciNet  Google Scholar 

  • NVIDIA Corporation. NVIDAM CuSolver. NVIDIA. https://developer.nvidia.com/cusolver (2015)

  • Dai, Y.-H., Hager, W.W., Schittkowski, K., Zhang, H.: The cyclic Barzilai-C̈Borwein method for unconstrained optimization. IMA J. Numer. Anal. 26(3), 604–627 (2006)

    Article  MathSciNet  Google Scholar 

  • Davis, Damek: A three-operator splitting scheme and its optimization applications. Set-Valued Var Anal 25(2017), 829–858 (2017)

    Article  MathSciNet  Google Scholar 

  • Deng, X., Liu, F., Huang, F.: Linear convergence rate of splitting algorithms for multi-block constrained convex minimizations. IEEE Access. 8, 120694–120700 (2020)

    Article  Google Scholar 

  • For most large underdetermined systems of linear equations the minimal \(l_{1}\)-norm solution is also the sparsest solution. Commun Pure Appl Math. 59(6), 797–829 (2006)

  • Donoho, D.L., Tsaig, Y.: Fast solution of \(\ell _{1}\) -norm minimization problems when the solution may be sparse. IEEE Trans Inf Theory. 54(11), 4789–4812 (2008)

  • Elgabli, A., Elghariani, A., Aggarwal, V., Bennis, M., Bell, M.: A proximal Jacobian ADMM approach for fast massive MIMO signal detection in low-latency communications. IEEE. pp. 1–6 (2019)

  • Field, J.D.: Relations between the statistics of natural images and the response properties of cortical cells. J Opt Soc Am A-Opt Image Sci Vis. 4(12), 2379–2394 (1987)

    Article  Google Scholar 

  • Gropp, W., Smith, B., McInnes, L.C.: PETSC 2.0. Portable Extensible Toolkit for Scientific Computation. United States (1995)

  • Han, D., Yuan, X.: A note on the alternating direction method of multipliers. J Optim Theory Appl. 155(1), 227–238 (2012)

    Article  MathSciNet  Google Scholar 

  • He, B., Yuan, X..: On non-ergodic convergence rate of Douglas—Rachford alternating direction method of multipliers. Numer. Math. 130, (3): 567C̈577 (2015)

  • Hong, M., Luo, Z-Q.: On the linear convergence of the alternating direction method of multipliers. Math. Program. 162, 1C̈2 (2017), 165C̈199 (2017)

  • Hu, W., Qin, X., Jiang, Q., Chen, J., An, H., Jia, W., Yang, C., Wang, L., Yang, C., Lin, L.: High performance computing of DGDFT for tens of thousands of atoms using millions of cores on Sunway TaihuLight. Sci Bull. 66(2), 111–119 (2021)

    Article  Google Scholar 

  • Huang, K., Aviyente, S.: Sparse Representation for signal classification. In Proceedings of the 19th International Conference on Neural Information Processing Systems (Canada) (NIPS’06). MIT Press, Cambridge. pp. 609C̈616 (2006)

  • Hubel, D.H., Wiesel, T.N.: Receptive fields of single neurones in the cat’s striate cortex. J Physiol. 148(3), 574–591 (1959)

    Article  Google Scholar 

  • Jenatton, R., Mairal, J., Obozinski, G., Bach, F..: Proximal methods for sparse hierarchical dictionary learning. Proceedings of the International Conference on Machine Learning (ICML). pp. 487–494 (2010)

  • Jin, J.W., Wen, S.: An algorithm twisted from generalized ADMM for multi-block separable convex minimization models. J. Comput. Appl. Math. 309(2017), 342–358 (2017)

    MathSciNet  Google Scholar 

  • Krause, A., Cevher, V..: Submodular dictionary selection for sparse representation. In Proceedings of the 27th International Conference on International Conference on Machine Learning (Haifa, Israel) (ICML’10). Omnipress, Madison. pp. 567C̈574. 9781605589077 (2010)

  • LAPACK.: LAPACK linear system solver. LAPACK. https://netlib.org/lapack/ (2000)

  • Lazebnik, S., Schmid, C., Ponce, J.: A sparse texture representation using affine-invariant regions. In 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings. 2, 109–123 (2003)

  • Li, M., Sun, D., Toh, K.-C.: A Convergent 3-block semi-proximal ADMM for convex minimization problems with one strongly convex block. Asia-Pacific J Operat Res. 32, 1550024 (2015)

    Article  MathSciNet  Google Scholar 

  • Lin, T., Ma, S., Zhang, S.: On the global linear convergence of the ADMM with multiblock variables. SIAM J Optim. 25(3), 1478–1497 (2015)

    Article  MathSciNet  Google Scholar 

  • Lin, T., Ma, S., Zhang, S.: Iteration complexity analysis of multi-block ADMM for a family of convex minimization without strong convexity. J Sci Comput. 69(1), 52–81 (2016)

    Article  MathSciNet  Google Scholar 

  • Lin, T., Ma, S., Zhang, S.: Iteration complexity analysis of multi-block ADMM for a family of convex minimization without strong convexity. J. Sci. Comput. 69(1): 52C̈81 (2016b)

  • Liu, H., Song, B., Qin, H., Qiu, Z.: An adaptive-ADMM algorithm with support and signal value detection for compressed sensing. IEEE Signal Process Lett. 20(4), 315–318 (2013). https://doi.org/10.1109/LSP.2013.2245893

    Article  Google Scholar 

  • Nesterov, Y.: Excessive gap technique in nonsmooth convex minimization. SIAM J. Optim. 16(1): 235C̈249. 1052-6234 (2005)

  • Patel, VM., Chellappa, R.: Sparse representations, compressive sensing and dictionaries for pattern recognition. In The First Asian Conference on Pattern Recognition. pp. 325–329 (2011)

  • Peng, G.-J.: Adaptive ADMM for dictionary learning in convolutional sparse representation. IEEE Trans Image Process. 28(7), 3408–3422 (2019). https://doi.org/10.1109/TIP.2019.2896541

    Article  MathSciNet  Google Scholar 

  • Plumbley, MD.: Recovery of sparse representations by polytope faces Pursuit. In Proceedings of the 6th International Conference on Independent Component Analysis and Blind Signal Separation (Charleston, SC) (ICA’06). Springer-Verlag, Berlin, Heidelberg. pp. 206C̈213. 3540326308 (2006)

  • Shen, Y., Zuo, Y., Yu, A.: A partially proximal S-ADMM for separable convex optimization with linear constraints. Appl. Numer. Math. 160(2021), 65–83 (2021)

    Article  MathSciNet  Google Scholar 

  • Shurong, Z.: Selection of components and degrees of smoothing via lasso in high dimensional nonparametric additive models. Comput Stat Data Anal 53(1), 164–175 (2008)

    Article  MathSciNet  Google Scholar 

  • Sun, H, Wang, J, Deng, T: On the global and linear convergence of direct extension of ADMM for 3-block separable convex minimization models. J Inequal Appli. pp. 1–14 (2016)

  • Sun, R., Luo, Z.-Q., Ye, Y.: On the efficiency of random permutation for ADMM and coordinate descent. Math Operat Res. 45, 1–14 (2020)

    Article  MathSciNet  Google Scholar 

  • Tibshirani, R.: Regression shrinkage and selection via the lasso: a retrospective. J R Stat Soc: Series B (Stat Methodol). 73(3), 267–288 (2011)

    Article  MathSciNet  Google Scholar 

  • Tropp, J.A.: Greed is good: algorithmic results for sparse approximation. IEEE Trans Inf Theory 50(10), 2231–2242 (2004)

    Article  MathSciNet  Google Scholar 

  • Wang, J., Yang, J., Yu, K., Lv, F., Huang, T., Gong, Y.: Locality-constrained linear coding for image classification. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. pp. 3360–3367. (2010) https://doi.org/10.1109/CVPR.2010.5540018

  • Wright, SJ., Nowak, RD., Figueiredo, MAT.: Sparse reconstruction by separable approximation. Trans. Sig. Proc. 57(7): 2479C̈2493 (2009)

  • Xu, Z., Figueiredo, MT., Yuan, X., Studer, C., Goldstein, T.: Adaptive relaxed ADMM: convergence theory and practical implementation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 7234–7243 (2017)

  • Yang, A.Y., Ma, Y., Wright, J., Ganesh, A., Sastry, S.: Robust face recognition via sparse representation. IEEE Trans Pattern Anal Mach Intell. 31, 210–227 (2009)

    Article  Google Scholar 

  • Yang, J., Wright, J., Huang, T., Ma, Y.: Image super-resolution as sparse representation of raw image patches. In 2008 IEEE Conference on Computer Vision and Pattern Recognition. pp. 1–8 (2008)

  • Zhang, Z., Xu, Y., Yang, J., Li, X., Zhang, D.: A survey of sparse representation: algorithms and applications. IEEE Access. 3, 490–530 (2015). https://doi.org/10.1109/ACCESS.2015.2430359

    Article  Google Scholar 

Download references

Acknowledgements

This material is based upon work supported by the National Science Foundationhttp://dx.doi.org/10.13039/100000001 under Grant No.nnnnnnn and Grant No.mmmmmmm. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author and do not necessarily reflect the views of the National Science Foundation.

Author information

Authors and Affiliations

  1. Computer Network Information Center, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing, China

    Yidong Chen & Zidong Han

  2. Shandong Computer Science Center (National Supercomputing Center in Jinan), Qilu University of Technology (Shandong Academy of Sciences), Jinan, 250013, Shandong, People’s Republic of China

    Jingshan Pan & Meng Guo

  3. Jinan Institute of Supercomputing Technology, Jinan, 250103, Shandong, People’s Republic of China

    Jingshan Pan & Meng Guo

  4. Central University of Finance and Economics, Beijing, China

    Yonghong Hu

  5. Computer Network Information Center, Chinese Academy of Sciences, Beijing, China

    Zhonghua Lu

Authors
  1. Yidong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jingshan Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zidong Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yonghong Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Meng Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Zhonghua Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhonghua Lu.

Ethics declarations

Conflict of interest

The authors declared that they have no conflicts of interest to this work. We declare that we do not have any commercial or associative interest that represents a conflict of interest in connection with the work submitted.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, Y., Pan, J., Han, Z. et al. BSPADMM: block splitting proximal ADMM for sparse representation with strong scalability. CCF Trans. HPC 6, 3–16 (2024). https://doi.org/10.1007/s42514-023-00164-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42514-023-00164-w

Keywords

Search

Navigation