Combining Lagrangian decomposition and excessive gap smoothing technique for solving large-scale separable convex optimization problems

  • Quoc Tran Dinh
  • Carlo Savorgnan
  • Moritz Diehl


A new algorithm for solving large-scale convex optimization problems with a separable objective function is proposed. The basic idea is to combine three techniques: Lagrangian dual decomposition, excessive gap and smoothing. The main advantage of this algorithm is that it automatically and simultaneously updates the smoothness parameters which significantly improves its performance. The convergence of the algorithm is proved under weak conditions imposed on the original problem. The rate of convergence is \(O(\frac {1}{k})\), where k is the iteration counter. In the second part of the paper, the proposed algorithm is coupled with a dual scheme to construct a switching variant in a dual decomposition framework. We discuss implementation issues and make a theoretical comparison. Numerical examples confirm the theoretical results.


Excessive gap Smoothing technique Lagrangian decomposition Proximal mappings Large-scale problem Separable convex optimization Distributed optimization 



The authors would like to thank Dr. Ion Necoara and Dr. Michel Baes for useful comments on the text and for pointing out some interesting references. Furthermore, the authors are grateful to Dr. Paschalis Tsiaflakis for providing the problem data in the last numerical example.

Research supported by Research Council KUL: CoE EF/05/006 Optimization in Engineering (OPTEC), IOF-SCORES4CHEM, GOA/10/009 (MaNet), GOA /10/11, several PhD/postdoc and fellow grants; Flemish Government: FWO: PhD/postdoc grants, projects G.0452.04, G.0499.04, G.0211.05, G.0226.06, G.0321.06, G.0302.07, G.0320.08, G.0558.08, G.0557.08, G.0588.09, G.0377.09, G.0712.11, research communities (ICCoS, ANMMM, MLDM); IWT: PhD Grants, Belgian Federal Science Policy Office: IUAP P6/04; EU: ERNSI; FP7-HDMPC, FP7-EMBOCON, ERC-HIGHWIND, Contract Research: AMINAL. Other: Helmholtz-viCERP, COMET-ACCM.


  1. 1.
    Bertsekas, D.P., Tsitsiklis, J.N.: Parallel and Distributed Computation: Numerical Methods. Prentice Hall, New York (1989) zbMATHGoogle Scholar
  2. 2.
    Boyd, S., Parikh, N., Chu, E., Peleato, B.: Distributed optimization and statistics via alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2011) CrossRefGoogle Scholar
  3. 3.
    Chen, G., Teboulle, M.: A proximal-based decomposition method for convex minimization problems. Math. Program. 64, 81–101 (1994) MathSciNetzbMATHCrossRefGoogle Scholar
  4. 4.
    Connejo, A.J., Mínguez, R., Castillo, E., García-Bertrand, R.: Decomposition Techniques in Mathematical Programming: Engineering and Science Applications. Springer, Berlin (2006) Google Scholar
  5. 5.
    Dolan, E.D., Moré, J.J.: Benchmarking optimization software with performance profiles. Math. Program. 91, 201–213 (2002) MathSciNetzbMATHCrossRefGoogle Scholar
  6. 6.
    Duchi, J.C., Agarwal, A., Wainwright, M.J.: Dual averaging for distributed optimization: Convergence analysis and network scaling. IEEE Trans. Autom. Control 57(3), 592–606 (2012) MathSciNetCrossRefGoogle Scholar
  7. 7.
    Eckstein, J., Bertsekas, D.: On the Douglas–Rachford splitting method and the proximal point algorithm for maximal monotone operators. Math. Program. 55, 293–318 (1992) MathSciNetzbMATHCrossRefGoogle Scholar
  8. 8.
    Facchinei, F., Pang, J.-S.: Finite-Dimensional Variational Inequalities and Complementarity Problems, vols. 1–2. Springer, Berlin (2003) Google Scholar
  9. 9.
    Goldfarb, D., Ma, S.: Fast multiple splitting algorithms for convex optimization. SIAM J. Optim. 22(2), 533–556 (2012) MathSciNetzbMATHCrossRefGoogle Scholar
  10. 10.
    Hamdi, A.: Two-level primal-dual proximal decomposition technique to solve large-scale optimization problems. Appl. Math. Comput. 160, 921–938 (2005) MathSciNetzbMATHCrossRefGoogle Scholar
  11. 11.
    Han, S.P., Lou, G.: A parallel algorithm for a class of convex programs. SIAM J. Control Optim. 26, 345–355 (1988) MathSciNetzbMATHCrossRefGoogle Scholar
  12. 12.
    Hariharan, L., Pucci, F.D.: Decentralized resource allocation in dynamic networks of agents. SIAM J. Optim. 19(2), 911–940 (2008) MathSciNetzbMATHCrossRefGoogle Scholar
  13. 13.
    He, B.S., Tao, M., Xu, M.H., Yuan, X.M.: Alternating directions based contraction method for generally separable linearly constrained convex programming problems. Optimization (2011). doi: 10.1080/02331934.2011.611885 Google Scholar
  14. 14.
    He, B.S., Yang, H., Wang, S.L.: Alternating directions method with self-adaptive penalty parameters for monotone variational inequalities. J. Optim. Theory Appl. 106, 349–368 (2000) MathSciNetCrossRefGoogle Scholar
  15. 15.
    He, B.S., Yuan, X.M.: On the O(1/n) convergence rate of the Douglas–Rachford alternating direction method. SIAM J. Numer. Anal. 50, 700–709 (2012) MathSciNetzbMATHCrossRefGoogle Scholar
  16. 16.
    Holmberg, K.: Experiments with primal-dual decomposition and subgradient methods for the uncapacitated facility location problem. Optimization 49(5–6), 495–516 (2001) MathSciNetzbMATHCrossRefGoogle Scholar
  17. 17.
    Holmberg, K., Kiwiel, K.C.: Mean value cross decomposition for nonlinear convex problem. Optim. Methods Softw. 21(3), 401–417 (2006) MathSciNetzbMATHCrossRefGoogle Scholar
  18. 18.
    Kojima, M., Megiddo, N., Mizuno, S., et al.: Horizontal and vertical decomposition in interior point methods for linear programs. Technical report, Information Sciences, Tokyo Institute of Technology, Tokyo (1993) Google Scholar
  19. 19.
    Lenoir, A., Mahey, P.: Accelerating convergence of a separable augmented Lagrangian algorithm. Technical report, LIMOS/RR-07-14, pp. 1–34 (2007) Google Scholar
  20. 20.
    Love, R.F., Kraemer, S.A.: A dual decomposition method for minimizing transportation costs in multifacility location problems. Transp. Sci. 7, 297–316 (1973) MathSciNetCrossRefGoogle Scholar
  21. 21.
    Mehrotra, S.: On the implementation of a primal-dual interior point method. SIAM J. Optim. 2(4), 575–601 (1992) MathSciNetzbMATHCrossRefGoogle Scholar
  22. 22.
    Necoara, I., Suykens, J.A.K.: Applications of a smoothing technique to decomposition in convex optimization. IEEE Trans. Autom. Control 53(11), 2674–2679 (2008) MathSciNetCrossRefGoogle Scholar
  23. 23.
    Nedíc, A., Ozdaglar, A.: Distributed subgradient methods for multi-agent optimization. IEEE Trans. Autom. Control 54, 48–61 (2009) CrossRefGoogle Scholar
  24. 24.
    Nesterov, Y.: A method for unconstrained convex minimization problem with the rate of convergence o(1/k 2). Dokl. Akad. Nauk SSSR 269, 543–547 (1983) (Translated as Soviet Math. Dokl.) MathSciNetGoogle Scholar
  25. 25.
    Nesterov, Y.: Introductory Lectures on Convex Optimization: A Basic Course. Applied Optimization, vol. 87. Kluwer Academic, Dordrecht (2004) zbMATHGoogle Scholar
  26. 26.
    Nesterov, Y.: Excessive gap technique in nonsmooth convex minimization. SIAM J. Optim. 16(1), 235–249 (2005) MathSciNetzbMATHCrossRefGoogle Scholar
  27. 27.
    Nesterov, Y.: Smooth minimization of non-smooth functions. Math. Program. 103(1), 127–152 (2005) MathSciNetzbMATHCrossRefGoogle Scholar
  28. 28.
    Neveen, G., Jochen, K.: Faster and simpler algorithms for multicommodity flow and other fractional packing problems. SIAM J. Comput. 37(2), 630–652 (2007) MathSciNetCrossRefGoogle Scholar
  29. 29.
    Ruszczyński, A.: On convergence of an augmented Lagrangian decomposition method for sparse convex optimization. Math. Oper. Res. 20, 634–656 (1995) MathSciNetzbMATHCrossRefGoogle Scholar
  30. 30.
    Samar, S., Boyd, S., Gorinevsky, D.: Distributed estimation via dual decomposition. In: Proceedings European Control Conference (ECC), Kos, Greece, pp. 1511–1516 (2007) Google Scholar
  31. 31.
    Spingarn, J.E.: Applications of the method of partial inverses to convex programming: decomposition. Math. Program. Ser. A 32, 199–223 (1985) MathSciNetzbMATHCrossRefGoogle Scholar
  32. 32.
    Tran-Dinh, Q., Necoara, I., Savorgnan, C., Diehl, M.: An inexact perturbed path-following method for Lagrangian decomposition in large-scale separable convex optimization. Int. Report 12-181, ESATSISTA, KU Leuven, Belgium (2012). SIAM J. Optim., accepted Google Scholar
  33. 33.
    Tseng, P.: Alternating projection-proximal methods for convex programming and variational inequalities. SIAM J. Optim. 7(4), 951–965 (1997) MathSciNetzbMATHCrossRefGoogle Scholar
  34. 34.
    Tsiaflakis, P., Diehl, M., Moonen, M.: Distributed spectrum management algorithms for multi-user DSL networks. IEEE Trans. Signal Process. 56(10), 4825–4843 (2008) MathSciNetCrossRefGoogle Scholar
  35. 35.
    Tsiaflakis, P., Necoara, I., Suykens, J.A.K., Moonen, M.: Improved dual decomposition based optimization for DSL dynamic spectrum management. IEEE Trans. Signal Process. 58(4), 2230–2245 (2010) MathSciNetCrossRefGoogle Scholar
  36. 36.
    Vania, D.S.E.: Finding approximate solutions for large scale linear programs. Ph.D. Thesis, No. 18188, ETH, Zurich (2009) Google Scholar
  37. 37.
    Venkat, A.N.: Distributed model predictive control: theory and applications. Ph.D. Thesis, University of Wisconsin-Madison (2006) Google Scholar
  38. 38.
    Wächter, A., Biegler, L.T.: On the implementation of a primal-dual interior point filter line search algorithm for large-scale nonlinear programming. Math. Program. 106(1), 25–57 (2006) MathSciNetzbMATHCrossRefGoogle Scholar
  39. 39.
    Zhao, G.: A Lagrangian dual method with self-concordant barriers for multistage stochastic convex programming. Math. Program. 102, 1–24 (2005) MathSciNetzbMATHCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2012

Authors and Affiliations

  • Quoc Tran Dinh
    • 1
    • 2
  • Carlo Savorgnan
    • 1
  • Moritz Diehl
    • 1
  1. 1.Department of Electrical Engineering (ESAT-SCD) and Optimization in Engineering Center (OPTEC)KU LeuvenHeverlee-LeuvenBelgium
  2. 2.Vietnam National UniversityHanoiVietnam

Personalised recommendations