Skip to main content
SpringerLink
Log in

Convergence of the Surrogate Lagrangian Relaxation Method

  • Published:
Journal of Optimization Theory and Applications Aims and scope Submit manuscript

Abstract

Studies have shown that the surrogate subgradient method, to optimize non-smooth dual functions within the Lagrangian relaxation framework, can lead to significant computational improvements as compared to the subgradient method. The key idea is to obtain surrogate subgradient directions that form acute angles toward the optimal multipliers without fully minimizing the relaxed problem. The major difficulty of the method is its convergence, since the convergence proof and the practical implementation require the knowledge of the optimal dual value. Adaptive estimations of the optimal dual value may lead to divergence and the loss of the lower bound property for surrogate dual values. The main contribution of this paper is on the development of the surrogate Lagrangian relaxation method and its convergence proof to the optimal multipliers, without the knowledge of the optimal dual value and without fully optimizing the relaxed problem. Moreover, for practical implementations, a stepsizing formula that guarantees convergence without requiring the optimal dual value has been constructively developed. The key idea is to select stepsizes in a way that distances between Lagrange multipliers at consecutive iterations decrease, and as a result, Lagrange multipliers converge to a unique limit. At the same time, stepsizes are kept sufficiently large so that the algorithm does not terminate prematurely. At convergence, the lower-bound property of the surrogate dual is guaranteed. Testing results demonstrate that non-smooth dual functions can be efficiently optimized, and the new method leads to faster convergence as compared to other methods available for optimizing non-smooth dual functions, namely, the simple subgradient method, the subgradient-level method, and the incremental subgradient method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. While Augmented Lagrangian relaxation has been a powerful method and can alleviate zigzagging, thereby reducing computational requirements, it is generally not used to optimize dual functions. Furthermore, the extra quadratic term makes the problem non-separable. Although methods were developed to overcome the resulting non-separability issue, they were not very effective.

  2. Strictly speaking, when dealing with inequality constraints \(g(x)\le 0\), distances between multipliers and projections of multiples from the previous iteration are considered rather than distances between multipliers.

  3. In the subgradient method, zero-subgradient implies that the optimum is obtained, and the algorithm terminates with the optimal primal solution. In the surrogate subgradient method, zero-surrogate subgradient implies that only a feasible solution is obtained, and the algorithm must proceed.

  4. Initial stepsize \(c^{0}\) is initialized to be a positive scalar, therefore, stepsizes \(c^{k}\), \(k\) = 1, 2, ... satisfying (18) are positive.

  5. When \(\beta ^k<<1\), the right-hand side of (35) decreases faster than the left-hand side as \(k\) increases. This leads to the contradiction, and the theorem is proved.

  6. For the GAP15900 instance, the implementation of the new method may resemble that of the interleaved method [14] since only one subproblem is optimized at a time. The important difference between the new method and the interleaved method is the stepsizing formula.

  7. Performance of all methods in Figs. 3, 4, 5 is tested by comparing distances to multipliers obtained by a subgradient method with non-summable stepsizes [22] after sufficiently many iterations (\(>\)20000).

References

  1. Ermoliev, Y.M.: Methods for solving nonlinear extremal problems. Cybernetics 2(4), 1–17 (1966)

    Article  Google Scholar 

  2. Polyak, B.T.: A general method of solving extremum problems. Sov. Math. Doklady 8, 593–597 (1967)

    MATH  Google Scholar 

  3. Polyak, B.T.: Minimization of unsmooth functionals. USSR Comput. Math. Math. Phys. 9(3), 14–29 (1969). (in Russian)

    Article  Google Scholar 

  4. Shor, N.Z.: On the rate of convergence of the generalized gradient method. Cybernetics 4(3), 79–80 (1968)

    Article  MathSciNet  Google Scholar 

  5. Shor, N.Z.: Generalized gradient methods for non-smooth functions and their applications to mathematical programming problems. Econ. Math. Methods 12(2), 337–356 (1976). (in Russian)

    MATH  Google Scholar 

  6. Goffin, J.-L.: On the finite convergence of the relaxation method for solving systems of inequalities. Operations Research Center Report ORC 71–36, University of California at Berkeley, Berkeley (1971).

  7. Goffin, J.-L., Kiwiel, K.: Convergence of a simple subgradient level method. Math. Program. 85(1), 207–211 (1999)

    Article  MATH  MathSciNet  Google Scholar 

  8. Nedic, A., Bertsekas, D.P.: Convergence rate of incremental subgradient algorithms. In: Uryasev, S., Pardalos, P.M. (eds.) Stochastic Optimization: Algorithms and Applications, pp. 263–304. Kluwer Academic, New York (2000)

    Google Scholar 

  9. Nedic, A., Bertsekas, D.: Incremental subgradient methods for nondifferentiable optimization. SIAM J. Optim. 56(1), 109–138 (2001)

    Article  MathSciNet  Google Scholar 

  10. Nesterov, Y.: Smooth minimization of non-smooth functions. Math. Program. 103(1), 127–152 (2005)

    Article  MATH  MathSciNet  Google Scholar 

  11. Bertsekas, D.P.: Incremental gradient, subgradient, and proximal methods for convex optimization: a survey. LIDS Technical Report no. 2848, MIT, (2010).

  12. Bertsekas, D.P.: Incremental proximal methods for large scale convex optimization. Math. Program. 129, 163–195 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  13. Lemarechal, C., Nemirovskii, A.S., Nesterov, Y.E.: New variants of bundle methods. Math. Program. 69, 111–147 (1995)

    Article  MATH  MathSciNet  Google Scholar 

  14. Kaskavelis, C.A., Caramanis, M.C.: Efficient Lagrangian relaxation algorithms for industry size job-shop scheduling problems. IIE Trans. 30(11), 1085–1097 (1998)

    Google Scholar 

  15. Zhao, X., Luh, P.B., Wang, J.: Surrogate gradient algorithm for Lagrangian relaxation. J. Optim. Theory Appl. 100(3), 699–712 (1999)

    Article  MATH  MathSciNet  Google Scholar 

  16. Luh, P.B., Blankson, W.E., Chen, Y., Yan, J.H., Stern, G.A., Chang, S.C., Zhao, F.: Payment cost minimization auction for the deregulated electricity markets using surrogate optimization. IEEE Trans. Power Syst. 21(2), 568–578 (2006)

    Article  Google Scholar 

  17. Sun, T., Zhao, Q.C., Luh, P.B.: On the surrogate gradient algorithm for Lagrangian relaxation. J. Optim. Theory Appl. 133(3), 413–416 (2007)

    Article  MATH  MathSciNet  Google Scholar 

  18. Chang, T.S.: Comments on “Surrogate gradient algorithm for Lagrangian relaxation”. J. Optim. Theory Appl. 137(3), 691–697 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  19. Bragin, M.A., Han, X., Luh, P.B., Yan, J.H.: Payment cost minimization using Lagrangian relaxation and modified surrogate optimization approach. In: Proceedings of the IEEE Power Engineering Society, General Meeting, Detroit, Michigan (2011)

  20. Bragin, M.A., Luh, P.B., Yan, J.H., Yu, N., Han, X., Stern, G.A.: An efficient surrogate subgradient method within Lagrangian relaxation for the payment cost minimization problem. In: Proceedings of the IEEE Power Engineering Society, General Meeting, San Diego (2012)

  21. Allen, E., Nelgason, R., Kennongton, J., Shettym, B.: A generalization of Polyak’s convergence result for subgradient optimization. Math. Program. 37(3), 309–317 (1987)

    Article  MATH  Google Scholar 

  22. Bertsekas, D.P.: Nonlinear Programming. Athena Scientific, Massachusetts (2008)

    Google Scholar 

  23. Wah, B.W., Chen, Y.X.: Subgoal partitioning and global search for solving temporal planning problems in mixed space. Int. J. Artif. Intell. Tools 13(4), 767–790 (2004)

    Article  Google Scholar 

  24. Chu, P.C., Beasley, J.E.: A genetic algorithm for the generalized assignment problem. Comput. Oper. Res. 24(1), 17–23 (1997)

    Article  MATH  MathSciNet  Google Scholar 

  25. Yagiura, M., Yamaguchi, T., Ibaraki, T.: A variable depth search algorithm with branching search for the generalized assignment problem. Optim. Methods Softw. 10, 419–441 (1998)

    Article  MATH  MathSciNet  Google Scholar 

  26. Yagiura, M., Ibaraki, T., Glover, F.: A path relinking approach with ejection chains for the generalized assignment problem. Eur. J. Oper. Res. 169(2), 548–569 (2006)

    Article  MATH  MathSciNet  Google Scholar 

  27. Avella, P., Boccia, M., Vasilyev, I.: A computational study of exact knapsack separation for the generalized assignment problem. Comput. Optim. Appl. 45(3), 543–555 (2010)

    Article  MATH  MathSciNet  Google Scholar 

  28. Posta, M., Ferland, J.A., Michelon, P.: An exact method with variable fixing for solving the generalized assignment problem. Comput. Optim. Appl. 52(3), 629–644 (2012)

    Article  MATH  MathSciNet  Google Scholar 

  29. Özbakir, L., Baykasoglu, A., Tapkan, P.: Bees algorithm for generalized assignment problem. Appl. Math. Comput. 215(11), 3782–3795 (2010)

    Article  MATH  MathSciNet  Google Scholar 

  30. Asahiro, Y., Ishibashi, M., Yamashita, M.: Independent and cooperative parallel search methods for the generalized assignment problem. Optim. Methods Softw. 18(2), 129–141 (2003)

    Article  MATH  MathSciNet  Google Scholar 

  31. Laguna, M., Kelly, J.P., Gonzalez-Velarde, J.L., Glover, F.: Tabu search for the multilevel generalized assignment problem. Eur. J. Oper. Res. 82, 176–189 (1995)

    Article  MATH  Google Scholar 

  32. Koopmans, T.C., Beckmann, M.J.: Assignment problems and the location of economic activities. Econometrica 25(1), 53–76 (1957)

    Article  MATH  MathSciNet  Google Scholar 

  33. Dickey, J.W., Hopkins, J.W.: Campus building arrangement using TOPAZ. Transp. Res. 6, 59–68 (1972)

    Article  Google Scholar 

  34. Elshafei, A.N.: Hospital layout as a quadratic assignment problem. Oper. Res. Q. 28, 167–179 (1977)

    Article  MATH  Google Scholar 

  35. Geoffrion, A.M., Graves, G.W.: Scheduling parallel production lines with changeover costs: practical applications of a quadratic assignment/LP approach. Oper. Res. 24, 595–610 (1976)

    Article  MATH  Google Scholar 

  36. Krarup, J., Pruzan, P.M.: Computer-aided layout design. Math. Program. Study 9, 75–94 (1978)

    Article  MathSciNet  Google Scholar 

  37. Burkard, R.E., Offermann, J.: Entwurf von Schreibmaschinentastaturen mittels quadratischer Zuordnungsprobleme. Zeitschrift fur Oper. Res. 21(4), 121–132 (1977). (in German)

    Google Scholar 

  38. Christofides, N., Benavent, E.: An exact algorithm for the quadratic assignment problem. Oper. Res. 37(5), 760–768 (1989)

    Article  MATH  MathSciNet  Google Scholar 

Download references

Acknowledgments

This work was supported in part by grants from Southern California Edison and by the National Science Foundation under Grant ECCS–1028870. The authors would like to acknowledge Congcong Wang and Yaowen Yu for their careful perusal of the paper, insightful comments and valuable suggestions during numerous discussions.

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, University of Connecticut, Storrs, CT , 06269-2157, USA

    Mikhail A. Bragin & Peter B. Luh

  2. Southern California Edison, Rosemead, CA , 91770, USA

    Joseph H. Yan, Nanpeng Yu & Gary A. Stern

Authors
  1. Mikhail A. Bragin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peter B. Luh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Joseph H. Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nanpeng Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gary A. Stern
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mikhail A. Bragin.

Additional information

Communicated by Fabián Flores-Bazàn.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bragin, M.A., Luh, P.B., Yan, J.H. et al. Convergence of the Surrogate Lagrangian Relaxation Method. J Optim Theory Appl 164, 173–201 (2015). https://doi.org/10.1007/s10957-014-0561-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10957-014-0561-3

Keywords

Mathematics Subject Classification (2000)

Search

Navigation