Lagrangian Relaxation

  • Claude Lemaréchal
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2241)


Lagrangian relaxation is a tool to find upper bounds on a given (arbitrary) maximization problem. Sometimes, the bound is exact and an optimal solution is found. Our aim in this paper is to review this technique, the theory behind it, its numerical aspects, its relation with other techniques such as column generation.


Dual Problem Dual Function Column Generation Lagrangian Relaxation Nonsmooth Optimization 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    F. Alizadeh. Interior point methods in semidefinite programming with applications to combinatorial optimization. SIAM Journal on Optimization, 5(1):13–51, 1995.MATHCrossRefMathSciNetGoogle Scholar
  2. 2.
    K. Anstreicher and L.A. Wolsey. On dual solutions in subgradient opimization. Unpublished manuscript, CORE, Louvain-la-Neuve, Belgium, 1993.Google Scholar
  3. 3.
    D.P. Bertsekas. Projected Newton methods for optimization problems with simple constraints. SIAM Journal on Control and Optimization, 20:221–246, 1982.MATHCrossRefMathSciNetGoogle Scholar
  4. 4.
    M.J. Best. Equivalence of some quadratic programming algorithms. Mathematical Programming, 30:71–87, 1984.MATHCrossRefMathSciNetGoogle Scholar
  5. 5.
    U. Brännlund. On relaxation methods for nonsmooth convex optimization. PhD thesis, Royal Institute of Technology-Stockholm, 1993.Google Scholar
  6. 6.
    P.M. Camerini, L. Fratta, and F. Maffioli. On improving relaxation methods by modified gradient techniques. Mathematical Programming Study, 3:26–34, 1975.MathSciNetGoogle Scholar
  7. 7.
    E. Cheney and A. Goldstein. Newton’s method for convex programming and Tchebyche. approximations. Numerische Mathematik, 1:253–268, 1959.MATHCrossRefMathSciNetGoogle Scholar
  8. 8.
    R. Correa and C. Lemaréchal. Convergence of some algorithms for convex minimization. Mathematical Programming, 62(2):261–275, 1993.CrossRefMathSciNetGoogle Scholar
  9. 9.
    A. Decarreau, D. Hilhorst, C. Lemarechal, and J. Navaza. Dual methods in entropy maximization. application to some problems in crystallography. SIAM Journal on Optimization, 2(2):173–197, 1992.MATHCrossRefMathSciNetGoogle Scholar
  10. 10.
    I. Ekeland and R. Temam. Convex Analysis and Variational Problems. North Holland, 1976; reprinted by SIAM, 1999.Google Scholar
  11. 11.
    Y.M. Ermol'ev. Methods of solution of nonlinear extremal problems. Kibernetica, 2(4):1–17, 1966.MathSciNetGoogle Scholar
  12. 12.
    H. Everett III. Generalized lagrange multiplier method for solving problems of optimum allocation of resources. Operations Research, 11:399–417, 1963.MATHMathSciNetGoogle Scholar
  13. 13.
    J.E. Falk. Lagrange multipliers and nonconvex programs. SIAM Journal on Control, 7(4):534–545, 1969.MATHCrossRefMathSciNetGoogle Scholar
  14. 14.
    S. Feltenmark and K. C. Kiwiel. Dual applications of proximal bundle methods, including lagrangian relaxation of nonconvex problems. SIAM Journal on Optimization, 10(3):697–721, 2000.MATHCrossRefMathSciNetGoogle Scholar
  15. 15.
    R. Fletcher. Practical Methods of Optimization. John Wiley & Sons, Chichester (second edition), 1987.MATHGoogle Scholar
  16. 16.
    A. Frangioni. Solving semidefinite quadratic problems within nonsmooth optimization algorithms. Computational Operational Research, 23(11):1099–1118, 1996.MATHCrossRefMathSciNetGoogle Scholar
  17. 17.
    M. Fukushima and L. Qi. A globally and superlinearly convergent algorithm for nonsmooth convex minimization. SIAM Journal on Optimization, 6:1106–1120, 1996.MATHCrossRefMathSciNetGoogle Scholar
  18. 18.
    C. Garrod and J.K. Percus. Reduction of the N-particle variational problem. Journal of Mathematical Physics, 5(12), 1964.Google Scholar
  19. 19.
    A.M. Geoffrion. Lagrangean relaxation for integer programming. Mathematical Programming Study, 2:82–114, 1974.MathSciNetGoogle Scholar
  20. 20.
    M. X. Goemans and D. P. Williamson. Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming. Journal of the ACM, 6:1115–1145, 1995.CrossRefMathSciNetGoogle Scholar
  21. 21.
    J.L. Goffin, A. Haurie, and J.Ph. Vial. Decomposition and nondifferentiable optimization with the projective algorithm. Management Science, 38(2):284–302, 1992.MATHGoogle Scholar
  22. 22.
    J.L. Goffin and J.Ph. Vial. Convex nondifferentiable optimization: a survey focussed on the analytic center cutting plane method. to appear in Optimization Methods and Software; also as HEC/Logilab Technical Report 99.02, Univ. of Geneva, Switzerland.Google Scholar
  23. 23.
    M. Held and R. Karp. The travelling salesman problem and minimum spanning trees: Part II. Mathematical Programming, 1:6–25, 1971.MATHCrossRefMathSciNetGoogle Scholar
  24. 24.
    C. Helmberg and F. Rendl. A spectral bundle method for semidefinite programming. SIAM Journal on Optimization, 10(3):673–696, 2000.MATHCrossRefMathSciNetGoogle Scholar
  25. 25.
    J.-B. Hiriart-Urruty and C. Lemaréchal. Convex Analysis and Minimization Algorithms. Springer Verlag, Heidelberg, 1993.Google Scholar
  26. 26.
    J.-B. Hiriart-Urruty and C. Lemaréchal. Fundamentals of Convex Analysis. Springer Verlag, Heidelberg, 2001. to appear.MATHGoogle Scholar
  27. 27.
    R.A. Horn and Ch.R. Johnson. Matrix Analysis. Cambridge University Press, 1989. (New edition, 1999).Google Scholar
  28. 28.
    J. E. Kelley. The cutting plane method for solving convex programs. J. Soc. Indust. Appl. Math., 8:703–712, 1960.CrossRefMathSciNetGoogle Scholar
  29. 29.
    K.C. Kiwiel. Methods of Descent for Nondifferentiable Optimization. Lecture Notes in Mathematics 1133. Springer Verlag, Heidelberg, 1985.MATHGoogle Scholar
  30. 30.
    K.C. Kiwiel. A method for solving certain quadratic programming problems arising in nonsmooth optimization. IMA Journal of Numerical Analysis, 6:137–152, 1986.MATHCrossRefMathSciNetGoogle Scholar
  31. 31.
    K.C. Kiwiel. A dual method for certain positive semidefinite quadratic programming problems. SIAM Journal on Scientific and Statistical Computing, 10:175–186, 1989.MATHCrossRefMathSciNetGoogle Scholar
  32. 32.
    K.C. Kiwiel. Proximity control in bundle methods for convex nondifferentiable minimization. Mathematical Programming, 46(1):105–122, 1990.MATHCrossRefMathSciNetGoogle Scholar
  33. 33.
    K.C. Kiwiel. Efficiency of proximal bundle methods. Journal of Optimization Theory and Applications, 104(3):589–603, 2000.MATHCrossRefMathSciNetGoogle Scholar
  34. 34.
    K.C. Kiwiel, T. Larsson, and P.O. Lindberg. The efficiency of ballstep subgradient level methods for convex optimization. Mathematics of Operations Research, 24(1):237–254, 1999.MATHMathSciNetGoogle Scholar
  35. 35.
    T. Larsson, M. Patriksson, and A.B. Strömberg. Ergodic, primal convergence in dual subgradient schemes for convex programming. Mathematical Programming, 86(2):283–312, 1999.MATHCrossRefMathSciNetGoogle Scholar
  36. 36.
    C. Lemaréchal, A.S. Nemirovskii, and Yu.E. Nesterov. New variants of bundle methods. Mathematical Programming, 69:111–148, 1995.CrossRefMathSciNetGoogle Scholar
  37. 37.
    C. Lemaréchal, Yu. Nesterov, and F. Oustry. Duality gap analysis for problems with quadratic constraints, 2001. In preparation.Google Scholar
  38. 38.
    C. Lemaréchal and F. Oustry. Semi-definite relaxations and lagrangian duality with application to combinatorial optimization. Rapport de Recherche 3710, Inria, 1999.
  39. 39.
    C. Lemaréchal and F. Oustry. Nonsmooth algorithms to solve semidefinite programs. In L. El Ghaoui and S-I. Niculescu, editors, Advances in Linear Matrix Inequality Methods in Control, Advances in Design and Control, 2, pages 57–77. SIAM, 2000.Google Scholar
  40. 40.
    C. Lemaréchal, F. Oustry, and C. Sagastizábal. The U-lagrangian of a convex function. Transactions of the AMS, 352(2):711–729, 2000.MATHCrossRefGoogle Scholar
  41. 41.
    C. Lemaréchal, F. Pellegrino, A. Renaud, and C. Sagastizábal. Bundle methods applied to the unit-commitment problem. In J. Doležal and J. Fidler, editors, System Modelling and Optimization, pages 395–402, 1996.Google Scholar
  42. 42.
    C. Lemaréchal and A. Renaud. A geometric study of duality gaps, with applications. Mathematical Programming, 90(3):399–427, 2001.MATHCrossRefMathSciNetGoogle Scholar
  43. 43.
    C. Lemaréchal and C. Sagastizábal. Variable metric bundle methods: from conceptual to implementable forms. Mathematical Programming, 76(3):393–410, 1997.CrossRefMathSciNetGoogle Scholar
  44. 44.
    C. Lemaréchal and J. Zowe. Some remarks on the construction of higher order algorithms in convex optimization. Applied Mathematics and Optimization, 10(1):51–68, 1983.MATHCrossRefMathSciNetGoogle Scholar
  45. 45.
    C. Lemaréchal and J. Zowe. The eclipsing concept to approximate a multi-valued mapping. Optimization, 22(1):3–37, 1991.MATHMathSciNetCrossRefGoogle Scholar
  46. 46.
    L. Lovász. On the Shannon capacity of a graph. IEEE Transactions on Information Theory, IT 25:1–7, 1979.CrossRefGoogle Scholar
  47. 47.
    D.G. Luenberger. Optimization by Vector Space Methods. Wiley, New York, 1969.MATHGoogle Scholar
  48. 48.
    L. Luksan and J. Vlcek. A bundle-newton method for nonsmooth unconstrained minimization. Mathematical Programming, 83A(3):373–391, 1998.MATHCrossRefMathSciNetGoogle Scholar
  49. 49.
    T.L. Magnanti, J.F. Shapiro, and M.H. Wagner. Generalized linear programming solves the dual. Management Science, 22(11):1195–1203, 1976.MATHCrossRefGoogle Scholar
  50. 50.
    R.E. Marsten, W.W. Hogan, and J.W. Blankenship. The boxstep method for largescale optimization. Operations Research, 23(3):389–405, 1975.MATHMathSciNetGoogle Scholar
  51. 51.
    R. Mifflin. A quasi-second-order proximal bundle algorithm. Mathematical Programming, 73(1):51–72, 1996.CrossRefMathSciNetGoogle Scholar
  52. 52.
    R. Mifflin and C. Sagastizábal. On VU-theory for functions with primal-dual gradient structure. SIAM Journal on Optimization, 11(2):547–571, 2000.MATHCrossRefMathSciNetGoogle Scholar
  53. 53.
    R. Mifflin, D.F. Sun, and L.Q. Qi. Quasi-Newton bundle-type methods for nondifferentiable convex optimization. SIAM Journal on Optimization, 8(2):583–603, 1998.MATHCrossRefMathSciNetGoogle Scholar
  54. 54.
    A.S. Nemirovskii and D. Yudin. Informational complexity and efficient methods for the solution of convex extremal problems. Ékonomika i Mathematicheskie Metody, 12:357–369, 1976. (in Russian. English translation: Matekon, 13, 3-25).Google Scholar
  55. 55.
    Yu.E. Nesterov. Complexity estimates of some cutting plane methods based on the analytic barrier. Mathematical Programming, 69(1):149–176, 1995.CrossRefMathSciNetGoogle Scholar
  56. 56.
    Yu.E. Nesterov and A.S. Nemirovskii. Interior-Point Polynomial Algorithms in Convex Programming. Number 13 in SIAM Studies in Applied Mathematics. SIAM, Philadelphia, 1994.Google Scholar
  57. 57.
    Yu.E. Nesterov and A.S. Nemirovskii. Interior-Point Polynomial Algorithms in Convex Programming. SIAM Studies in Applied Mathematics 13. SIAM, Philadelphia, 1994.Google Scholar
  58. 58.
    J. Nocedal and S.J. Wright. Numerical Optimization. Springer Verlag, New York, 1999.MATHCrossRefGoogle Scholar
  59. 59.
    F. Oustry. A second-order bundle method to minimize the maximum eigenvalue function. Mathematical Programming, 89(1):1–33, 2000.MATHCrossRefMathSciNetGoogle Scholar
  60. 60.
    S. Poljak, F. Rendl, and H. Wolkowicz. A recipe for semidefinite relaxation for (0,1)-quadratic programming. Journal of Global Optimization, 7:51–73, 1995.MATHCrossRefMathSciNetGoogle Scholar
  61. 61.
    B.T. Polyak. A general method for solving extremum problems. Soviet Mathematics Doklady, 8:593–597, 1967.MATHGoogle Scholar
  62. 62.
    B.N. Pshenichnyi. The Linearization Method for Constrained Optimization. Springer Verlag, 1994.Google Scholar
  63. 63.
    C.R. Reeves. Modern Heuristic Techniques for Combinatorial Problems. Blackwell Scientific Publications, New York, 1993.MATHGoogle Scholar
  64. 64.
    N. Shor. Utilization of the operation of space dilatation in the minimization of convex functions. Cybernetics, 6(1):7–15, 1970.CrossRefMathSciNetGoogle Scholar
  65. 65.
    N.Z. Shor. Minimization methods for non-differentiable functions. Springer Verlag, Berlin, 1985.MATHGoogle Scholar
  66. 66.
    N.Z. Shor and A.S. Davydov. Method of opbtaining estimates in quadratic extremal probems with boolean variables. Cybernetics, 21(2):207–211, 1985.MATHCrossRefMathSciNetGoogle Scholar
  67. 67.
    N.Z. Shor and N.G. Zhurbenko. A method for minimization, using the spacedilation operation in the direction of difference between two gradient sequences. Cybernetics, 7(3):450–459, 1971.CrossRefGoogle Scholar
  68. 68.
    V.N. Solov'ev. The subdifferential and the directional derivatives of the maximum of a family of convex functions. Izvestiya: Mathematics, 62(4):807–832, 1998.MATHCrossRefMathSciNetGoogle Scholar
  69. 69.
    G. Sonnevend. An “analytical centre” for polyhedra and new classes of global algorithms for linear (smooth, convex) programming. In A. Prékopa, J. Szelezsan, and B. Strazicky, editors, Proc. 12th IFIP Conf. System Modelling and Optimization, L.N. in Control and Information Sciences, pages 866–875. Springer Verlag, Berlin, 1986.Google Scholar
  70. 70.
    D.M. Topkis. A note on cutting-plane methods without nested constraint sets. Operations Research, 18:1216–1224, 1970.MATHMathSciNetGoogle Scholar
  71. 71.
    T. Terlaky. On lp programming. European Journal of Operational Research, 22:70–100, 1985.MATHCrossRefMathSciNetGoogle Scholar
  72. 72.
    H. Uzawa. Iterative methods for concave programming. In K. Arrow, L. Hurwicz, and H. Uzawa, editors, Studies in Linear and Nonlinear Programming, pages 154–165. Stanford University Press, 1959.Google Scholar
  73. 73.
    L. Vandenberghe and S. Boyd. Semidefinite programming. SIAM Review, 38(1):49–95, 1996.MATHCrossRefMathSciNetGoogle Scholar
  74. 74.
    P. Wolfe. A method of conjugate subgradients for minimizing nondifferentiable functions. Mathematical Programming Study, 3:145–173, 1975.MathSciNetGoogle Scholar
  75. 75.
    L.A. Wolsey. Integer Programming. Wiley-Interscience, 1998.Google Scholar
  76. 76.
    S.J. Wright. Primal-Dual Interior-Point Methods. SIAM Publication, Philadelphia, 1997.MATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2001

Authors and Affiliations

  • Claude Lemaréchal
    • 1
  1. 1.InriaMontbonnotFrance

Personalised recommendations