Advertisement

Optimization and Engineering

, Volume 17, Issue 2, pp 263–287 | Cite as

Variations and extension of the convex–concave procedure

  • Thomas LippEmail author
  • Stephen Boyd
Article
First Online:

Abstract

We investigate the convex–concave procedure, a local heuristic that utilizes the tools of convex optimization to find local optima of difference of convex (DC) programming problems. The class of DC problems includes many difficult problems such as the traveling salesman problem. We extend the standard procedure in two major ways and describe several variations. First, we allow for the algorithm to be initialized without a feasible point. Second, we generalize the algorithm to include vector inequalities. We then present several examples to demonstrate these algorithms.

Keywords

Convex optimization Convex concave procedure Sequential optimization Difference of convex programming 
This is a preview of subscription content, log in to check access.

Notes

Acknowledgments

We would like to thank our many reviewers for their comments which improved this paper and in particular for highlighting much of the recent work in DCA. This research was made possible by the National Science Foundation Graduate Research Fellowship, Grant DGE-1147470 and by the Cleve B. Moler Stanford Graduate Fellowship.

Supplementary material

11081_2015_9294_MOESM1_ESM.zip (127 kb)
Supplementary material 1 (zip 127 KB)

References

  1. Absil PA, Mahony R, Sepulchre R (2009) Optimization algorithms on matrix manifolds. Princeton University Press, PrincetonzbMATHGoogle Scholar
  2. Agin N (1966) Optimum seeking with branch and bound. Manag Sci 13:176–185CrossRefGoogle Scholar
  3. Beck A, Ben-Tal A, Tetrushvili L (2010) A sequential parametric convex approximation method with applications to nonconvex truss topology design problems. J Glob Optim 47(1):29–51MathSciNetzbMATHCrossRefGoogle Scholar
  4. Belkin M, Niyogi P (2003) Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput 15(6):1373–1396zbMATHCrossRefGoogle Scholar
  5. Boggs PT, Tolle JW (1995) Sequential quadratic programming. Acta Numer 4(1):1–51MathSciNetzbMATHCrossRefGoogle Scholar
  6. Boyd S, Vandenberghe L (2004) Convex Optim. Cambridge University Press, CambridgezbMATHCrossRefGoogle Scholar
  7. Boyd S, Hast M, Åström KJ (2015) MIMO PID tuning via iterated LMI restriction. Int J Robust Nonlinear Control To appearGoogle Scholar
  8. Byrd RH, Gilbert JC, Nocedal J (2000) A trust region method based on interior point techniques for nonlinear programming. Math Program 89(1):149–185MathSciNetzbMATHCrossRefGoogle Scholar
  9. Byrne C (2000) Block-iterative interior point optimization methods for image reconstruction from limited data. Inverse Probl 16(5):1405MathSciNetzbMATHCrossRefGoogle Scholar
  10. Crawford JM, Auton LD (1996) Experimental results on the crossover point in random 3-SAT. Artif Intell 81(1):31–57MathSciNetCrossRefGoogle Scholar
  11. CVX Research I (2012) CVX: MATLAB software for disciplined convex programming, version 2.0. http://cvxr.com.cvx
  12. De Leeuw J (1977) Applications of convex analysis to multidimensional scaling. In: Barra JR, Brodeau F, Romier G, Van Cutsem B (eds) Recent developments in statistics. North Holland Publishing, Amsterdam, pp 133–146Google Scholar
  13. Dempster AP, Laird NM, Rubin DB (1977) Maximum likelihood from incomplete data via the EM algorithm. J R Stat Soc Ser B (Methodol) 39:1–38MathSciNetzbMATHGoogle Scholar
  14. Di Pillo G, Grippo L (1989) Exact penalty functions in constrained optimization. SIAM J Control Optim 27(6):1333–1360MathSciNetzbMATHCrossRefGoogle Scholar
  15. Diamond S, Boyd S (2015) CVXPY: a python-embedded modeling language for convex optimization. J Mach Learn Res Mach Learn Open Sour Softw To appearGoogle Scholar
  16. Diamond S, Chu E, Boyd S (2014) CVXPY: a python-embedded modeling language for convex optimization, version 0.2. http://cvxpy.org/
  17. Domahidi A, Chu E, Boyd S (2013) ECOS: an SOCP solver for embedded systems. In: European control conference, pp 3071–3076Google Scholar
  18. Drira A, Pierreval H, Hajri-Gabouh S (2007) Facility layout problems: a survey. Annu Rev Control 31(2):255–267CrossRefGoogle Scholar
  19. du Merle O, Villeneuve D, Desrosiers J, Hansen P (1999) Stabilized column generation. Discret Math 194(1):229–237MathSciNetzbMATHCrossRefGoogle Scholar
  20. Edelman A, Tomás AA, Smith TS (1998) The geometry of algorithms with orthogonality constraints. SIAM J Matrix Anal Appl 20(2):303–353MathSciNetzbMATHCrossRefGoogle Scholar
  21. Elzinga J, Moore TJ (1975) A central cutting plane algorithm for convex programming problems. Math Program 8:134–145MathSciNetzbMATHCrossRefGoogle Scholar
  22. Falk JE, Hoffmann KR (1976) A successive underestimation method for concave minimization problems. Math Oper Res 1(3):251–259zbMATHCrossRefGoogle Scholar
  23. Falk JE, Soland RM (1969) An algorithm for separable nonconvex programming problems. Manag Sci 15(9):550–569MathSciNetzbMATHCrossRefGoogle Scholar
  24. Fardad M, Jovanović MR (2014) On the design of optimal structured and sparse feedback gains via sequential convex programming. In: American control conference, pp 2426–2431Google Scholar
  25. Garcia Palomares UM, Mangasarian OL (1976) Superlinearly convergent quasi-newton algorithms for nonlinearly constrained optimization problems. Math Program 11(1):1–13MathSciNetzbMATHCrossRefGoogle Scholar
  26. Gill PE, Wong E (2012) Sequential quadratic programming methods. In: Mixed integer nonlinear programming, Springer, pp 147–224Google Scholar
  27. Grant M, Boyd S (2008) Graph implementation for nonsmooth convex programs. In: Blondel V, Boyd S, Kimura H (eds) Recent Advances in learning and control, lecture notes in control and information sciences, Springer. http://stanford.edu/ boyd/graph_dcp.html
  28. Han SP, Mangasarian OL (1979) Exact penalty functions in nonlinear programming. Math Program 17(1):251–269MathSciNetzbMATHCrossRefGoogle Scholar
  29. Hanan M, Kurtzberg J (1972) Placement techniques. In: Breuer MA (ed) Design automation of digital systems, vol 1. Prentice-Hall, Upper Saddle River, pp 213–282Google Scholar
  30. Hartman P (1959) On functions representable as a difference of convex functions. Pac J Math 9(3):707–713MathSciNetzbMATHCrossRefGoogle Scholar
  31. Hillestad RJ (1975) Optimization problems subject to a budget constraint with economies of scale. Oper Res 23(6):1091–1098MathSciNetzbMATHCrossRefGoogle Scholar
  32. Hillestad RJ, Jacobsen SE (1980a) Linear programs with an additional reverse convex constraint. Appl Math Optim 6(1):257–269MathSciNetzbMATHCrossRefGoogle Scholar
  33. Hillestad RJ, Jacobsen SE (1980b) Reverse convex programming. Appl Math Optim 6(1):63–78MathSciNetzbMATHCrossRefGoogle Scholar
  34. Horst R (1986) A general class of branch-and-bound methods in global optimization with some new approaches for concave minimization. J Optim Theory Appl 51(2):271–291MathSciNetzbMATHCrossRefGoogle Scholar
  35. Horst R, Thoai NV (1999) DC programming: overview. J Optim Theory Appl 103(1):1–43MathSciNetzbMATHCrossRefGoogle Scholar
  36. Horst R, Tuy H (1996) Global Optimization, 3rd edn. Springer, New YorkzbMATHCrossRefGoogle Scholar
  37. Horst R, Phong TQ, Thoai NV, De Vries J (1991a) On solving a DC programming problem by a sequence of linear programs. J Glob Optim 1(2):183–203zbMATHCrossRefGoogle Scholar
  38. Horst R, Thoai NV, Benson HP (1991b) Concave minimization via conical partitions and polyhedral outer approximation. Math Program 50:259–274MathSciNetzbMATHCrossRefGoogle Scholar
  39. Horst R, Pardalos PM, Thoai NV (1995) Introduction to global optimization. Kluwer Academic Publishers, DordrechtzbMATHGoogle Scholar
  40. Karp RM (1972) Reducibility among combinatorial problems. In: Thatcher JW, Miller RE (eds) Complexity of computer computation. Plenum, Berlin, pp 85–104CrossRefGoogle Scholar
  41. Kelly JE Jr (1960) The cutting-plane method for solving convex programs. J Soc Ind Appl Math 8(4):703–712MathSciNetCrossRefGoogle Scholar
  42. Lanckreit GR, Sriperumbudur BK (2009) On the convergence of the concave-convex procedure. Adv Neural Inf Process Syst, 1759–1767Google Scholar
  43. Lange K (2004) Optimization. Springer texts in statistics. Springer, New YorkGoogle Scholar
  44. Lange K, Hunter DR, Yang I (2000) Optimization transfer using surrogate objective functions. J Comput Graph Stat 9(1):1–20MathSciNetGoogle Scholar
  45. Lawler EL, Wood DE (1966) Branch-and-bound methods: a survey. Oper Res 14:699–719MathSciNetzbMATHCrossRefGoogle Scholar
  46. Le HM, Le Thi HA, Pham Dinh T, Bouvry P (2010) A combined DCA: GA for constructing highly nonlinear balanced Boolean functions in cryptography. J Glob Optim 47(4):597–613MathSciNetzbMATHCrossRefGoogle Scholar
  47. Le Thi HA (2003) Solving large scale molecular distance geometry problems by a smoothing technique via the Gaussian transform and DC programming. J Glob Optim 27(4):375–397MathSciNetzbMATHCrossRefGoogle Scholar
  48. Le Thi HA (2015) DC programming and DCA: local and global approaches—theory, algorithms and applications. http://lita.sciences.univ-metz.fr/lethi/DCA.html
  49. Le Thi HA, Pham Dinh T (1997) Solving a class of linearly constrainted indefinite quadratic problems by DC algorithms. J Glob Optim 11(3):253–285zbMATHCrossRefGoogle Scholar
  50. Le Thi HA, Pham Dinh T (2003) Large-scale molecular optimization for distance matrices by a DC optimization approach. SIAM J Optim 14(1):77–114MathSciNetzbMATHCrossRefGoogle Scholar
  51. Le Thi HA, Pham Dinh T (2008) A continuous approach for the concave cost supply problem via DC programming and DCA. Discret Appl Math 156(3):325–338MathSciNetzbMATHCrossRefGoogle Scholar
  52. Le Thi HA, Pham Dinh T, Muu LD (1996) Numerical solution for optimization over the efficient set by d.c. optimization algorithms. Oper Res Lett 19:117–128MathSciNetzbMATHCrossRefGoogle Scholar
  53. Le Thi HA, Pham Dinh T, Muu LD (1998) A combined D.C. optimization-ellipsoidal branch-and-bound algorithm for solving nonconvex quadratic programming problems. J Comb Optim 2(1):9–28MathSciNetzbMATHCrossRefGoogle Scholar
  54. Le Thi HA, Pham Dinh T, Thoai NV (2002) Combination between global and local methods for solving an optimization problem over the efficient set. Eur J Oper Res 142(2):258–270MathSciNetzbMATHCrossRefGoogle Scholar
  55. Le Thi HA, Moeini M, Pham Dinh T (2009a) DC programming approach for portfolio optimization under step increasing transaction costs. Optimization 58(3):267–289MathSciNetzbMATHCrossRefGoogle Scholar
  56. Le Thi HA, Pham Dinh T, Nguyen CN, Thoai NV (2009b) DC programming techniques for solving a class of nonlinear bilevel programs. J Glob Optim 44(3):313–337MathSciNetzbMATHCrossRefGoogle Scholar
  57. Le Thi HA, Pham Dinh T, Ngai HV (2012) Exact penalty and error bounds in DC programming. J Glob Optim 52(3):509–535MathSciNetzbMATHCrossRefGoogle Scholar
  58. Le Thi HA, Huynh VN, Pham Dinh T (2014) DC programming and DCA for general DC programs. Adv Comput Methods Knowl Eng, 15–35Google Scholar
  59. Lipp T, Boyd S (2014) MATLAB and python examples of convex-concave procedure. http://www.stanford.edu/ boyd/software/cvx_ccv_examples/
  60. Little RJA, Rubin DB (1987) Statistical analysis with missing data. Wiley, New YorkzbMATHGoogle Scholar
  61. Maranas CD, Floudas CA (1997) Global optimization in generalized geometric programming. Comput Chem Eng 21(4):351–369MathSciNetCrossRefGoogle Scholar
  62. McLachlan G, Krishnan T (2007) The EM algorithm and extensions. Wiley, HobokenzbMATHGoogle Scholar
  63. Meyer R (1970) The validity of a family of optimization methods. SIAM J Control 8(1):41–54MathSciNetzbMATHCrossRefGoogle Scholar
  64. Mitchell D, Selman B, Levesque H (1992) Hard and easy distributions and SAT problems. Proc Tenth Natl Conf Artif Intell 92:459–465Google Scholar
  65. MOSEK ApS (2013) MOSEK version 7.0. http://www.mosek.com
  66. Mueller JB, Larsson R (2008) Collision avoidance maneuver planning with robust optimization. In: International ESA conference on guidance, navigation and control systems, TraleeGoogle Scholar
  67. Mutapcic A, Boyd S (2009) Cutting-set methods for robust convex optimization with pessimizing oracles. Optim Methods Softw 24(3):381–406MathSciNetzbMATHCrossRefGoogle Scholar
  68. Muu LD (1985) A convergent algorithm for solving linear programs with an additional reverse convex constraint. Kybernetika 21(6):428–435MathSciNetzbMATHGoogle Scholar
  69. Naghsh MM, Modarres-Hashemi M, ShahbazPanahi S, Soltanalian M, Stoica P (2013) Unified optimization framework for multi-static radar code design using information-theoretic criteria. IEEE Trans Signal Process 61(21):5401–5416CrossRefGoogle Scholar
  70. Nam GJ, Cong J (eds) (2007) Modern circuit placement. Springer, New YorkGoogle Scholar
  71. Ndiaye BM, Pham Dinh T, Le Thi HA (2012) DC programming and DCA for large-scale two-dimensional packing problems. Springer, New YorkCrossRefGoogle Scholar
  72. Niu YS, Pham Dinh T (2014) DC programming approaches for BMI and QMI feasibility problems. Adv Comput Methods Knowl Eng 6:37–63zbMATHCrossRefGoogle Scholar
  73. Nocedal J, Wright SJ (2006) Numerical optimization. Springer, New YorkzbMATHGoogle Scholar
  74. Pham Dinh T, Le Thi HA (1997) Convex analysis approach to DC programming: theory, algorithms, and applications. Acta Math Vietnam 22(1):289–355MathSciNetzbMATHGoogle Scholar
  75. Pham Dinh T, Le Thi HA (1998) A DC optimization algorithm for solving the trust-region subproblem. SIAM J Optim 8(2):476–505MathSciNetzbMATHCrossRefGoogle Scholar
  76. Pham Dinh T, Le Thi HA (2014) Recent advances in DC programming and DCA. Trans Comput Intell XIII 13:1–37CrossRefGoogle Scholar
  77. Pham Dinh T, Souad EB (1986) Algorithms for solving a class of nonconvex optimization problems. Methods of subgradients. In: Hiriart-Urruty JB (ed) FERMAT Days 85: mathematics for optimization. Elsevier Scince Publishers B. V., Amsterdam, pp 249–271Google Scholar
  78. Pham Dinh T, Souad EB (1988) Duality in D.C. (difference of convex functions) optimization. subgradient methods. In: International Series of Numerical Mathematics, vol 84, Birkhäuser BaselGoogle Scholar
  79. Pham Dinh T, Le Thi HA, Akoa F (2008) Combining DCA (DC algorithms) and interior point techniques for large-scale nonconvex quadratic programming. Optim Methods Softw 23(4):609–629MathSciNetzbMATHCrossRefGoogle Scholar
  80. Pham Dinh T, Nguyen CN, Le Thi HA (2010) An efficient combined DCA and B&B using DC/SDP relaxation for globally solving binary quadratic programs. J Glob Optim 48(4):595–632MathSciNetzbMATHCrossRefGoogle Scholar
  81. Powell MJD (1978) A fast algorithm for nonlinearly constrained optimization calculations. Springer, New York, pp 144–157zbMATHGoogle Scholar
  82. Robinson SM (1972) A quadratically-convergent algorithm for general nonlinear programming problems. Math Program 3(1):145–156MathSciNetzbMATHCrossRefGoogle Scholar
  83. Shulman J, Ho J, Lee A, Awwal I, Bradlow H, Abbeel P (2013) Finding locally optimal, collision-free trajectories with sequential convex optimization. In: Robotics: science and systems, vol 9. pp 1–10Google Scholar
  84. Singh SP, Sharma RRK (2006) A review of different approaches to the facility layout problems. Int J Adv Manuf Technol 30(5):425–433CrossRefGoogle Scholar
  85. Smola AJ, Vishwanathan SVN, Hofmann T (2005) Kernel methods for missing variables. In: Proceedings of 10th international workshop on artificial intelligence and statisticsGoogle Scholar
  86. Soland RM (1971) An algorithm for separable nonconvex programming problems II: nonconvex constraints. Manag Sci 17(11):759–773MathSciNetzbMATHCrossRefGoogle Scholar
  87. Specht E (2013) Packomania. http://www.packomania.com/
  88. Stiefel E (1935–1936) Richtungsfelder und fernparallelismus in n-dimensionalem mannig faltigkeiten. Comment Math Helv 8:305–353Google Scholar
  89. Stoica P, Babu P (2012) SPICE and LIKES: two hyperparameter-free methods for sparse-parameter estimation. Signal Process 92(8):1580–1590CrossRefGoogle Scholar
  90. Svanberg K (1987) The method of moving asymptotes-a new method for structural optimization. Int J Numer Methods Eng 24(2):359–373MathSciNetzbMATHCrossRefGoogle Scholar
  91. Thoai NV, Tuy H (1980) Convergent algorithms for minimizing a concave function. Math Oper Res 5(4):556–566MathSciNetzbMATHCrossRefGoogle Scholar
  92. Toh KC, Todd MJ, Tutuncu RH (1999) SDPT3—a MATLAB software package for semidefinite programming. Optim Methods Softw 11:545–581MathSciNetzbMATHCrossRefGoogle Scholar
  93. Tutuncu RH, Toh KC, Todd MJ (2003) Solving semidfinite-quadratic-linear programs using SDPT3. Math Program 95(2):189–217MathSciNetzbMATHCrossRefGoogle Scholar
  94. Tuy H (1983) On outer approximation methods for solving concave minimization problems. Acta Math Vietnam 8(2):3–34MathSciNetzbMATHGoogle Scholar
  95. Tuy H (1986) A general deterministic approach to global optimization via D.C. programming. N-Holl Math Stud 129:273–303MathSciNetzbMATHCrossRefGoogle Scholar
  96. Tuy H, Horst R (1988) Convergence and restart in branch-and-bound algorithms for global optimization. Applications to concave minimization and DC optimization problems. Math Program 41(2):161–183MathSciNetzbMATHCrossRefGoogle Scholar
  97. Wilson RB (1963) A simplicial algorithm for concave programming. PhD thesis, Gradaute School of Business Administration, Harvard UniversityGoogle Scholar
  98. Yamada S, Tanino T, Inuiguchi M (2000) Inner approximation method for a reverse convex programming problem. J Optim Theory Appl 107(2):355–389MathSciNetzbMATHCrossRefGoogle Scholar
  99. Yu CNJ, Joachims T (2009) Learning structural SVMs with latent variables. In: Proceedings of the 26th annual international conference on machine learning, pp 1169–1176Google Scholar
  100. Yuille AL, Rangarajan A (2003) The concave-convex procedure. Neural Comput 15(4):915–936zbMATHCrossRefGoogle Scholar
  101. Zangwill WI (1969) Nonlinear programming: a unified approach. Prentice-Hall Inc, Englewood CliffszbMATHGoogle Scholar
  102. Zillober C (2001) Global convergence of a nonlinear programming method using convex optimization. Numer Algorithms 27(3):265–289MathSciNetzbMATHCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  1. 1.Stanford UniversityStanfordUSA
  2. 2.Stanford UniversityStanfordUSA

Personalised recommendations

Cite article

Buy options