Skip to main content
SpringerLink
Log in

Variations and extension of the convex–concave procedure

Optimization and Engineering volume 17, pages 263–287 (2016)Cite this article

Abstract

We investigate the convex–concave procedure, a local heuristic that utilizes the tools of convex optimization to find local optima of difference of convex (DC) programming problems. The class of DC problems includes many difficult problems such as the traveling salesman problem. We extend the standard procedure in two major ways and describe several variations. First, we allow for the algorithm to be initialized without a feasible point. Second, we generalize the algorithm to include vector inequalities. We then present several examples to demonstrate these algorithms.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

References

  • Absil PA, Mahony R, Sepulchre R (2009) Optimization algorithms on matrix manifolds. Princeton University Press, Princeton

    MATH  Google Scholar 

  • Agin N (1966) Optimum seeking with branch and bound. Manag Sci 13:176–185

    Article  Google Scholar 

  • Beck A, Ben-Tal A, Tetrushvili L (2010) A sequential parametric convex approximation method with applications to nonconvex truss topology design problems. J Glob Optim 47(1):29–51

    Article  MathSciNet  MATH  Google Scholar 

  • Belkin M, Niyogi P (2003) Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput 15(6):1373–1396

    Article  MATH  Google Scholar 

  • Boggs PT, Tolle JW (1995) Sequential quadratic programming. Acta Numer 4(1):1–51

    Article  MathSciNet  MATH  Google Scholar 

  • Boyd S, Vandenberghe L (2004) Convex Optim. Cambridge University Press, Cambridge

    Book  MATH  Google Scholar 

  • Boyd S, Hast M, Åström KJ (2015) MIMO PID tuning via iterated LMI restriction. Int J Robust Nonlinear Control To appear

  • Byrd RH, Gilbert JC, Nocedal J (2000) A trust region method based on interior point techniques for nonlinear programming. Math Program 89(1):149–185

    Article  MathSciNet  MATH  Google Scholar 

  • Byrne C (2000) Block-iterative interior point optimization methods for image reconstruction from limited data. Inverse Probl 16(5):1405

    Article  MathSciNet  MATH  Google Scholar 

  • Crawford JM, Auton LD (1996) Experimental results on the crossover point in random 3-SAT. Artif Intell 81(1):31–57

    Article  MathSciNet  Google Scholar 

  • CVX Research I (2012) CVX: MATLAB software for disciplined convex programming, version 2.0. http://cvxr.com.cvx

  • De Leeuw J (1977) Applications of convex analysis to multidimensional scaling. In: Barra JR, Brodeau F, Romier G, Van Cutsem B (eds) Recent developments in statistics. North Holland Publishing, Amsterdam, pp 133–146

    Google Scholar 

  • Dempster AP, Laird NM, Rubin DB (1977) Maximum likelihood from incomplete data via the EM algorithm. J R Stat Soc Ser B (Methodol) 39:1–38

    MathSciNet  MATH  Google Scholar 

  • Di Pillo G, Grippo L (1989) Exact penalty functions in constrained optimization. SIAM J Control Optim 27(6):1333–1360

    Article  MathSciNet  MATH  Google Scholar 

  • Diamond S, Boyd S (2015) CVXPY: a python-embedded modeling language for convex optimization. J Mach Learn Res Mach Learn Open Sour Softw To appear

  • Diamond S, Chu E, Boyd S (2014) CVXPY: a python-embedded modeling language for convex optimization, version 0.2. http://cvxpy.org/

  • Domahidi A, Chu E, Boyd S (2013) ECOS: an SOCP solver for embedded systems. In: European control conference, pp 3071–3076

  • Drira A, Pierreval H, Hajri-Gabouh S (2007) Facility layout problems: a survey. Annu Rev Control 31(2):255–267

    Article  Google Scholar 

  • du Merle O, Villeneuve D, Desrosiers J, Hansen P (1999) Stabilized column generation. Discret Math 194(1):229–237

    Article  MathSciNet  MATH  Google Scholar 

  • Edelman A, Tomás AA, Smith TS (1998) The geometry of algorithms with orthogonality constraints. SIAM J Matrix Anal Appl 20(2):303–353

    Article  MathSciNet  MATH  Google Scholar 

  • Elzinga J, Moore TJ (1975) A central cutting plane algorithm for convex programming problems. Math Program 8:134–145

    Article  MathSciNet  MATH  Google Scholar 

  • Falk JE, Hoffmann KR (1976) A successive underestimation method for concave minimization problems. Math Oper Res 1(3):251–259

    Article  MATH  Google Scholar 

  • Falk JE, Soland RM (1969) An algorithm for separable nonconvex programming problems. Manag Sci 15(9):550–569

    Article  MathSciNet  MATH  Google Scholar 

  • Fardad M, Jovanović MR (2014) On the design of optimal structured and sparse feedback gains via sequential convex programming. In: American control conference, pp 2426–2431

  • Garcia Palomares UM, Mangasarian OL (1976) Superlinearly convergent quasi-newton algorithms for nonlinearly constrained optimization problems. Math Program 11(1):1–13

    Article  MathSciNet  MATH  Google Scholar 

  • Gill PE, Wong E (2012) Sequential quadratic programming methods. In: Mixed integer nonlinear programming, Springer, pp 147–224

  • Grant M, Boyd S (2008) Graph implementation for nonsmooth convex programs. In: Blondel V, Boyd S, Kimura H (eds) Recent Advances in learning and control, lecture notes in control and information sciences, Springer. http://stanford.edu/ boyd/graph_dcp.html

  • Han SP, Mangasarian OL (1979) Exact penalty functions in nonlinear programming. Math Program 17(1):251–269

    Article  MathSciNet  MATH  Google Scholar 

  • Hanan M, Kurtzberg J (1972) Placement techniques. In: Breuer MA (ed) Design automation of digital systems, vol 1. Prentice-Hall, Upper Saddle River, pp 213–282

    Google Scholar 

  • Hartman P (1959) On functions representable as a difference of convex functions. Pac J Math 9(3):707–713

    Article  MathSciNet  MATH  Google Scholar 

  • Hillestad RJ (1975) Optimization problems subject to a budget constraint with economies of scale. Oper Res 23(6):1091–1098

    Article  MathSciNet  MATH  Google Scholar 

  • Hillestad RJ, Jacobsen SE (1980a) Linear programs with an additional reverse convex constraint. Appl Math Optim 6(1):257–269

    Article  MathSciNet  MATH  Google Scholar 

  • Hillestad RJ, Jacobsen SE (1980b) Reverse convex programming. Appl Math Optim 6(1):63–78

    Article  MathSciNet  MATH  Google Scholar 

  • Horst R (1986) A general class of branch-and-bound methods in global optimization with some new approaches for concave minimization. J Optim Theory Appl 51(2):271–291

    Article  MathSciNet  MATH  Google Scholar 

  • Horst R, Thoai NV (1999) DC programming: overview. J Optim Theory Appl 103(1):1–43

    Article  MathSciNet  MATH  Google Scholar 

  • Horst R, Tuy H (1996) Global Optimization, 3rd edn. Springer, New York

    Book  MATH  Google Scholar 

  • Horst R, Phong TQ, Thoai NV, De Vries J (1991a) On solving a DC programming problem by a sequence of linear programs. J Glob Optim 1(2):183–203

    Article  MATH  Google Scholar 

  • Horst R, Thoai NV, Benson HP (1991b) Concave minimization via conical partitions and polyhedral outer approximation. Math Program 50:259–274

    Article  MathSciNet  MATH  Google Scholar 

  • Horst R, Pardalos PM, Thoai NV (1995) Introduction to global optimization. Kluwer Academic Publishers, Dordrecht

    MATH  Google Scholar 

  • Karp RM (1972) Reducibility among combinatorial problems. In: Thatcher JW, Miller RE (eds) Complexity of computer computation. Plenum, Berlin, pp 85–104

    Chapter  Google Scholar 

  • Kelly JE Jr (1960) The cutting-plane method for solving convex programs. J Soc Ind Appl Math 8(4):703–712

    Article  MathSciNet  Google Scholar 

  • Lanckreit GR, Sriperumbudur BK (2009) On the convergence of the concave-convex procedure. Adv Neural Inf Process Syst, 1759–1767

  • Lange K (2004) Optimization. Springer texts in statistics. Springer, New York

    Google Scholar 

  • Lange K, Hunter DR, Yang I (2000) Optimization transfer using surrogate objective functions. J Comput Graph Stat 9(1):1–20

    MathSciNet  Google Scholar 

  • Lawler EL, Wood DE (1966) Branch-and-bound methods: a survey. Oper Res 14:699–719

    Article  MathSciNet  MATH  Google Scholar 

  • Le HM, Le Thi HA, Pham Dinh T, Bouvry P (2010) A combined DCA: GA for constructing highly nonlinear balanced Boolean functions in cryptography. J Glob Optim 47(4):597–613

    Article  MathSciNet  MATH  Google Scholar 

  • Le Thi HA (2003) Solving large scale molecular distance geometry problems by a smoothing technique via the Gaussian transform and DC programming. J Glob Optim 27(4):375–397

    Article  MathSciNet  MATH  Google Scholar 

  • Le Thi HA (2015) DC programming and DCA: local and global approaches—theory, algorithms and applications. http://lita.sciences.univ-metz.fr/lethi/DCA.html

  • Le Thi HA, Pham Dinh T (1997) Solving a class of linearly constrainted indefinite quadratic problems by DC algorithms. J Glob Optim 11(3):253–285

    Article  MATH  Google Scholar 

  • Le Thi HA, Pham Dinh T (2003) Large-scale molecular optimization for distance matrices by a DC optimization approach. SIAM J Optim 14(1):77–114

    Article  MathSciNet  MATH  Google Scholar 

  • Le Thi HA, Pham Dinh T (2008) A continuous approach for the concave cost supply problem via DC programming and DCA. Discret Appl Math 156(3):325–338

    Article  MathSciNet  MATH  Google Scholar 

  • Le Thi HA, Pham Dinh T, Muu LD (1996) Numerical solution for optimization over the efficient set by d.c. optimization algorithms. Oper Res Lett 19:117–128

    Article  MathSciNet  MATH  Google Scholar 

  • Le Thi HA, Pham Dinh T, Muu LD (1998) A combined D.C. optimization-ellipsoidal branch-and-bound algorithm for solving nonconvex quadratic programming problems. J Comb Optim 2(1):9–28

    Article  MathSciNet  MATH  Google Scholar 

  • Le Thi HA, Pham Dinh T, Thoai NV (2002) Combination between global and local methods for solving an optimization problem over the efficient set. Eur J Oper Res 142(2):258–270

    Article  MathSciNet  MATH  Google Scholar 

  • Le Thi HA, Moeini M, Pham Dinh T (2009a) DC programming approach for portfolio optimization under step increasing transaction costs. Optimization 58(3):267–289

    Article  MathSciNet  MATH  Google Scholar 

  • Le Thi HA, Pham Dinh T, Nguyen CN, Thoai NV (2009b) DC programming techniques for solving a class of nonlinear bilevel programs. J Glob Optim 44(3):313–337

    Article  MathSciNet  MATH  Google Scholar 

  • Le Thi HA, Pham Dinh T, Ngai HV (2012) Exact penalty and error bounds in DC programming. J Glob Optim 52(3):509–535

    Article  MathSciNet  MATH  Google Scholar 

  • Le Thi HA, Huynh VN, Pham Dinh T (2014) DC programming and DCA for general DC programs. Adv Comput Methods Knowl Eng, 15–35

  • Lipp T, Boyd S (2014) MATLAB and python examples of convex-concave procedure. http://www.stanford.edu/ boyd/software/cvx_ccv_examples/

  • Little RJA, Rubin DB (1987) Statistical analysis with missing data. Wiley, New York

    MATH  Google Scholar 

  • Maranas CD, Floudas CA (1997) Global optimization in generalized geometric programming. Comput Chem Eng 21(4):351–369

    Article  MathSciNet  Google Scholar 

  • McLachlan G, Krishnan T (2007) The EM algorithm and extensions. Wiley, Hoboken

    MATH  Google Scholar 

  • Meyer R (1970) The validity of a family of optimization methods. SIAM J Control 8(1):41–54

    Article  MathSciNet  MATH  Google Scholar 

  • Mitchell D, Selman B, Levesque H (1992) Hard and easy distributions and SAT problems. Proc Tenth Natl Conf Artif Intell 92:459–465

    Google Scholar 

  • MOSEK ApS (2013) MOSEK version 7.0. http://www.mosek.com

  • Mueller JB, Larsson R (2008) Collision avoidance maneuver planning with robust optimization. In: International ESA conference on guidance, navigation and control systems, Tralee

  • Mutapcic A, Boyd S (2009) Cutting-set methods for robust convex optimization with pessimizing oracles. Optim Methods Softw 24(3):381–406

    Article  MathSciNet  MATH  Google Scholar 

  • Muu LD (1985) A convergent algorithm for solving linear programs with an additional reverse convex constraint. Kybernetika 21(6):428–435

    MathSciNet  MATH  Google Scholar 

  • Naghsh MM, Modarres-Hashemi M, ShahbazPanahi S, Soltanalian M, Stoica P (2013) Unified optimization framework for multi-static radar code design using information-theoretic criteria. IEEE Trans Signal Process 61(21):5401–5416

    Article  Google Scholar 

  • Nam GJ, Cong J (eds) (2007) Modern circuit placement. Springer, New York

    Google Scholar 

  • Ndiaye BM, Pham Dinh T, Le Thi HA (2012) DC programming and DCA for large-scale two-dimensional packing problems. Springer, New York

    Book  Google Scholar 

  • Niu YS, Pham Dinh T (2014) DC programming approaches for BMI and QMI feasibility problems. Adv Comput Methods Knowl Eng 6:37–63

    Article  MATH  Google Scholar 

  • Nocedal J, Wright SJ (2006) Numerical optimization. Springer, New York

    MATH  Google Scholar 

  • Pham Dinh T, Le Thi HA (1997) Convex analysis approach to DC programming: theory, algorithms, and applications. Acta Math Vietnam 22(1):289–355

    MathSciNet  MATH  Google Scholar 

  • Pham Dinh T, Le Thi HA (1998) A DC optimization algorithm for solving the trust-region subproblem. SIAM J Optim 8(2):476–505

    Article  MathSciNet  MATH  Google Scholar 

  • Pham Dinh T, Le Thi HA (2014) Recent advances in DC programming and DCA. Trans Comput Intell XIII 13:1–37

    Article  Google Scholar 

  • Pham Dinh T, Souad EB (1986) Algorithms for solving a class of nonconvex optimization problems. Methods of subgradients. In: Hiriart-Urruty JB (ed) FERMAT Days 85: mathematics for optimization. Elsevier Scince Publishers B. V., Amsterdam, pp 249–271

    Google Scholar 

  • Pham Dinh T, Souad EB (1988) Duality in D.C. (difference of convex functions) optimization. subgradient methods. In: International Series of Numerical Mathematics, vol 84, Birkhäuser Basel

  • Pham Dinh T, Le Thi HA, Akoa F (2008) Combining DCA (DC algorithms) and interior point techniques for large-scale nonconvex quadratic programming. Optim Methods Softw 23(4):609–629

    Article  MathSciNet  MATH  Google Scholar 

  • Pham Dinh T, Nguyen CN, Le Thi HA (2010) An efficient combined DCA and B&B using DC/SDP relaxation for globally solving binary quadratic programs. J Glob Optim 48(4):595–632

    Article  MathSciNet  MATH  Google Scholar 

  • Powell MJD (1978) A fast algorithm for nonlinearly constrained optimization calculations. Springer, New York, pp 144–157

    MATH  Google Scholar 

  • Robinson SM (1972) A quadratically-convergent algorithm for general nonlinear programming problems. Math Program 3(1):145–156

    Article  MathSciNet  MATH  Google Scholar 

  • Shulman J, Ho J, Lee A, Awwal I, Bradlow H, Abbeel P (2013) Finding locally optimal, collision-free trajectories with sequential convex optimization. In: Robotics: science and systems, vol 9. pp 1–10

  • Singh SP, Sharma RRK (2006) A review of different approaches to the facility layout problems. Int J Adv Manuf Technol 30(5):425–433

    Article  Google Scholar 

  • Smola AJ, Vishwanathan SVN, Hofmann T (2005) Kernel methods for missing variables. In: Proceedings of 10th international workshop on artificial intelligence and statistics

  • Soland RM (1971) An algorithm for separable nonconvex programming problems II: nonconvex constraints. Manag Sci 17(11):759–773

    Article  MathSciNet  MATH  Google Scholar 

  • Specht E (2013) Packomania. http://www.packomania.com/

  • Stiefel E (1935–1936) Richtungsfelder und fernparallelismus in n-dimensionalem mannig faltigkeiten. Comment Math Helv 8:305–353

  • Stoica P, Babu P (2012) SPICE and LIKES: two hyperparameter-free methods for sparse-parameter estimation. Signal Process 92(8):1580–1590

    Article  Google Scholar 

  • Svanberg K (1987) The method of moving asymptotes-a new method for structural optimization. Int J Numer Methods Eng 24(2):359–373

    Article  MathSciNet  MATH  Google Scholar 

  • Thoai NV, Tuy H (1980) Convergent algorithms for minimizing a concave function. Math Oper Res 5(4):556–566

    Article  MathSciNet  MATH  Google Scholar 

  • Toh KC, Todd MJ, Tutuncu RH (1999) SDPT3—a MATLAB software package for semidefinite programming. Optim Methods Softw 11:545–581

    Article  MathSciNet  MATH  Google Scholar 

  • Tutuncu RH, Toh KC, Todd MJ (2003) Solving semidfinite-quadratic-linear programs using SDPT3. Math Program 95(2):189–217

    Article  MathSciNet  MATH  Google Scholar 

  • Tuy H (1983) On outer approximation methods for solving concave minimization problems. Acta Math Vietnam 8(2):3–34

    MathSciNet  MATH  Google Scholar 

  • Tuy H (1986) A general deterministic approach to global optimization via D.C. programming. N-Holl Math Stud 129:273–303

    Article  MathSciNet  MATH  Google Scholar 

  • Tuy H, Horst R (1988) Convergence and restart in branch-and-bound algorithms for global optimization. Applications to concave minimization and DC optimization problems. Math Program 41(2):161–183

    Article  MathSciNet  MATH  Google Scholar 

  • Wilson RB (1963) A simplicial algorithm for concave programming. PhD thesis, Gradaute School of Business Administration, Harvard University

  • Yamada S, Tanino T, Inuiguchi M (2000) Inner approximation method for a reverse convex programming problem. J Optim Theory Appl 107(2):355–389

    Article  MathSciNet  MATH  Google Scholar 

  • Yu CNJ, Joachims T (2009) Learning structural SVMs with latent variables. In: Proceedings of the 26th annual international conference on machine learning, pp 1169–1176

  • Yuille AL, Rangarajan A (2003) The concave-convex procedure. Neural Comput 15(4):915–936

    Article  MATH  Google Scholar 

  • Zangwill WI (1969) Nonlinear programming: a unified approach. Prentice-Hall Inc, Englewood Cliffs

    MATH  Google Scholar 

  • Zillober C (2001) Global convergence of a nonlinear programming method using convex optimization. Numer Algorithms 27(3):265–289

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgments

We would like to thank our many reviewers for their comments which improved this paper and in particular for highlighting much of the recent work in DCA. This research was made possible by the National Science Foundation Graduate Research Fellowship, Grant DGE-1147470 and by the Cleve B. Moler Stanford Graduate Fellowship.

Author information

Authors and Affiliations

  1. Stanford University, Packard Building, Room 243, 350 Serra Mall, Stanford, CA, 94305, USA

    Thomas Lipp

  2. Stanford University, Packard Building, Room 254, 350 Serra Mall, Stanford, CA, 94305, USA

    Stephen Boyd

Authors
  1. Thomas Lipp
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stephen Boyd
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Thomas Lipp.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (zip 127 KB)

Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lipp, T., Boyd, S. Variations and extension of the convex–concave procedure. Optim Eng 17, 263–287 (2016). https://doi.org/10.1007/s11081-015-9294-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11081-015-9294-x

Keywords

search

Navigation