Mathematical Programming

, Volume 149, Issue 1–2, pp 301–327 | Cite as

Complexity analysis of interior point algorithms for non-Lipschitz and nonconvex minimization

  • Wei Bian
  • Xiaojun ChenEmail author
  • Yinyu Ye
Full Length Paper Series A


We propose a first order interior point algorithm for a class of non-Lipschitz and nonconvex minimization problems with box constraints, which arise from applications in variable selection and regularized optimization. The objective functions of these problems are continuously differentiable typically at interior points of the feasible set. Our first order algorithm is easy to implement and the objective function value is reduced monotonically along the iteration points. We show that the worst-case iteration complexity for finding an \(\epsilon \) scaled first order stationary point is \(O(\epsilon ^{-2})\). Furthermore, we develop a second order interior point algorithm using the Hessian matrix, and solve a quadratic program with a ball constraint at each iteration. Although the second order interior point algorithm costs more computational time than that of the first order algorithm in each iteration, its worst-case iteration complexity for finding an \(\epsilon \) scaled second order stationary point is reduced to \(O(\epsilon ^{-3/2})\). Note that an \(\epsilon \) scaled second order stationary point must also be an \(\epsilon \) scaled first order stationary point.


Constrained non-Lipschitz optimization Complexity analysis  Interior point method First order algorithm Second order algorithm 

Mathematics Subject Classifcation (2010)

90C30 90C26 65K05 49M37 



We would like to thank Prof. Sven Leyffer, the anonymous Associate Editor and referees for their insightful and constructive comments, which help us to enrich the content and improve the presentation of the results in this paper. The authors acknowledge the support of the Department of Applied Mathematics, the Hong Kong Polytechnic University for the visit of Yinyu Ye.


  1. 1.
    Bian, W., Chen, X.: Worst-case complexity of smoothing quadratic regularization methods for non-Lipschitzian optimization. SIAM J. Optim. 28, 1718–1741 (2013)CrossRefMathSciNetGoogle Scholar
  2. 2.
    Bruckstein, A.M., Donoho, D.L., Elad, M.: From sparse solutions of systems of equations to sparse modeling of signals and images. SIAM Rev. 51, 34–81 (2009)CrossRefzbMATHMathSciNetGoogle Scholar
  3. 3.
    Cartis, C., Gould, N.I.M., Toint, PhL: On the evaluation complexity of composite function minimization with applications to nonconvex nonlinear programming. SIAM J. Optim. 21, 1721–1739 (2011)CrossRefzbMATHMathSciNetGoogle Scholar
  4. 4.
    Cartis, C., Gould, N.I.M., Toint, PhL: Adaptive cubic regularisation methods for unconstrained optimization, Part I: motivation, convergence and numerical results. Math. Program. 127, 245–295 (2011)CrossRefzbMATHMathSciNetGoogle Scholar
  5. 5.
    Cartis, C., Gould, N.I.M., Toint, PhL: Adaptive cubic regularisation methods for unconstrained optimization, Part II: worst-case function- and derivative-evaluation complexity. Math. Program. 130, 295–319 (2011)CrossRefzbMATHMathSciNetGoogle Scholar
  6. 6.
    Cartis, C., Gould, N.I.M., Toint, PhL: An adaptive cubic regularization algorithm for nonconvex optimization with convex constraints and its function-evaluation complexity. IMA J. Numer. Anal. 32, 1661–1695 (2012)CrossRefMathSciNetGoogle Scholar
  7. 7.
    Cartis, C., Gould, N.I.M., Toint, PhL: Complexity bounds for second-order optimality in unconstrained optimization. J. Complex. 28, 93–108 (2012)CrossRefzbMATHMathSciNetGoogle Scholar
  8. 8.
    Chen, X.: Smoothing methods for nonsmooth, novonvex minimization. Math. Program. 134, 71–99 (2012)CrossRefzbMATHMathSciNetGoogle Scholar
  9. 9.
    Chen, X., Ge, D., Wang, Z., Ye, Y.: Complexity of unconstrained \(L_2\)-\(L_p\) minimization. Math. Program. 143, 371–383 (2014)Google Scholar
  10. 10.
    Chen, X., Ng, M., Zhang, C.: Nonconvex \(l_p\) regularization and box constrained model for image restoration. IEEE Trans. Image Process. 21, 4709–4721 (2012)CrossRefMathSciNetGoogle Scholar
  11. 11.
    Chen, X., Niu, L., Yuan, Y.: Optimality conditions and smoothing trust region Newton method for non-Lipschitz optimization. SIAM J. Optim. 23, 1528–1552 (2013)CrossRefzbMATHMathSciNetGoogle Scholar
  12. 12.
    Chen, X., Xu, F., Ye, Y.: Lower bound theory of nonzero entries in solutions of \(l_2\)-\(l_{p}\) minimization. SIAM J. Sci. Comput. 32, 2832–2852 (2010)CrossRefzbMATHMathSciNetGoogle Scholar
  13. 13.
    Chen, X., Zhou, W.: Smoothing nonlinear conjugate gradient method for image restoration using nonsmooth nonconvex minimization. SIAM J. Imaging Sci. 3, 765–790 (2010)CrossRefzbMATHMathSciNetGoogle Scholar
  14. 14.
    Fan, J.: Comments on ’Wavelets in stastics: a review’ by A. Antoniadis. J. Ital. Stat. Soc. 6, 131–138 (1997)CrossRefGoogle Scholar
  15. 15.
    Garmanjani, R., Vicente, L.N.: Smoothing and worst case complexity for direct-search methods in nonsmooth optimization. IMA J. Numer. Anal. 33, 1008–1028 (2013)Google Scholar
  16. 16.
    Ge, D., Jiang, X., Ye, Y.: A note on the complexity of \(L_p\) minimization. Math. Program. 21, 1721–1739 (2011)MathSciNetGoogle Scholar
  17. 17.
    Gratton, S., Sartenaer, A., Toint, PhL: Recursive trust-region methods for multiscale nonlinear optimization. SIAM J. Optim. 19, 414–444 (2008)CrossRefzbMATHMathSciNetGoogle Scholar
  18. 18.
    Griewank, A.: The Modification of Newton’s Method for Unconstrained Optimization by Bounding Cubic Terms. Technical report NA/12 (1981). University of Cambridge, UK Department of Applied Mathematics and Theoretical Physics (1981)Google Scholar
  19. 19.
    Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning Data Mining, Inference, and Prediction. Springer, New York (2009)zbMATHGoogle Scholar
  20. 20.
    Huang, J., Horowitz, J.L., Ma, S.: Asymptotic properties of bridge estimators in sparse high-dimensional regression models. Ann. Stat. 36, 587–613 (2008)CrossRefzbMATHMathSciNetGoogle Scholar
  21. 21.
    Nesterov, Y.: Introductory Lectures on Convex Optimization, Applied Optimization. Kluwer, Dordrecht (2004)CrossRefGoogle Scholar
  22. 22.
    Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton’s method and its global performance. Math. Program. 108, 177–205 (2006)CrossRefzbMATHMathSciNetGoogle Scholar
  23. 23.
    Nesterov, Y.: Accelerating the cubic regularization of Newton’s method on convex problems. Math. Program. 112, 159–181 (2008)CrossRefzbMATHMathSciNetGoogle Scholar
  24. 24.
    Nikolova, M., Ng, M.K., Zhang, S., Ching, W.-K.: Efficient reconstruction of piecewise constant images using nonsmooth nonconvex minimization. SIAM J. Imaging Sci. 1, 2–25 (2008)CrossRefzbMATHMathSciNetGoogle Scholar
  25. 25.
    Tibshirani, R.: Shrinkage and selection via the Lasso. J. R. Stat. Soc. Ser. B 58, 267–288 (1996)zbMATHMathSciNetGoogle Scholar
  26. 26.
    Vavasis, S.A., Zippel, R.: Proving Polynomial Time for Sphere-Constrained Quadratic Programming, Technical Report 90-1182, Department of Computer Science. Cornell University, Ithaca, NY (1990)Google Scholar
  27. 27.
    Vavasis, S.A.: Nonlinear Optimization: Complexity Issues. Oxford Sciences, New York (1991)zbMATHGoogle Scholar
  28. 28.
    Ye, Y.: Interior Point Algorithms: Theory and Analysis. Wiley, New York (1997)CrossRefzbMATHGoogle Scholar
  29. 29.
    Ye, Y.: On the complexity of approximating a KKT point of quadratic programming. Math. Program. 80, 195–211 (1998)CrossRefzbMATHGoogle Scholar
  30. 30.
    Ye, Y.: A new complexity result on minimization of a quadratic function with a sphere constraint. In: Floudas, C., Pardalos, P.M. (eds.) Recent Advances in Global Optimization. Princeton University Press, Princeton (1992)Google Scholar
  31. 31.
    Zhang, C.-H.: Nearly unbiased variable selection under minimax concave penalty. Ann. Stat. 38, 894–942 (2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg and Mathematical Optimization Society 2014

Authors and Affiliations

  1. 1.Department of MathematicsHarbin Institute of TechnologyHarbinChina
  2. 2.Department of Applied MathematicsThe Hong Kong Polytechnic UniversityHong KongChina
  3. 3.Department of Management Science and EngineeringStanford UniversityStanfordUSA

Personalised recommendations