Mathematical Programming

, Volume 149, Issue 1–2, pp 301–327 | Cite as

Complexity analysis of interior point algorithms for non-Lipschitz and nonconvex minimization

Full Length Paper Series A

Abstract

We propose a first order interior point algorithm for a class of non-Lipschitz and nonconvex minimization problems with box constraints, which arise from applications in variable selection and regularized optimization. The objective functions of these problems are continuously differentiable typically at interior points of the feasible set. Our first order algorithm is easy to implement and the objective function value is reduced monotonically along the iteration points. We show that the worst-case iteration complexity for finding an \(\epsilon \) scaled first order stationary point is \(O(\epsilon ^{-2})\). Furthermore, we develop a second order interior point algorithm using the Hessian matrix, and solve a quadratic program with a ball constraint at each iteration. Although the second order interior point algorithm costs more computational time than that of the first order algorithm in each iteration, its worst-case iteration complexity for finding an \(\epsilon \) scaled second order stationary point is reduced to \(O(\epsilon ^{-3/2})\). Note that an \(\epsilon \) scaled second order stationary point must also be an \(\epsilon \) scaled first order stationary point.

Keywords

Constrained non-Lipschitz optimization Complexity analysis  Interior point method First order algorithm Second order algorithm 

Mathematics Subject Classifcation (2010)

90C30 90C26 65K05 49M37 

References

  1. 1.
    Bian, W., Chen, X.: Worst-case complexity of smoothing quadratic regularization methods for non-Lipschitzian optimization. SIAM J. Optim. 28, 1718–1741 (2013)CrossRefMathSciNetGoogle Scholar
  2. 2.
    Bruckstein, A.M., Donoho, D.L., Elad, M.: From sparse solutions of systems of equations to sparse modeling of signals and images. SIAM Rev. 51, 34–81 (2009)CrossRefMATHMathSciNetGoogle Scholar
  3. 3.
    Cartis, C., Gould, N.I.M., Toint, PhL: On the evaluation complexity of composite function minimization with applications to nonconvex nonlinear programming. SIAM J. Optim. 21, 1721–1739 (2011)CrossRefMATHMathSciNetGoogle Scholar
  4. 4.
    Cartis, C., Gould, N.I.M., Toint, PhL: Adaptive cubic regularisation methods for unconstrained optimization, Part I: motivation, convergence and numerical results. Math. Program. 127, 245–295 (2011)CrossRefMATHMathSciNetGoogle Scholar
  5. 5.
    Cartis, C., Gould, N.I.M., Toint, PhL: Adaptive cubic regularisation methods for unconstrained optimization, Part II: worst-case function- and derivative-evaluation complexity. Math. Program. 130, 295–319 (2011)CrossRefMATHMathSciNetGoogle Scholar
  6. 6.
    Cartis, C., Gould, N.I.M., Toint, PhL: An adaptive cubic regularization algorithm for nonconvex optimization with convex constraints and its function-evaluation complexity. IMA J. Numer. Anal. 32, 1661–1695 (2012)CrossRefMathSciNetGoogle Scholar
  7. 7.
    Cartis, C., Gould, N.I.M., Toint, PhL: Complexity bounds for second-order optimality in unconstrained optimization. J. Complex. 28, 93–108 (2012)CrossRefMATHMathSciNetGoogle Scholar
  8. 8.
    Chen, X.: Smoothing methods for nonsmooth, novonvex minimization. Math. Program. 134, 71–99 (2012)CrossRefMATHMathSciNetGoogle Scholar
  9. 9.
    Chen, X., Ge, D., Wang, Z., Ye, Y.: Complexity of unconstrained \(L_2\)-\(L_p\) minimization. Math. Program. 143, 371–383 (2014)Google Scholar
  10. 10.
    Chen, X., Ng, M., Zhang, C.: Nonconvex \(l_p\) regularization and box constrained model for image restoration. IEEE Trans. Image Process. 21, 4709–4721 (2012)CrossRefMathSciNetGoogle Scholar
  11. 11.
    Chen, X., Niu, L., Yuan, Y.: Optimality conditions and smoothing trust region Newton method for non-Lipschitz optimization. SIAM J. Optim. 23, 1528–1552 (2013)CrossRefMATHMathSciNetGoogle Scholar
  12. 12.
    Chen, X., Xu, F., Ye, Y.: Lower bound theory of nonzero entries in solutions of \(l_2\)-\(l_{p}\) minimization. SIAM J. Sci. Comput. 32, 2832–2852 (2010)CrossRefMATHMathSciNetGoogle Scholar
  13. 13.
    Chen, X., Zhou, W.: Smoothing nonlinear conjugate gradient method for image restoration using nonsmooth nonconvex minimization. SIAM J. Imaging Sci. 3, 765–790 (2010)CrossRefMATHMathSciNetGoogle Scholar
  14. 14.
    Fan, J.: Comments on ’Wavelets in stastics: a review’ by A. Antoniadis. J. Ital. Stat. Soc. 6, 131–138 (1997)CrossRefGoogle Scholar
  15. 15.
    Garmanjani, R., Vicente, L.N.: Smoothing and worst case complexity for direct-search methods in nonsmooth optimization. IMA J. Numer. Anal. 33, 1008–1028 (2013)Google Scholar
  16. 16.
    Ge, D., Jiang, X., Ye, Y.: A note on the complexity of \(L_p\) minimization. Math. Program. 21, 1721–1739 (2011)MathSciNetGoogle Scholar
  17. 17.
    Gratton, S., Sartenaer, A., Toint, PhL: Recursive trust-region methods for multiscale nonlinear optimization. SIAM J. Optim. 19, 414–444 (2008)CrossRefMATHMathSciNetGoogle Scholar
  18. 18.
    Griewank, A.: The Modification of Newton’s Method for Unconstrained Optimization by Bounding Cubic Terms. Technical report NA/12 (1981). University of Cambridge, UK Department of Applied Mathematics and Theoretical Physics (1981)Google Scholar
  19. 19.
    Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning Data Mining, Inference, and Prediction. Springer, New York (2009)MATHGoogle Scholar
  20. 20.
    Huang, J., Horowitz, J.L., Ma, S.: Asymptotic properties of bridge estimators in sparse high-dimensional regression models. Ann. Stat. 36, 587–613 (2008)CrossRefMATHMathSciNetGoogle Scholar
  21. 21.
    Nesterov, Y.: Introductory Lectures on Convex Optimization, Applied Optimization. Kluwer, Dordrecht (2004)CrossRefGoogle Scholar
  22. 22.
    Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton’s method and its global performance. Math. Program. 108, 177–205 (2006)CrossRefMATHMathSciNetGoogle Scholar
  23. 23.
    Nesterov, Y.: Accelerating the cubic regularization of Newton’s method on convex problems. Math. Program. 112, 159–181 (2008)CrossRefMATHMathSciNetGoogle Scholar
  24. 24.
    Nikolova, M., Ng, M.K., Zhang, S., Ching, W.-K.: Efficient reconstruction of piecewise constant images using nonsmooth nonconvex minimization. SIAM J. Imaging Sci. 1, 2–25 (2008)CrossRefMATHMathSciNetGoogle Scholar
  25. 25.
    Tibshirani, R.: Shrinkage and selection via the Lasso. J. R. Stat. Soc. Ser. B 58, 267–288 (1996)MATHMathSciNetGoogle Scholar
  26. 26.
    Vavasis, S.A., Zippel, R.: Proving Polynomial Time for Sphere-Constrained Quadratic Programming, Technical Report 90-1182, Department of Computer Science. Cornell University, Ithaca, NY (1990)Google Scholar
  27. 27.
    Vavasis, S.A.: Nonlinear Optimization: Complexity Issues. Oxford Sciences, New York (1991)MATHGoogle Scholar
  28. 28.
    Ye, Y.: Interior Point Algorithms: Theory and Analysis. Wiley, New York (1997)CrossRefMATHGoogle Scholar
  29. 29.
    Ye, Y.: On the complexity of approximating a KKT point of quadratic programming. Math. Program. 80, 195–211 (1998)CrossRefMATHGoogle Scholar
  30. 30.
    Ye, Y.: A new complexity result on minimization of a quadratic function with a sphere constraint. In: Floudas, C., Pardalos, P.M. (eds.) Recent Advances in Global Optimization. Princeton University Press, Princeton (1992)Google Scholar
  31. 31.
    Zhang, C.-H.: Nearly unbiased variable selection under minimax concave penalty. Ann. Stat. 38, 894–942 (2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg and Mathematical Optimization Society 2014

Authors and Affiliations

  1. 1.Department of MathematicsHarbin Institute of TechnologyHarbinChina
  2. 2.Department of Applied MathematicsThe Hong Kong Polytechnic UniversityHong KongChina
  3. 3.Department of Management Science and EngineeringStanford UniversityStanfordUSA

Personalised recommendations