Advertisement

Mathematical Programming

, Volume 174, Issue 1–2, pp 359–390 | Cite as

Level-set methods for convex optimization

  • Aleksandr Y. AravkinEmail author
  • James V. Burke
  • Dmitry Drusvyatskiy
  • Michael P. Friedlander
  • Scott Roy
Full Length Paper Series B
  • 290 Downloads

Abstract

Convex optimization problems arising in applications often have favorable objective functions and complicated constraints, thereby precluding first-order methods from being immediately applicable. We describe an approach that exchanges the roles of the objective with one of the constraint functions, and instead approximately solves a sequence of parametric level-set problems. Two Newton-like zero-finding procedures for nonsmooth convex functions, based on inexact evaluations and sensitivity information, are introduced. It is shown that they lead to efficient solution schemes for the original problem. We describe the theoretical and practical properties of this approach for a broad range of problems, including low-rank semidefinite optimization, sparse optimization, and gauge optimization.

Keywords

First-order methods Root-finding Semidefinite and conic programming 

Mathematics Subject Classification

90C25 65K10 49M29 90-08 

Notes

Acknowledgements

The authors extend their sincere thanks to three anonymous referees who provided an extensive list of corrections and suggestions that helped us to streamline our presentation.

References

  1. 1.
    Aravkin, A.Y., Burke, J., Friedlander, M.P.: Variational properties of value functions. SIAM J. Optim. 23(3), 1689–1717 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Bach, F.: Duality between subgradient and conditional gradient methods. SIAM J. Optim. 25(1), 115–129 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2(1), 183–202 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Beck, A., Teboulle, M.: Mirror descent and nonlinear projected subgradient methods for convex optimization. Oper. Res. Lett. 31(3), 167–175 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Ben-Tal, A., Nemirovski, A.: Lectures on Modern Convex Optimization. MPS/SIAM Series on Optimization. SIAM, Philadelphia (2001)CrossRefzbMATHGoogle Scholar
  6. 6.
    Bertsekas, D.P.: Convex Optimization Algorithms. Athena Scientific, Belmont (2015)zbMATHGoogle Scholar
  7. 7.
    Biswas, P., Ye, Y.: Semidefinite programming for ad hoc wireless sensor network localization. In Proceedings of the 3rd International Symposium on Information Processing in Sensor Networks, pp. 46–54. ACM (2004)Google Scholar
  8. 8.
    Biswas, P., Liang, T.C., Toh, K.C., Ye, Y., Wang, T.C.: Semidefinite programming approaches for sensor network localization with noisy distance measurements. IEEE Trans. Autom. Sci. Eng. 3(4), 360–371 (2006)CrossRefGoogle Scholar
  9. 9.
    Biswas, P., Lian, T.C., Wang, T.C., Ye, Y.: Semidefinite programming based algorithms for sensor network localization. ACM Trans. Sens. Netw. (TOSN) 2(2), 188–220 (2006)CrossRefGoogle Scholar
  10. 10.
    Borwein, J.M., Lewis, A.S.: Convex Analysis and Nonlinear Optimization. Theory and Examples. CMS Books in Mathematics/ouvrages de Mathématiques de la SMC, 3. Springer, New York (2000)CrossRefzbMATHGoogle Scholar
  11. 11.
    Boyd, S., Parikh, N., Chu, R., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2011)CrossRefzbMATHGoogle Scholar
  12. 12.
    Brucker, P.: An \(o(n)\) algorithm for quadratic knapsack problems. Oper. Res. Lett. 3(3), 163–166 (1984)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Candès, E.J., Li, X., Ma, Y., Wright, J.: Robust principal component analysis? J. Assoc. Comput. Mach. 58(3), 1–37 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Candès, E.J., Tao, T.: The power of convex relaxation: near-optimal matrix completion. IEEE Trans. Inf. Theory 56(5), 2053–2080 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Candès, E.J., Li, X., Ma, Y., Wright, J.: Robust principal component analysis? J. ACM 58(3), 11–11137 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Candès, E.J., Strohmer, T., Voroninski, V.: Phaselift: exact and stable signal recovery from magnitude measurements via convex programming. Commun. Pure Appl. Math. 66, 1241–1274 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Chandrasekaran, V., Parrilo, P.A., Willsky, A.S.: Latent variable graphical model selection via convex optimization. Ann. Stat. 40(4), 1935–2357 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Chen, S.S., Donoho, D.L., Saunders, M.A.: Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 20(1), 33–61 (1999)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Cox, B., Juditsky, A., Nemirovski, A.: Dual subgradient algorithms for large-scale nonsmooth learning problems. Math. Program. 148(1–2), 143–180 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Drusvyatskiy, D., Pataki, G., Wolkowicz, H.: Coordinate shadows of semidefinite and Euclidean distance matrices. SIAM J. Optim. 25(2), 1160–1178 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Drusvyatskiy, D., Krislock, N., Voronin, Y.L., Wolkowicz, H.: Noisy Euclidean distance realization: robust facial reduction and the Pareto frontier. Preprint arXiv:1410.6852 (2014)
  22. 22.
    Ennis, R.H., McGuire, G.C.: Computer Algebra Recipes: A Gourmet’s Guide to the Mathematical Models of Science. Springer, Berlin (2001)CrossRefzbMATHGoogle Scholar
  23. 23.
    Fazel, M.: Matrix rank minimization with applications. Eng. Dept, Stanford University, PhD diss, Elec (2002)Google Scholar
  24. 24.
    Frank, M., Wolfe, P.: An algorithm for quadratic programming. Naval Res. Logist. Q. 3, 95–110 (1956)MathSciNetCrossRefGoogle Scholar
  25. 25.
    Friedlander, M.P., Macêdo, I., Pong, T.K.: Gauge optimization and duality. SIAM J. Optim. 24(4), 1999–2022 (2014).  https://doi.org/10.1137/130940785 MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite element approximations. Comput. Math. Appl. 2(1), 17–40 (1976)CrossRefzbMATHGoogle Scholar
  27. 27.
    Hager, W.W., Huang, S.J., Pardalos, P.M., Prokopyev, O.A. (eds.): Multiscale Optimization Methods and Applications, Nonconvex Optimization and Its Applications, vol. 82. Springer (2006)Google Scholar
  28. 28.
    Harchaoui, Z., Juditsky, A., Nemirovski, A.: Conditional gradient algorithms for norm-regularized smooth convex optimization. Math. Program. 152(1–2, Ser. A), 75–112 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  29. 29.
    Martin, J.: Revisiting Frank-Wolfe: projection-free sparse convex optimization. In: Proc. 30th Intern. Conf. Machine Learning (ICML-13), pp. 427–435 (2013)Google Scholar
  30. 30.
    Krislock, N., Wolkowicz, H.: Explicit sensor network localization using semidefinite representations and facial reductions. SIAM J. Optim. 20(5), 2679–2708 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  31. 31.
    Lehoucq, R.B., Sorensen, D.C., Yang, C.: ARPACK users’ guide: solution of large-scale eigenvalue problems with implicitly restarted Arnoldi methods, vol. 6. SIAM, Philadelphia (1998)CrossRefzbMATHGoogle Scholar
  32. 32.
    Lemaréchal, C.: An extension of Davidon methods to nondifferentiable problems. Math. Program. Stud. 3, 95–109 (1975)CrossRefzbMATHGoogle Scholar
  33. 33.
    Lemaréchal, C., Nemirovskii, A., Nesterov, Y.: New variants of bundle methods. Math. Program. 69(1, Ser. B), 111–147 (1995). (Nondifferentiable and large-scale optimization (Geneva, 1992))MathSciNetCrossRefzbMATHGoogle Scholar
  34. 34.
    Liberti, L., Lavor, C., Maculan, N., Mucherino, A.: Euclidean distance geometry and applications. SIAM Rev. 56(1), 3–69 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  35. 35.
    Ling, S., Strohmer, T.: Self-calibration and biconvex compressive sensing. CoRR arXiv:1501.06864 (2015)
  36. 36.
    Luke, R., Burke, J., Lyons, R.: Optical wavefront reconstruction: theory and numerical methods. SIAM Rev. 44, 169–224 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  37. 37.
    Markowitz, H.M.: Mean-Variance Analysis in Portfolio Choice and Capital Markets. Frank J, Fabozzi Associates, New Hope (1987)zbMATHGoogle Scholar
  38. 38.
    Marquardt, D.: An algorithm for least-squares estimation of nonlinear parameters. SIAM J. Appl. Math. 11, 431–441 (1963)MathSciNetCrossRefzbMATHGoogle Scholar
  39. 39.
    Miettinen, K.: Nonlinear Multi-objective Optimization. Springer, Berlin (1999)Google Scholar
  40. 40.
    Morrison, D.D.: Methods for nonlinear least squares problems and convergence proofs. In: Lorell, J., Yagi, F. (eds.)Proceedings of the Seminar on Tracking Programs and Orbit Determination, pp. 1–9. Jet Propulsion Laboratory, Pasadena (1960)Google Scholar
  41. 41.
    Nemirovski, A.: Prox-method with rate of convergence \(O(1/t)\) for variational inequalities with Lipschitz continuous monotone operators and smooth convex-concave saddle point problems. SIAM J. Optim. 15(1), 229–251 (2004)MathSciNetCrossRefzbMATHGoogle Scholar
  42. 42.
    Nesterov, Y.: Introductory Lectures on Convex Optimization. Kluwer Academic, Dordrecht (2004)CrossRefzbMATHGoogle Scholar
  43. 43.
    Nesterov, Y.: Smooth minimization of non-smooth functions. Math. Program. 103(1), 127–152 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  44. 44.
    Osborne, M.R., Presnell, B., Turlach, B.A.: A new approach to variable selection in least squares problems. IMA J. Numer. Anal. 20(3), 389–403 (2000)MathSciNetCrossRefzbMATHGoogle Scholar
  45. 45.
    Peng, Y., Ganesh, A., Wright, J., Xu, W., Ma, Y.: RASL: robust alignment by sparse and low-rank decomposition for linearly correlated images. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2233–2246 (2012)CrossRefGoogle Scholar
  46. 46.
    Recht, B., Fazel, M., Parrilo, P.A.: Guaranteed minimum rank solutions to linear matrix equations via nuclear norm minimization. SIAM Rev. 52(3), 471–501 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  47. 47.
    Recht, B., Fazel, M., Parrilo, P.A.: Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Rev. 52(3), 471–501 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  48. 48.
    Renegar, J.: Linear programming, complexity theory and elementary functional analysis. Math. Program. 70(3, Ser. A), 279–351 (1995)MathSciNetzbMATHGoogle Scholar
  49. 49.
    Renegar, J.: A framework for applying subgradient methods to conic optimization problems. Preprint arXiv:1503.02611 (2015)
  50. 50.
    Rockafellar, R.T.: Convex Analysis. Priceton Landmarks in Mathematics. Princeton University Press, Princeton (1970)Google Scholar
  51. 51.
    Tseng, P.: Approximation accuracy, gradient methods, and error bound for structured convex optimization. Math. Program. 125(2), 263–295 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  52. 52.
    van den Berg, E., Friedlander, M.P.: Sparse optimization with least-squares constraints. SIAM J. Optim. 21(4), 1201–1229 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  53. 53.
    van den Berg, E., Friedlander, M.P.: Probing the Pareto frontier for basis pursuit solutions. SIAM J. Sci. Comput. 31(2), 890–912 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  54. 54.
    Waldspurger, I., d’Aspremont, A., Mallat, S.: Phase recovery, maxcut and complex semidefinite programming. Math. Program. 149(1–2), 47–81 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  55. 55.
    Weinberger, K.Q., Sha, F., Saul, L.K.: Learning a kernel matrix for nonlinear dimensionality reduction. In: Proceedings of the Twenty-first International Conference on Machine Learning. ICML ’04, p. 106. ACM, New York (2004)Google Scholar
  56. 56.
    Wolfe, P.: A method of conjugate subgradients for minimizing nondifferentiable functions. Math. Program. Stud. 3, 145–173 (1975)MathSciNetCrossRefzbMATHGoogle Scholar
  57. 57.
    Wright, J., Ganesh, A., Rao, S., Ma, Y.: Robust principal component analysis: exact recovery of corrupted low-rank matrices by convex optimization. In: Neural Information Processing Systems (NIPS) (2009)Google Scholar
  58. 58.
    Wright, S.J., Nowak, R.D., Figueiredo, M.A.T.: Sparse reconstruction by separable approximation. IEEE Trans. Signal Process. 57(7), 2479–2493 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  59. 59.
    Yin, W.: Analysis and generalizations of the linearized bregman method. SIAM J. Imaging Sci. 3(4), 856–877 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  60. 60.
    Zhang, Z., Liang, X., Ganesh, A., Ma, Y.: TILT: transform invariant low-rank textures. In: Kimmel, R., Klette, R., Sugimoto, A. (eds.) Computer Vision—Accv 2010. Vol. 6494 of Lecture Notes in Computer Science, pp. 314–328. Springer, Berlin (2011)Google Scholar
  61. 61.
    Zheng, P., Aravkin, A.Y., Ramamurthy, K.N., Thiagarajan, J.J: Learning robust representations for computer vision. In: The IEEE International Conference on Computer Vision (ICCV) Workshops (2017)Google Scholar
  62. 62.
    Zou, H., Hastie, T.: Regularization and variable selection via the elastic net. J. R. Stat. Soc. Ser. B 67, 301–320 (2005)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature and Mathematical Optimization Society 2018

Authors and Affiliations

  1. 1.Department of Applied MathematicsUniversity of WashingtonSeattleUSA
  2. 2.Department of MathematicsUniversity of WashingtonSeattleUSA
  3. 3.Department of Computer ScienceUniversity of British ColumbiaVancouverCanada

Personalised recommendations