Advertisement

Computational Optimization and Applications

, Volume 71, Issue 3, pp 795–828 | Cite as

A majorization–minimization algorithm for split feasibility problems

  • Jason Xu
  • Eric C. Chi
  • Meng Yang
  • Kenneth Lange
Article
  • 174 Downloads

Abstract

The classical multi-set split feasibility problem seeks a point in the intersection of finitely many closed convex domain constraints, whose image under a linear mapping also lies in the intersection of finitely many closed convex range constraints. Split feasibility generalizes important inverse problems including convex feasibility, linear complementarity, and regression with constraint sets. When a feasible point does not exist, solution methods that proceed by minimizing a proximity function can be used to obtain optimal approximate solutions to the problem. We present an extension of the proximity function approach that generalizes the linear split feasibility problem to allow for non-linear mappings. Our algorithm is based on the principle of majorization–minimization, is amenable to quasi-Newton acceleration, and comes complete with convergence guarantees under mild assumptions. Furthermore, we show that the Euclidean norm appearing in the proximity function of the non-linear split feasibility problem can be replaced by arbitrary Bregman divergences. We explore several examples illustrating the merits of non-linear formulations over the linear case, with a focus on optimization for intensity-modulated radiation therapy.

Keywords

Majorize–minimize Nonlinear split feasibility Intensity modulated radiation therapy Proximity function minimization Constrained regression 

Notes

Acknowledgements

We thank Steve Wright and Dávid Papp for their help with the IMRT data examples, and thank Patrick Combettes for helpful comments. We also thank two anonymous referees for their constructive comments and thoughtful feedback.

References

  1. 1.
    Alber, M., Meedt, G., Nusslin, F., Reemtsen, R.: On the degeneracy of the IMRT optimization problem. Med. Phys. 29(11), 2584–2589 (2002)CrossRefGoogle Scholar
  2. 2.
    Antoniadis, A., Gijbels, I., Nikolova, M.: Penalized likelihood regression for generalized linear models with non-quadratic penalties. Ann. Inst. Stat. Math. 63(3), 585–615 (2011)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Bauschke, H.H., Borwein, J.M., Combettes, P.L.: Essential smoothness, essential strict convexity, and legendre functions in banach spaces. Commun. Contemp. Math. 3(04), 615–647 (2001)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Becker, M.P., Yang, I., Lange, K.: EM algorithms without missing data. Stat. Methods Med. Res. 6, 38–54 (1997)CrossRefGoogle Scholar
  5. 5.
    Biegler, L.T., Zavala, V.M.: Large-scale nonlinear programming using IPOPT: an integrating framework for enterprise-wide dynamic optimization. Comput. Chem. Eng. 33(3), 575–582 (2009)CrossRefGoogle Scholar
  6. 6.
    Bregman, L.M.: The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. USSR Comput. Math. Math. Phys. 7(3), 200–217 (1967)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Byrne, C.L.: Iterative image reconstruction algorithms based on cross-entropy minimization. IEEE Trans. Image Process. 2(1), 96–103 (1993)CrossRefGoogle Scholar
  8. 8.
    Byrne, C.L.: Iterative projection onto convex sets using multiple Bregman distances. Inverse Probl. 15(5), 1295 (1999)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Byrne, C.L.: Iterative oblique projection onto convex sets and the split feasibility problem. Inverse Probl. 18(2), 441–453 (2002)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Byrne, C.L.: Sequential unconstrained minimization algorithms for constrained optimization. Inverse Probl. 24(1), 015013 (2008)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Byrne, C.L.: Alternating minimization as sequential unconstrained minimization: a survey. J. Optim. Theory Appl. 156(3), 554–566 (2013)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Byrne, C.L.: An elementary proof of convergence for the forward-backward splitting algorithm. J. Nonlinear Convex Anal. 15(4), 681–691 (2014)MathSciNetzbMATHGoogle Scholar
  13. 13.
    Cegielski, A.: General method for solving the split common fixed point problem. J. Optim. Theory Appl. 165(2), 385–404 (2015)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Censor, Y.: Row-action methods for huge and sparse systems and their applications. SIAM Rev. 23(4), 444–466 (1981)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Censor, Y.: Weak and strong superiorization: between feasibility-seeking and minimization. Analele Univ. Ovidius Constanta Ser. Mat. 23(3), 41–54 (2015)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Censor, Y., Bortfeld, T., Martin, B., Trofimov, A.: A unified approach for inversion problems in intensity-modulated radiation therapy. Phys. Med. Biol. 51(10), 2353 (2006)CrossRefGoogle Scholar
  17. 17.
    Censor, Y., Elfving, T.: A multiprojection algorithm using Bregman projections in a product space. Numer. Algorithms 8(2), 221–239 (1994)MathSciNetCrossRefGoogle Scholar
  18. 18.
    Censor, Y., Elfving, T., Kopf, N., Bortfeld, T.: The multiple-sets split feasibility problem and its applications for inverse problems. Inverse Probl. 21(6), 2071–2084 (2005)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Censor, Y., Gibali, A., Reich, S.: Algorithms for the split variational inequality problem. Numer. Algorithms 59(2), 301–323 (2012)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Censor, Y., Motova, A., Segal, A.: Perturbed projections and subgradient projections for the multiple-sets split feasibility problem. J. Math. Anal. Appl. 327(2), 1244–1256 (2007)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Censor, Y., Zenios, S.A.: Parallel Optimization: Theory, Algorithms, and Applications. Oxford University Press on Demand, Oxford (1997)zbMATHGoogle Scholar
  22. 22.
    Chi, E.C., Lange, K.: A look at the generalized heron problem through the lens of majorization–minimization. Am. Math. Mon. 121(2), 95–108 (2014)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Chi, E.C., Zhou, H., Lange, K.: Distance majorization and its applications. Math. Program. Ser. A 146(1–2), 409–436 (2014)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Combettes, P.L.: Inconsistent signal feasibility problems: least-squares solutions in a product space. IEEE Trans. Signal Process. 42(11), 2955–2966 (1994)CrossRefGoogle Scholar
  25. 25.
    Combettes, P.L.: Solving monotone inclusions via compositions of nonexpansive averaged operators. Optimization 53(5–6), 475–504 (2004)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Combettes, P.L., Bondon, P.: Hard-constrained inconsistent signal feasibility problems. IEEE Trans. Signal Process. 47(9), 2460–2468 (1999)CrossRefGoogle Scholar
  27. 27.
    Combettes, P.L., Wajs, V.R.: Signal recovery by proximal forward–backward splitting. Multiscale Model. Simul. 4(4), 1168–1200 (2005)MathSciNetCrossRefGoogle Scholar
  28. 28.
    Craft, D., Bangert, M., Long, T., Papp, D., Unkelbach, J.: Shared data for intensity modulated radiation therapy (IMRT) optimization research: the CORT dataset. GigaSci. 3(1), 37 (2014)CrossRefGoogle Scholar
  29. 29.
    Davenport, M.A., Duarte, M.F., Eldar, Y.C., Kutyniok, G.: Chapter 1. Introduction to compressed sensing. Compressed Sensing: Theory and Applications. Cambridge University Press, Cambridge (2012)Google Scholar
  30. 30.
    Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the EM algorithm. J. R. Stat. Soc. Ser. B (Methodol.) 39(1), 1–38 (1977)MathSciNetzbMATHGoogle Scholar
  31. 31.
    Dingping, W., Qibin, D., Erli, W., Hang, Z.: The Split Feasibility Problem in Hilbert Space, pp. 1149–1154. Springer, Berlin (2013)Google Scholar
  32. 32.
    Ehrgott, M., Güler, c, Hamacher, H.W., Shao, L.: Mathematical optimization in intensity modulated radiation therapy. Ann. Oper. Res. 175(1), 309–365 (2010)MathSciNetCrossRefGoogle Scholar
  33. 33.
    Fan, J., Lv, J.: A selective overview of variable selection in high dimensional feature space. Stat. Sin. 20(1), 101 (2010)MathSciNetzbMATHGoogle Scholar
  34. 34.
    Févotte, C., Idier, J.: Algorithms for nonnegative matrix factorization with the \(\beta \)-divergence. Neural Comput. 23(9), 2421–2456 (2011)MathSciNetCrossRefGoogle Scholar
  35. 35.
    Fiacco, A.V., McCormick, G.P.: Nonlinear Programming: Sequential Unconstrained Minimization Techniques. Classics in Applied Mathematics. SIAM, PA (1990)CrossRefGoogle Scholar
  36. 36.
    Gibali, A., Küfer, K., Süss, P.: Successive linear programing approach for solving the nonlinear split feasibility problem. J. Nonlinear Convex Anal. 15(2), 345–353 (2014)MathSciNetGoogle Scholar
  37. 37.
    Gibali, A., Küfer, K.-H., Reem, D., Süss, P.: A generalized projection-based scheme for solving convex constrained optimization problems. Comput. Optim. Appl. 70(3), 737–762 (2018)MathSciNetCrossRefGoogle Scholar
  38. 38.
    Goldstein, T., Studer, C., Baraniuk, R.: A field guide to forward-backward splitting with a FASTA implementation. arXiv:1411.3406 [cs.NA] (2014)
  39. 39.
    Gordon, R., Bender, R., Herman, G.T.: Algebraic reconstruction techniques (ART) for three-dimensional electron microscopy and X-ray photography. J. Theor. Biol. 29(3), 471–481 (1970)CrossRefGoogle Scholar
  40. 40.
    Hou, Q., Wang, J., Chen, Y., Galvin, J.M.: An optimization algorithm for intensity modulated radiotherapy-the simulated dynamics with dose-volume constraints. Med. Phys. 30(1), 61–68 (2003)CrossRefGoogle Scholar
  41. 41.
    Lange, K.: A gradient algorithm locally equivalent to the EM algorithm. J. R. Stat. Soc. Ser. B (Methodol.) 57(2), 425–437 (1995)MathSciNetzbMATHGoogle Scholar
  42. 42.
    Lange, K.: Numerical Analysis for Statisticians. Statistics and Computing, 2nd edn. Springer, New York (2010)CrossRefGoogle Scholar
  43. 43.
    Lange, K.: Optimization. Springer Texts in Statistics, 2nd edn. Springer, New York (2013)Google Scholar
  44. 44.
    Lange, K.: MM Optimization Algorithms. SIAM, PA (2016)CrossRefGoogle Scholar
  45. 45.
    Lange, K., Hunter, D.R., Yang, I.: Optimization transfer using surrogate objective functions (with discussion). J. Comput. Gr. Stat. 9, 1–20 (2000)Google Scholar
  46. 46.
    Li, Z., Han, D., Zhang, W.: A self-adaptive projection-type method for nonlinear multiple-sets split feasibility problem. Inverse Probl. Sci. Eng. 21(1), 155–170 (2013)MathSciNetCrossRefGoogle Scholar
  47. 47.
    Llacer, J., Deasy, J.O., Bortfeld, T.R., Solberg, T.D., Promberger, C.: Absence of multiple local minima effects in intensity modulated optimization with dose-volume constraints. Phys. Med. Biol. 48(2), 183 (2003)CrossRefGoogle Scholar
  48. 48.
    Lorenz, D.A., Schöpfer, F., Wenger, S.: The linearized Bregman method via split feasibility problems: analysis and generalizations. SIAM J. Imaging Sci. 7(2), 1237–1262 (2014)MathSciNetCrossRefGoogle Scholar
  49. 49.
    Luenberger, D.G., Ye, Y.: Linear and Nonlinear Programming. International series in operations research & management science, volume 228, 4th edn. Springer, New York (2016)CrossRefGoogle Scholar
  50. 50.
    Masad, E., Reich, S.: A note on the multiple-set split convex feasibility problem in Hilbert space. J. Nonlinear Convex Anal. 8(3), 367–371 (2007)MathSciNetzbMATHGoogle Scholar
  51. 51.
    McCullagh, P., Nelder, J.A.: Generalized Linear Models, vol. 37. CRC press, Boca Raton (1989)CrossRefGoogle Scholar
  52. 52.
    Mordukhovich, B.S., Nam, N.M.: Applications of variational analysis to a generalized Fermat–Torricelli problem. J. Optim. Theory Appl. 148, 431–454 (2011)MathSciNetCrossRefGoogle Scholar
  53. 53.
    Mordukhovich, B.S., Nam, N.M., Salinas, J.: Applications of variational analysis to a generalized Heron problem. Appl. Anal. 91(10), 1915–1942 (2011)MathSciNetCrossRefGoogle Scholar
  54. 54.
    Mordukhovich, B.S., Nam, N.M., Salinas, J.: Solving a generalized Heron problem by means of convex analysis. Am. Math. Mon. 119(2), 87–99 (2012)MathSciNetCrossRefGoogle Scholar
  55. 55.
    Moré, J.J.: The Levenberg–Marquardt algorithm: implementation and theory. In: Numerical analysis, pp. 105–116. Springer (1978)Google Scholar
  56. 56.
    Moudafi, A.: A note on the split common fixed-point problem for quasi-nonexpansive operators. Nonlinear Anal. Theory Methods Appl. 74(12), 4083–4087 (2011)MathSciNetCrossRefGoogle Scholar
  57. 57.
    Murty, K.G., Yu, F.-T.: Linear Complementarity, Linear and Nonlinear Programming, vol. 3. Citeseer, Heldermann, Berlin (1988)zbMATHGoogle Scholar
  58. 58.
    Palta, J.R., Mackie, T.R. (eds.): Intensity-Modulated Radiation Therapy: The State of The Art. Medical Physics Publishing, Madison (2003)Google Scholar
  59. 59.
    Park, M.Y., Hastie, T.: L1-regularization path algorithm for generalized linear models. J. R. Stat. Soc. Ser. B (Stat. Methodol.) 69(4), 659–677 (2007)MathSciNetCrossRefGoogle Scholar
  60. 60.
    Polson, N.G., Scott, J.G., Willard, B.T.: Proximal algorithms in statistics and machine learning. Stat. Sci. 30(4), 559–581 (2015)MathSciNetCrossRefGoogle Scholar
  61. 61.
    R Core Team. R: a language and environment for statistical computing. R foundation for statistical computing, Vienna (2017)Google Scholar
  62. 62.
    Rockafellar, R.T., Wets, R.J.-B.: Variational Analysis. Springer, New York (1998)CrossRefGoogle Scholar
  63. 63.
    Shepard, D.M., Ferris, M.C., Olivera, G.H., Mackie, T.R.: Optimizing the delivery of radiation therapy to cancer patients. SIAM Rev. 41(4), 721–744 (1999)CrossRefGoogle Scholar
  64. 64.
    Tibshirani, R.: Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B (Methodol.) 58(1), 267–288 (1996)MathSciNetzbMATHGoogle Scholar
  65. 65.
    Wang, F., Xu, H.-K.: Cyclic algorithms for split feasibility problems in Hilbert spaces. Nonlinear Anal. Theory Methods Appl. 74(12), 4105–4111 (2011)MathSciNetCrossRefGoogle Scholar
  66. 66.
    Wright, S.J., Nowak, R.D., Figueiredo, M.A.T.: Sparse reconstruction by separable approximation. IEEE Trans. Signal Process. 57(7), 2479–2493 (2009)MathSciNetCrossRefGoogle Scholar
  67. 67.
    Xu, H.-K.: A variable Krasnosel’skii Mann algorithm and the multiple-set split feasibility problem. Inverse Probl. 22(6), 2021–2034 (2006)CrossRefGoogle Scholar
  68. 68.
    Xu, H.-K.: Iterative methods for the split feasibility problem in infinite-dimensional Hilbert spaces. Inverse Probl. 26(10), 105018 (2010)MathSciNetCrossRefGoogle Scholar
  69. 69.
    Xu, J., Chi, E., Lange, K.: Generalized linear model regression under distance-to-set penalties. In: Advances in Neural Information Processing Systems, pp. 1385–1395 (2017)Google Scholar
  70. 70.
    Zhang, X., Liu, H., Wang, X., Dong, L., Wu, Q., Mohan, R.: Speed and convergence properties of gradient algorithms for optimization of IMRT. Med. Phy. 31(5), 1141–1152 (2004)CrossRefGoogle Scholar
  71. 71.
    Zhou, H., Alexander, D., Lange, K.: A quasi-Newton acceleration for high-dimensional optimization algorithms. Stat. Comput. 21, 261–273 (2011)MathSciNetCrossRefGoogle Scholar
  72. 72.
    Zou, H., Hastie, T.: Regularization and variable selection via the elastic net. J. R. Stat. Soc. Ser. B (Methodol.) 67(2), 301–320 (2005)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Statistical ScienceDuke UniversityDurhamUSA
  2. 2.Department of StatisticsNorth Carolina State UniversityRaleighUSA
  3. 3.Departments of Biomathematics, Human Genetics, and StatisticsUniversity of CaliforniaLos AngelesUSA

Personalised recommendations