Mathematical Programming

, Volume 142, Issue 1–2, pp 397–434 | Cite as

A feasible method for optimization with orthogonality constraints

  • Zaiwen WenEmail author
  • Wotao Yin
Full Length Paper Series A


Minimization with orthogonality constraints (e.g., \(X^\top X = I\)) and/or spherical constraints (e.g., \(\Vert x\Vert _2 = 1\)) has wide applications in polynomial optimization, combinatorial optimization, eigenvalue problems, sparse PCA, p-harmonic flows, 1-bit compressive sensing, matrix rank minimization, etc. These problems are difficult because the constraints are not only non-convex but numerically expensive to preserve during iterations. To deal with these difficulties, we apply the Cayley transform—a Crank-Nicolson-like update scheme—to preserve the constraints and based on it, develop curvilinear search algorithms with lower flops compared to those based on projections and geodesics. The efficiency of the proposed algorithms is demonstrated on a variety of test problems. In particular, for the maxcut problem, it exactly solves a decomposition formulation for the SDP relaxation. For polynomial optimization, nearest correlation matrix estimation and extreme eigenvalue problems, the proposed algorithms run very fast and return solutions no worse than those from their state-of-the-art algorithms. For the quadratic assignment problem, a gap 0.842 % to the best known solution on the largest problem “tai256c” in QAPLIB can be reached in 5 min on a typical laptop.


Orthogonality constraint Spherical constraint Stiefel manifold Cayley transformation Curvilinear search  Polynomial optimization Maxcut SDP Nearest correlation matrix Eigenvalue and eigenvector Invariant subspace Quadratic assignment problem 

Mathematics Subject Classification (2010)

49Q99 65K05 90C22 90C26 90C27 90C30 



We would like to thank Yin Zhang, Xin Liu, and Shiqian Ma for the discussions on optimization conditions, Jiawang Nie for the discussions on polynomial optimization, Chao Yang for the discussions on the Kohn–Sham equation, as well as Franz Rendl and Etienne de Klerk for their comments on QAPs. We would also like to thank Defeng Sun and Yan Gao for sharing their code PenCorr and their improvement for Major, as well as sharing the test data for the nearest correlation matrix problem. The authors are grateful to Adrian Lewis, the Associate Editor and three anonymous referees for their detailed and valuable comments and suggestions.


  1. 1.
    Absil, P.-A., Baker, C.G., Gallivan, K.A.: Trust-region methods on Riemannian manifolds. Found. Comput. Math. 7, 303–330 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Absil, P.-A., Mahony, R., Sepulchre, R.: Optimization Algorithms on Matrix Manifolds. Princeton University Press, Princeton (2008)zbMATHGoogle Scholar
  3. 3.
    Adler, R.L., Dedieu, J.-P., Margulies, J.Y., Martens, M., Shub, M.: Newton’s method on Riemannian manifolds and a geometric model for the human spine. IMA J. Numer. Anal. 22, 359–390 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Baker, C.G., Absil, P.-A., Gallivan, K.A.: An implicit trust-region method on Riemannian manifolds. IMA J. Numer. Anal. 28, 665–689 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Barzilai, J., Borwein, J.M.: Two-point step size gradient methods. IMA J. Numer. Anal. 8, 141–148 (1988)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Bencteux, G., Cancés, E., Hager, W.W., Le Bris, C.: Analysis of a quadratic programming decomposition algorithm. SIAM J. Numer. Anal. 47, 4517–4539 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Benson, S.J., Ye, Y., Zhang, X.: Solving large-scale sparse semidefinite programs for combinatorial optimization. SIAM J. Optim. 10, 443–461 (2000)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Boufounos, P., Baraniuk, R.: 1-bit Compressive Sensing. Conf. on. Info. Science and Systems (CISS), Princeton (2008)Google Scholar
  9. 9.
    Brace, I., Manton, J.H.: An improved BFGS-on-manifold algorithm for computing weighted low rank approximations. In Proceedings of the 17th International Symposium on Mathematical Theory of Networks and Systems, pp. 1735–1738 (2006)Google Scholar
  10. 10.
    Burer, S., Monteiro, R.D.C.: A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization. Math. Program. 95, 329–357 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Burkard, R.E., Karisch, S.E., Rendl, F.: QAPLIB: a quadratic assignment problem library. J. Global Optim. 10, 391–403 (1997)MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Cancès, E., Le Bris, C., Lions, P.-L.: Molecular simulation and related topics: some open mathematical problems. Nonlinearity 21, T165–T176 (2008)CrossRefzbMATHGoogle Scholar
  13. 13.
    Chang, G.J., Huang, L.-H., Yeh, H.-G.: On the rank of a cograph. Linear Algebra Appl. 429, 601–605 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Dai, Y.-H., Fletcher, R.: Projected Barzilai-Borwein methods for large-scale box-constrained quadratic programming. Numer. Math. 100, 21–47 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    d’Aspremont, A., El Ghaoui, L., Jordan, M., Lanckriet, G.R.: A direct formulation for sparse PCA using semidefinite programming. SIAM Rev. 49, 434–448 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Davis, T.A.: The University of Florida Sparse Matrix Collection, Technical Report. University of Florida, Florida (2010)Google Scholar
  17. 17.
    Dolan, E.D., Moré, J.J., Munson, T.S.: Benchmarking Optimization Software with cops 3.0, Technical Report. Mathematics and Computer Science Division, Argonne National Laboratory (February 2004)Google Scholar
  18. 18.
    Edelman, A., Arias, T.A., Smith, S.T.: The geometry of algorithms with orthogonality constraints. SIAM J. Matrix Anal. Appl. 20, 303–353 (1999)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Fletcher, R.: Practical Methods of Optimization. John Wiley & Sons (2000)Google Scholar
  20. 20.
    Francisco, J.B., Martínez, J.M., Martínez, L., Pisnitchenko, F.: Inexact restoration method for minimization problems arising in electronic structure calculations. Comput. Opt. Appl. 10(3), 555–590 (2011)Google Scholar
  21. 21.
    Friedland, S., Nocedal, J., Overton, M.L.: The formulation and analysis of numerical methods for inverse eigenvalue problems. SIAM J. Numer. Anal. 24, 634–667 (1987)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Gao, Y., Sun, D.: A Majorized Penalty Approach for Calibrating Rank Constrained Correlation Matrix Problems, Technical Report. National University of Singapore (2010)Google Scholar
  23. 23.
    Goldfarb, D., Wen, Z., Yin, W.: A curvilinear search method for the p-harmonic flow on spheres. SIAM J. Imaging Sci. 2, 84–109 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  24. 24.
    Golub, G.H., Van Loan, C.F.: Matrix Computations. Johns Hopkins Studies in the Mathematical Sciences, Johns Hopkins University Press, Baltimore (1996)zbMATHGoogle Scholar
  25. 25.
    Grippo, L., Palagi, L., Piccialli, V.: An unconstrained minimization method for solving low-rank sdp relaxations of the maxcut problem. Math. Program., 1–28 (2009). doi: 10.1007/s10107-009-0275-8
  26. 26.
    Grubišić, I., Pietersz, R.: Efficient rank reduction of correlation matrices. Linear Algebra Appl. 422, 629–653 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  27. 27.
    He, S., Li, Z., Zhang, S.: Approximation algorithms for homogeneous polynomial optimization with quadratic constraints. Math. Program. 125, 353–383 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  28. 28.
    Helmberg, C., Rendl, F.: A spectral bundle method for semidefinite programming. SIAM J. Optim. 10, 673–696 (2000)MathSciNetCrossRefzbMATHGoogle Scholar
  29. 29.
    Helmke, U., Moore, J.B.: Optimization and dynamical systems. In: Brockett, R. (ed.) Communications and Control Engineering Series. Springer, London (1994)Google Scholar
  30. 30.
    Horn, R.A., Johnson, C.R.: Matrix Analysis. Cambridge University Press, Cambridge (1985)CrossRefzbMATHGoogle Scholar
  31. 31.
    Kokiopoulou, E., Chen, J., Saad, Y.: Trace Optimization and Eigenproblems in Dimension Reduction Methods, Technical report. University of Minnesota (2010)Google Scholar
  32. 32.
    Kružík, M., Prohl, A.: Recent developments in the modeling, analysis, and numerics of ferromagnetism. SIAM Rev. 48, 439–483 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  33. 33.
    Laska, J.N., Wen, Z., Yin, W., Baraniuk, R.G.: Trust, but Verify: Fast and Accurate Signal Recovering from 1-bit Compressive Measurements, Technical report. Rice University (2010)Google Scholar
  34. 34.
    Lu, Z., Zhang, Y.: An Augmented Lagrangian Approach for Sparse Principal Component, Analysis, arXiv:0907.2079 (2009)Google Scholar
  35. 35.
    Malick, J., Povh, J., Rendl, F., Wiegele, A.: Regularization methods for semidefinite programming. SIAM J. Optim. 20, 336–356 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  36. 36.
    Moré, J.J., Thuente, D.J.: Line search algorithms with guaranteed sufficient decrease. ACM Trans. Math. Softw. 20, 286–307 (1994)CrossRefzbMATHGoogle Scholar
  37. 37.
    Nemirovski, A.: Sums of random symmetric matrices and quadratic optimization under orthogonality constraints. Math. Program. 109, 283–317 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  38. 38.
    Nie, J.: Regularization Methods for Sum of Squares Relaxations in Large Scale Polynomial Optimization, Technical report. Department of Mathematics, UCSD (2009)Google Scholar
  39. 39.
    Nishimori, Y., Akaho, S.: Learning algorithms utilizing quasi-geodesic flows on the stiefel manifold. Neurocomputing 67, 106–135 (2005)CrossRefGoogle Scholar
  40. 40.
    Nocedal, J., Wright, S.J.: Numerical Optimization. Springer Series in Operations Research and Financial Engineering, 2nd edn. Springer, New York (2006)Google Scholar
  41. 41.
    Owren, B., Welfert, B.: The Newton iteration on Lie groups. BIT 40, 121–145 (2000)MathSciNetCrossRefzbMATHGoogle Scholar
  42. 42.
    Pietersz, R., Groenen, P.J.F.: Rank reduction of correlation matrices by majorization. Quant. Finance 4, 649–662 (2004)MathSciNetCrossRefGoogle Scholar
  43. 43.
    Qi, C., Gallivan, K.A., Absil, P.-A.: Riemannian BFGS algorithm with applications. In: Diehl, M., Glineur, F., Jarlebring, E., Michiels, W. (eds.) Recent Advances in Optimization and its Applications in Engineering. Springer, Berlin (2010)Google Scholar
  44. 44.
    Raydan, M.: The Barzilai and Borwein gradient method for the large scale unconstrained minimization problem. SIAM J. Optim. 7, 26–33 (1997)MathSciNetCrossRefzbMATHGoogle Scholar
  45. 45.
    Rebonato, R., Jäckel, P.: The most general methodology to create a valid correlation matrix for risk management and option pricing purposes. J. Risk 2, 17–27 (1999)Google Scholar
  46. 46.
    Roy, R., Kailath, T.: Esprit—estimation of signal parameters via rotational invariance techniques. IEEE Trans. Acoust, Speech, Signal Process. 37(7), 984–995 (1989)Google Scholar
  47. 47.
    Schneider, R., Rohwedder, T., Neelov, A., Blauert, J.: Direct minimization for calculating invariant subspaces in density functional computations of the electronic structure. J. Comput. Math. 27, 360–387 (2009)MathSciNetzbMATHGoogle Scholar
  48. 48.
    Shub, M.: Some remarks on dynamical systems and numerical analysis. In Dynamical Systems and Partial Differential Equations (Caracas, 1984), pp. 69–91. Univ. Simon Bolivar, Caracas (1986)Google Scholar
  49. 49.
    Simon, D., Abell, J.: A majorization algorithm for constrained correlation matrix approximation. Linear Algebra Appl. 432, 1152–1164 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  50. 50.
    Smith, S.T.: Geometric Optimization Methods for Adaptive Filtering. ProQuest LLC, Ann Arbor, MI. PhD Thesis, Harvard University (1993)Google Scholar
  51. 51.
    Smith, S.T.: Optimization techniques on Riemannian manifolds. In Hamiltonian and Gradient Flows, Algorithms and Control, vol. 3, pp. 113–136. Fields Inst. Commun., Amer. Math. Soc., Providence, RI (1994)Google Scholar
  52. 52.
    Sun, W., Yuan, Y.-X.: Optimization Theory and Methods, vol. 1 of Springer Optimization and its Applications. Springer, New York (2006)Google Scholar
  53. 53.
    Udrişte, C.: Convex Functions and Optimization Methods on Riemannian Manifolds, vol. 297 of Mathematics and its Applications. Kluwer Academic Publishers Group, Dordrecht (1994)CrossRefGoogle Scholar
  54. 54.
    Vese, L.A., Osher, S.J.: Numerical methods for \(p\)-harmonic flows and applications to image processing. SIAM J. Numer. Anal. 40, 2085–2104 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  55. 55.
    Wächter, A., Biegler, L.T.: On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming. Math. Program. 106, 25–57 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  56. 56.
    Ward, J.: Space-Time Adaptive Processing for Airborne Radar, Technical report of MIT (1994)Google Scholar
  57. 57.
    Weber, V., VandeVondele, J., Hütter, J., Niklasson, A.M.: Direct energy functional minimization under orthogonality constraints. J. Chem. Phys. 128(8), 084113 (2008)Google Scholar
  58. 58.
    Witten, D.M., Tibshirani, R., Hastie, T.: A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis. Biostatistics 10, 515–534 (2009)CrossRefGoogle Scholar
  59. 59.
    Yang, B.: Projection approximation subspace tracking. IEEE Trans. Signal Process. 43, 95–107 (1995)CrossRefzbMATHGoogle Scholar
  60. 60.
    Yang, C., Meza, J.C., Lee, B., Wang, L.-W.: Kssolv-a matlab toolbox for solving the kohn-sham equations. ACM Trans. Math. Softw. 36, 1–35 (2009)MathSciNetCrossRefGoogle Scholar
  61. 61.
    Yang, C., Meza, J.C., Wang, L.-W.: A constrained optimization algorithm for total energy minimization in electronic structure calculations. J. Comput. Phys. 217, 709–721 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  62. 62.
    Zhang, H., Hager, W.W.: A nonmonotone line search technique and its application to unconstrained optimization. SIAM J. Optim. 14, 1043–1056 (2004)MathSciNetCrossRefzbMATHGoogle Scholar
  63. 63.
    Zhao, X., Sun, D., Toh, K.: A Newton-CG augmented lagrangian method for semidefinite programming. SIAM J. Optim. 20, 1737–1765 (2010)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer and Mathematical Optimization Society 2012

Authors and Affiliations

  1. 1.Department of Mathematics and Institute of Natural SciencesShanghai Jiaotong UniversityShanghaiChina
  2. 2.Department of Computational and Applied MathematicsRice UniversityHoustonUSA

Personalised recommendations