Skip to main content

Optimization on flag manifolds

Mathematical Programming (2021)Cite this article

Abstract

A flag is a sequence of nested subspaces. Flags are ubiquitous in numerical analysis, arising in finite elements, multigrid, spectral, and pseudospectral methods for numerical pde; they arise in the form of Krylov subspaces in matrix computations, and as multiresolution analysis in wavelets constructions. They are common in statistics too—principal component, canonical correlation, and correspondence analyses may all be viewed as methods for extracting flags from a data set. The main goal of this article is to develop the tools needed for optimizing over a set of flags, which is a smooth manifold called the flag manifold, and it contains the Grassmannian as the simplest special case. We will derive closed-form analytic expressions for various differential geometric objects required for Riemannian optimization algorithms on the flag manifold; introducing various systems of extrinsic coordinates that allow us to parameterize points, metrics, tangent spaces, geodesics, distances, parallel transports, gradients, Hessians in terms of matrices and matrix operations; and thereby permitting us to formulate steepest descent, conjugate gradient, and Newton algorithms on the flag manifold using only standard numerical linear algebra.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price includes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2

Notes

  1. Discounting manifolds that can be realized as products or open subsets of these manifolds, e.g., those considered in [1, 31, 35, 44].

  2. More precisely, the covariant derivative associated with the Levi-Civita connection on M.

References

  1. Absil, P.A., Amodei, L., Meyer, G.: Two Newton methods on the manifold of fixed-rank matrices endowed with Riemannian quotient geometries. Comput. Stat. 29(3–4), 569–590 (2014)

    MathSciNet  MATH  Article  Google Scholar 

  2. Absil, P.A., Baker, C.G., Gallivan, K.A.: Trust-region methods on Riemannian manifolds. Found. Comput. Math. 7(3), 303–330 (2007)

    MathSciNet  MATH  Article  Google Scholar 

  3. Absil, P.A., Mahony, R., Sepulchre, R.: Riemannian geometry of Grassmann manifolds with a view on algorithmic computation. Acta Appl. Math. 80(2), 199–220 (2004)

    MathSciNet  MATH  Article  Google Scholar 

  4. Absil, P.A., Mahony, R., Sepulchre, R.: Optimization Algorithms on Matrix Manifolds. Princeton University Press, Princeton, NJ (2008)

    MATH  Book  Google Scholar 

  5. Alekseevsky, D., Arvanitoyeorgos, A.: Riemannian flag manifolds with homogeneous geodesics. Trans. Am. Math. Soc. 359(8), 3769–3789 (2007)

    MathSciNet  MATH  Article  Google Scholar 

  6. Ammar, G., Martin, C.: The geometry of matrix eigenvalue methods. Acta Appl. Math. 5(3), 239–278 (1986)

    MathSciNet  MATH  Article  Google Scholar 

  7. Axler, S.: Linear Algebra Done Right. Undergraduate Texts in Mathematics, 3rd edn. Springer, Cham (2015)

    MATH  Book  Google Scholar 

  8. Balzano, L., Nowak, R., Recht, B.: Online identification and tracking of subspaces from highly incomplete information. In: 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp. 704–711. IEEE (2010)

  9. Bellman, R., Fan, K.: On systems of linear inequalities in Hermitian matrix variables. In: Proceedings on Symposium Pure Mathematics, vol. VII, pp. 1–11. American Mathematical Society, Providence, RI (1963)

  10. Boothby, W.M.: An Introduction to Differentiable Manifolds and Riemannian Geometry. Pure and Applied Mathematics, vol. 120, 2nd edn. Academic Press Inc, Orlando, FL (1986)

    MATH  Google Scholar 

  11. Borel, A.: La cohomologie mod \(2\) de certains espaces homogènes. Comment. Math. Helv. 27, 165–197 (1953)

    MathSciNet  MATH  Article  Google Scholar 

  12. Borel, A.: Sur la cohomologie des espaces fibrés principaux et des espaces homogènes de groupes de Lie compacts. Ann. Math. 2(57), 115–207 (1953)

    MATH  Article  Google Scholar 

  13. Borel, A.: Linear Algebraic Groups. Graduate Texts in Mathematics, vol. 126, 2nd edn. Springer-Verlag, New York (1991)

    Book  Google Scholar 

  14. Borel, A., Serre, J.P.: Groupes de Lie et puissances réduites de Steenrod. Am. J. Math. 75, 409–448 (1953)

    MATH  Article  Google Scholar 

  15. Boyd, S., Kim, S.J., Vandenberghe, L., Hassibi, A.: A tutorial on geometric programming. Optim. Eng. 8(1), 67–127 (2007)

    MathSciNet  MATH  Article  Google Scholar 

  16. Cheeger, J., Ebin, D.G.: Comparison Theorems in Riemannian Geometry. AMS Chelsea Publishing, Providence, RI (2008)

    MATH  Google Scholar 

  17. Chern, S.S.: On the characteristic classes of complex sphere bundles and algebraic varieties. Am. J. Math. 75, 565–597 (1953)

    MathSciNet  MATH  Article  Google Scholar 

  18. Chikuse, Y.: Statistics on Special Manifolds. Lecture Notes in Statistics, vol. 174. Springer-Verlag, New York (2003)

    MATH  Book  Google Scholar 

  19. Curtef, O., Dirr, G., Helmke, U.: Riemannian optimization on tensor products of Grassmann manifolds: applications to generalized Rayleigh-quotients. SIAM J. Matrix Anal. Appl. 33(1), 210–234 (2012)

    MathSciNet  MATH  Article  Google Scholar 

  20. Edelman, A., Arias, T.A., Smith, S.T.: The geometry of algorithms with orthogonality constraints. SIAM J. Matrix Anal. Appl. 20(2), 303–353 (1999)

    MathSciNet  MATH  Article  Google Scholar 

  21. Ehresmann, C.: Sur la topologie de certains espaces homogènes. Ann. Math. (2) 35(2), 396–443 (1934)

    MathSciNet  MATH  Article  Google Scholar 

  22. Gabay, D.: Minimizing a differentiable function over a differential manifold. J. Optim. Theory Appl. 37(2), 177–219 (1982)

    MathSciNet  MATH  Article  Google Scholar 

  23. Goodman, R., Wallach, N.R.: Symmetry, Representations, and Invariants, Graduate Texts in Mathematics, vol. 255. Springer, Dordrecht (2009)

    MATH  Book  Google Scholar 

  24. Grassmann, H.: A New Branch of Mathematics. Open Court Publishing Co., Chicago, IL (1995)

    MATH  Google Scholar 

  25. Helgason, S.: Differential Geometry, Lie Groups, and Symmetric Spaces, Graduate Studies in Mathematics, vol. 34. American Mathematical Society, Providence, RI (2001)

    MATH  Google Scholar 

  26. Helmke, U., Hüper, K., Trumpf, J.: Newton’s method on Graßmann manifolds. Preprint (2007). arxiv:0709.2205

  27. Howe, R., Lee, S.T.: Spherical harmonics on Grassmannians. Colloq. Math. 118(1), 349–364 (2010)

    MathSciNet  MATH  Article  Google Scholar 

  28. Jordan, J., Helmke, U.: Controllability of the QR-algorithm on Hessenberg flags. In: Proceeding of the Fifteenth International Symposium on Mathematical Theory of Network and Systems (MTNS 2002) (2002)

  29. Kobayashi, S., Nomizu, K.: Foundations of Differential Geometry. Vols. I, II. Wiley Classics Library. Wiley, New York (1996)

    Google Scholar 

  30. Kowalski, O., Szenthe, J.: Erratum: “On the existence of homogeneous geodesics in homogeneous Riemannian manifolds” [Geom. Dedicata 81 (2000), no. 1-3, 209–214; MR1772203 (2001f:53104)]. Geom. Dedicata 84(1–3), 331–332 (2001)

    MathSciNet  Article  Google Scholar 

  31. Lim, L.H., Wong, K.S.W., Ye, K.: Numerical algorithms on the affine Grassmannian. SIAM J. Matrix Anal. Appl. 40(2), 371–393 (2019)

    MathSciNet  MATH  Article  Google Scholar 

  32. Lim, L.H., Ye, K.: Numerical algorithms on the flag manifold. Preprint (2019)

  33. Lundström, E., Eldén, L.: Adaptive eigenvalue computations using Newton’s method on the Grassmann manifold. SIAM J. Matrix Anal. Appl. 23(3), 819–839 (2001/02)

  34. Mardia, K.V., Kent, J.T., Bibby, J.M.: Multivariate Analysis. Probability and Mathematical Statistics: A Series of Monographs and Textbooks. Academic Press, London (1979)

    Google Scholar 

  35. Massart, E., Absil, P.A.: Quotient geometry with simple geodesics for the manifold of fixed-rank positive-semidefinite matrices. SIAM J. Matrix Anal. Appl. 41(1), 171–198 (2020)

    MathSciNet  MATH  Article  Google Scholar 

  36. Monk, D.: The geometry of flag manifolds. Proc. Lond. Math. Soc. 3(9), 253–286 (1959)

    MathSciNet  MATH  Article  Google Scholar 

  37. Nicolaescu, L.I.: Lectures on the Geometry of Manifolds. World Scientific Publishing Co. Pte. Ltd., Hackensack, NJ (2007)

    MATH  Book  Google Scholar 

  38. Nishimori, Y., Akaho, S., Plumbley, M.D.: Riemannian optimization method on the flag manifold for independent subspace analysis. In: International Conference on Independent Component Analysis and Signal Separation, pp. 295–302. Springer (2006)

  39. Nishimori, Y., Akaho, S., Plumbley, M.D.: Natural conjugate gradient on complex flag manifolds for complex independent subspace analysis. In: International Conference on Artificial Neural Networks, pp. 165–174. Springer (2008)

  40. Nocedal, J., Wright, S.J.: Numerical Optimization. Springer Series in Operations Research and Financial Engineering, 2nd edn. Springer, New York (2006)

    Google Scholar 

  41. O’Neill, B.: Semi-Riemannian Geometry, Pure and Applied Mathematics, vol. 103. Academic Press Inc, New York (1983)

    Google Scholar 

  42. Pennec, X.: Barycentric subspace analysis on manifolds. Ann. Stat. 46(6A), 2711–2746 (2018)

    MathSciNet  MATH  Article  Google Scholar 

  43. Ring, W., Wirth, B.: Optimization methods on Riemannian manifolds and their application to shape space. SIAM J. Optim. 22(2), 596–627 (2012)

    MathSciNet  MATH  Article  Google Scholar 

  44. Savas, B., Lim, L.H.: Quasi-Newton methods on Grassmannians and multilinear approximations of tensors. SIAM J. Sci. Comput. 32(6), 3352–3393 (2010)

    MathSciNet  MATH  Article  Google Scholar 

  45. Schulz, V.H.: A Riemannian view on shape optimization. Found. Comput. Math. 14(3), 483–501 (2014)

    MathSciNet  MATH  Article  Google Scholar 

  46. Stiefel, E.: Richtungsfelder und Fernparallelismus in n-dimensionalen Mannigfaltigkeiten. Comment. Math. Helv. 8(1), 305–353 (1935)

    MathSciNet  MATH  Article  Google Scholar 

  47. Tojo, K.: Totally geodesic submanifolds of naturally reductive homogeneous spaces. Tsukuba J. Math. 20(1), 181–190 (1996)

    MathSciNet  MATH  Article  Google Scholar 

  48. Vandereycken, B.: Low-rank matrix completion by Riemannian optimization. SIAM J. Optim. 23(2), 1214–1236 (2013)

    MathSciNet  MATH  Article  Google Scholar 

  49. Wang, L., Wang, X., Feng, J.: Subspace distance analysis with application to adaptive Bayesian algorithm for face recognition. Pattern Recogn. 39(3), 456–464 (2006)

    MATH  Article  Google Scholar 

  50. Wang, R., Shan, S., Chen, X., Gao, W.: Manifold-manifold distance with application to face recognition based on image set. In: IEEE Conference on Computer Vision and Pattern Recognition, 2008. CVPR 2008, pp. 1–8. IEEE (2008)

  51. Wen, Z., Yin, W.: A feasible method for optimization with orthogonality constraints. Math. Program. 142(1–2, Ser. A), 397–434 (2013)

    MathSciNet  MATH  Article  Google Scholar 

Download references

Acknowledgements

We thank the two anonymous referees for their exceptionally helpful suggestions and comments. KY is partially supported by National Key Research and Development Program of China No. 2018YFA0306702 and National Key Research and Development Program of China No. 2020YFA0712300, NSFC Grant No. 11801548 and NSFC Grant No. 11688101. LHL is supported by DARPA D15AP00109, HR00112190040, NSF IIS 1546413, DMS 1854831.

Author information

Authors and Affiliations

  1. KLMM, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing, 100190, China

    Ke Ye

  2. Department of Statistics, University of Chicago, Chicago, IL, 60637-1514, USA

    Ken Sze-Wai Wong

  3. Computational and Applied Mathematics Initiative, Department of Statistics, University of Chicago, Chicago, IL, 60637-1514, USA

    Lek-Heng Lim

Authors
  1. Ke Ye
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ken Sze-Wai Wong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lek-Heng Lim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lek-Heng Lim.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

KY is supported by NSFC Grant No. 11801548, NSFC Grant No. 11688101, and National Key R&D Program of China Grant No. 2018YFA0306702 and 2020YFA0712300. LHL is supported by DARPA D15AP00109 and HR00112190040, NSF IIS 1546413, DMS 1854831.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Ye, K., Wong, K.SW. & Lim, LH. Optimization on flag manifolds. Math. Program. (2021). https://doi.org/10.1007/s10107-021-01640-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10107-021-01640-3

Keywords

  • Flag manifold
  • Riemannian optimization
  • Manifold optimization
  • Multiscale
  • Multiresolution

Mathematics Subject Classification

  • 62H12
  • 14M15
  • 90C30
  • 62H10
  • 68T10