Skip to main content
SpringerLink
Log in

Global Optimization with Orthogonality Constraints via Stochastic Diffusion on Manifold

  • Published:
Journal of Scientific Computing Aims and scope Submit manuscript

Abstract

Orthogonality constrained optimization is widely used in applications from science and engineering. Due to the non-convex orthogonality constraints, many numerical algorithms often can hardly achieve the global optimality. We aim at establishing an efficient scheme for finding global minimizers under one or more orthogonality constraints. The main concept is based on the noisy gradient flow constructed from stochastic differential equations (SDEs) on the Stiefel manifold, the differential geometric characterization of orthogonality constraints. We derive an explicit representation of SDE on the Stiefel manifold endowed with a canonical metric and propose a numerically efficient scheme to simulate this SDE based on Cayley transformation with a theoretical convergence guarantee. The convergence to global optimizers is proved under second-order continuity. The effectiveness and efficiency of the proposed algorithms are demonstrated on a variety of problems including homogeneous polynomial optimization, bi-quadratic optimization, stability number computation, and 3D structure determination from common lines in Cryo-EM.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. Note that the SDE (7) is equivalent to the one with Itô’s calculus, i.e., \({\mathrm {d}}x(t) = -\nabla f(x(t)) {\mathrm {d}}t + \sigma (t) {\mathrm {d}}B(t)\) since the diffusion coefficient does not involve the space term. The latter is often used in literature due to the popularity and the ease of formulation of Itô’s calculus. We intend to adopt the Stratonovich calculus in order to keep consistency with the manifold-constrained case discussed in subsequent sections.

  2. For example, when we say we compute m runs of RS-local with N cycles for each, we actually generate mN random initial points and call local solver for mN times, and every N instances are grouped together to give the best solution within one run.

References

  1. Absil, P.A., Mahony, R., Sepulchre, R.: Optimization Algorithms on Matrix Manifolds. Princeton University Press, Princeton (2008)

    Book  MATH  Google Scholar 

  2. Aharon, M., Elad, M., Bruckstein, A.: K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans. Signal Process. 54, 4311–4322 (2006)

    Article  MATH  Google Scholar 

  3. Aluffi-Pentini, F., Parisi, V., Zirilli, F.: Global optimization and stochastic differential equations. J. Optim. Theory Appl. 47, 1–16 (1985)

    Article  MathSciNet  MATH  Google Scholar 

  4. Barzilai, J., Borwein, J.M.: Two-point step size gradient methods. IMA J. Numer. Anal. 8, 141–148 (1988)

    Article  MathSciNet  MATH  Google Scholar 

  5. Boufounos, P.T., Baraniuk, R.G.: 1-Bit compressive sensing. In: 42nd Annual Conference on Information Sciences and Systems, CISS 2008, pp. 16–21. IEEE (2008)

  6. Cai, J.-F., Ji, H., Shen, Z., Ye, G.-B.: Data-driven tight frame construction and image denoising. Appl. Comput. Harmon. Anal. 37, 89–105 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  7. Chiang, T.-S., Hwang, C.-R., Sheu, S.J.: Diffusion for global optimization in \(\mathbb{R}^n \). SIAM J. Control Optim. 25, 737–753 (1987)

    Article  MathSciNet  MATH  Google Scholar 

  8. Chow, S.-N., Yang, T.-S., Zhou, H.-M.: Global optimizations by intermittent diffusion. In: Adamatzky, A., Chen, G. (eds.) Chaos, CNN, Memristors and Beyond, pp. 466–479. World Scientific, Singapore (2013)

    Chapter  Google Scholar 

  9. Edelman, A., Arias, T.A., Smith, S.T.: The geometry of algorithms with orthogonality constraint. SIAM J. Matrix Anal. Appl. 20, 303–353 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  10. Geman, S., Hwang, C.-R.: Diffusions for global optimization. SIAM J. Control Optim. 24, 1031–1043 (1986)

    Article  MathSciNet  MATH  Google Scholar 

  11. Gidas, B.: Global optimization via the Langevin equation. In: 24th IEEE Conference on Decision and Control, pp. 774–778. IEEE (1985)

  12. Goldfarb, D., Wen, Z., Yin, W.: A curvilinear search method for p-harmonic flows on spheres. SIAM J. Imaging Sci. 2, 84–109 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  13. Gu, X., Yau, S.-T.: Global conformal surface parameterization. In: Proceedings of the 2003 Eurographics/ACM SIGGRAPH Symposium on Geometry Processing, pp. 127–137. Eurographics Association (2003)

  14. Hairer, E., Lubich, C., Wanner, G.: Geometric Numerical Integration: Structure-Preserving Algorithms for Ordinary Differential Equations, vol. 31. Springer, Berlin (2006)

    MATH  Google Scholar 

  15. Helmberg, C., Rendl, F.: A spectral bundle method for semidefinite programming. SIAM J. Optim. 10, 673–696 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  16. Henkel, O.: Sphere-packing bounds in the Grassmann and Stiefel manifolds. Trans. Inf. Theory 51, 3445–3456 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  17. Hsu, E.: Stochastic Analysis on Manifolds, Vol. 38 of Graduate Studies in Mathematics, vol. 38. American Mathematical Society, Providence (2002)

    Google Scholar 

  18. Ksendal, B.Ø.: Stochastic Differential Equations, Universitext, 6th edn. Springer, Berlin (2003)

    Book  Google Scholar 

  19. Lai, R., Wen, Z., Yin, W., Gu, X., Lui, L.M.: Folding-free global conformal mapping for genus-0 surfaces by harmonic energy minimization. J. Sci. Comput. 58, 705–725 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  20. Laska, J.N., Wen, Z., Yin, W., Baraniuk, R.G.: Trust, but verify: fast and accurate signal recovery from 1-bit compressive measurements. IEEE Trans. Signal Process. 59, 5289–5301 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  21. Lin, S.-Y., Luskin, M.: Relaxation methods for liquid crystal problems. SIAM J. Numer. Anal. 26, 1310–1324 (1989)

    Article  MathSciNet  MATH  Google Scholar 

  22. Ling, C., Nie, J., Qi, L., Ye, Y.: Biquadratic optimization over unit spheres and semidefinite programming relaxations. SIAM J. Optim. 20, 1286–1310 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  23. Liu, X., Srivastava, A.: Stochastic search for optimal linear representations of images on spaces with orthogonality constraints. In: Rangarajan, A., Figueiredo, M., Zerubia, J. (eds.) Lecture Notes in Computer Science, pp. 3–20. Springer, Berlin (2003)

    Google Scholar 

  24. Liu, X., Srivastava, A., Gallivan, K.: Optimal linear representations of images for object recognition. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Proceedings. IEEE Computer Society (2003)

  25. Liu, X., Wen, Z., Wang, X., Ulbrich, M., Yuan, Y.: On the analysis of the discretized Kohn–Sham density functional theory. SIAM J. Numer. Anal. 53, 1758–1785 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  26. Malham, S.J., Wiese, A.: Stochastic lie group integrators. SIAM J. Sci. Comput. 30, 597–617 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  27. Markowich, P.A., Villani, C.: On the trend to equilibrium for the Fokker–Planck equation: an interplay between physics and functional analysis. Mat. Contemp. 19, 1–29 (2000)

    MathSciNet  MATH  Google Scholar 

  28. Motzkin, T.S., Straus, E.G.: Maxima for graphs and a new proof of a theorem of Turán. Can. J. Math. 17, 533–540 (1965)

    Article  MATH  Google Scholar 

  29. Munthe-Kaas, H.: Runge–Kutta methods on lie groups. BIT Numer. Math. 38, 92–111 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  30. Nocedal, J., Wright, S.J.: Numerical Optimization, Springer Series in Operations Research and Financial Engineering, 2nd edn. Springer, New York (2006)

    Google Scholar 

  31. Oprea, J.: Differential Geometry and Its Applications, Classroom Resource Materials Series, 2nd edn. Mathematical Association of America, Washington (2007)

    Google Scholar 

  32. Ozoliņš, V., Lai, R., Caflisch, R., Osher, S.: Compressed modes for variational problems in mathematics and physics. Proc. Natl. Acad. Sci. 110, 18368–18373 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  33. Parpas, P., Rustem, B.: An algorithm for the global optimization of a class of continuous minimax problems. J. Optim. Theory Appl. 141, 461–473 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  34. Parpas, P., Rustem, B.: Convergence analysis of a global optimization algorithm using stochastic differential equations. J. Glob. Optim. 45, 95–110 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  35. Parpas, P., Rustem, B., Pistikopoulos, E.N.: Linearly constrained global optimization and stochastic differential equations. J. Glob. Optim. 36, 191–217 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  36. Parpas, P., Rustem, B., Pistikopoulos, E.N.: Global optimization of robust chance constrained problems. J. Glob. Optim. 43, 231–247 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  37. Rubinshtein, E., Srivastava, A.: Optimal linear projections for enhancing desired data statistics. Stat. Comput. 20, 267–282 (2009)

    Article  MathSciNet  Google Scholar 

  38. Singer, A., Shkolnisky, Y.: Three-dimensional structure determination from common lines in Cryo-EM by eigenvectors and semidefinite programming. SIAM J. Imaging Sci. 4, 543–572 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  39. Sloane, NJA.: Challenge problems: independent sets in graphs. https://oeis.org/A265032/a265032.html (2015). Accessed May 2019

  40. Stratonovich, R.L.: A new representation for stochastic integrals and equations. SIAM J. Control 4, 362–371 (1966)

    Article  MathSciNet  MATH  Google Scholar 

  41. Tang, B., Sapiro, G., Caselles, V.: Color image enhancement via chromaticity diffusion. IEEE Trans. Image Process. 10, 701–707 (2001)

    Article  MATH  Google Scholar 

  42. Vese, L.A., Osher, S.J.: Numerical methods for p-harmonic flows and applications to image processing. SIAM J. Numer. Anal. 40, 2085–2104 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  43. Villani, C.: Optimal Transport, Vol. 338 of Grundlehren der mathematischen Wissenschaften, vol. 338. Springer, Berlin (2009)

    Google Scholar 

  44. Wen, Z., Milzarek, A., Ulbrich, M., Zhang, H.: Adaptive regularized self-consistent field iteration with exact Hessian for electronic structure calculation. SIAM J. Sci. Comput. 35, A1299–A1324 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  45. Wen, Z., Yin, W.: A feasible method for optimization with orthogonality constraints. Math. Program. 142, 397–434 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  46. Yin, G., Yin, K.: Global optimization using diffusion perturbations with large noise intensity. Acta Math. Appl. Sin. Engl. Ser. 22, 529–542 (2006)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

R. Lai’s work was supported in part by NSF DMS-1522645 and an NSF Career Award DMS-1752934. Z. Wen’s work was supported in part by National Natural Science Foundation of China (Grant Nos. 11831002, 91730302 and 11421101).

Author information

Authors and Affiliations

  1. Institute for Computational and Mathematical Engineering, Stanford University, Stanford, CA, 94305, USA

    Honglin Yuan

  2. H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, GA, 30332, USA

    Xiaoyi Gu

  3. Department of Mathematics, Rensselaer Polytechnic Institute, Troy, NY, 12180, USA

    Rongjie Lai

  4. Beijing International Center for Mathematical Research, Center for Data Science, and National Engineering Laboratory for Big Data Analysis and Applications, Peking University, Beijing, China

    Zaiwen Wen

Authors
  1. Honglin Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaoyi Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rongjie Lai
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zaiwen Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zaiwen Wen.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix A: Computational Results of Sect. 5.3

See Tables 1 and 2

Table 2 Computational results of stability number (ii)
Full size table

Appendix B: Computational Results of Sect. 5.4

See Tables 3 and 4.

Table 3 The MSE of the eigs, RS-local and IDDM for random dataset
Full size table
Table 4 The MSE of the eigs, RS-local and IDDM for dataset from [38]
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yuan, H., Gu, X., Lai, R. et al. Global Optimization with Orthogonality Constraints via Stochastic Diffusion on Manifold. J Sci Comput 80, 1139–1170 (2019). https://doi.org/10.1007/s10915-019-00971-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10915-019-00971-w

Keywords

Mathematics Subject Classification

Search

Navigation