Skip to main content
SpringerLink
Log in

Differential-geometric Newton method for the best rank-(R 1, R 2, R 3) approximation of tensors

  • Original Paper
  • Published:
Numerical Algorithms Aims and scope Submit manuscript

Abstract

An increasing number of applications are based on the manipulation of higher-order tensors. In this paper, we derive a differential-geometric Newton method for computing the best rank-(R 1, R 2, R 3) approximation of a third-order tensor. The generalization to tensors of order higher than three is straightforward. We illustrate the fast quadratic convergence of the algorithm in a neighborhood of the solution and compare it with the known higher-order orthogonal iteration (De Lathauwer et al., SIAM J Matrix Anal Appl 21(4):1324–1342, 2000). This kind of algorithms are useful for many problems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Absil, P.A., Ishteva, M., De Lathauwer, L., Van Huffel, S.: A geometric Newton method for Oja’s vector field. (Accepted for publication in Neural Computation). doi:10.1162/neco.2008.04-08-749 (2008)

  2. Absil, P.A., Mahony, R., Sepulchre, R.: Optimization Algorithms on Matrix Manifolds. Princeton University Press, Princeton (2008)

    MATH  Google Scholar 

  3. Acar, E., Bingol, C.A., Bingol, H., Bro, R., Yener, B.: Multiway analysis of epilepsy tensors. Bioinformatics 23(13), i10–i18 (2007) (ISMB 2007 Conference Proceedings)

    Article  Google Scholar 

  4. Adler, R.L., Dedieu, J.P., Margulies, J.Y., Martens, M., Shub, M.: Newton’s method on Riemannian manifolds and a geometric model for the human spine. IMA J. Numer. Anal. 22(3), 359–390 (2002)

    Article  MATH  MathSciNet  Google Scholar 

  5. Alvarez, F., Bolte, J., Munier, J.: A unifying local convergence result for Newton’s method in Riemannian manifolds. Found. Comput. Math. 8(2), 197–226 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  6. Baranyi, P., Varkonyi-Koczy, A.R., Yam, Y., Patton, R.J.: Adaptation of TS fuzzy models without complexity expansion: HOSVD-based approach. IEEE Trans. Inst. Meas. 54(1), 52–60 (2005)

    Article  Google Scholar 

  7. Carroll, J., Chang, J.: Analysis of individual differences in multidimensional scaling via an N-way generalization of “Eckart-Young” decomposition. Psychometrika 35(3), 283–319 (1970)

    Article  MATH  Google Scholar 

  8. Comon, P.: Independent component analysis, a new concept? Signal Process. 36(3), 287–314 (1994) (Special Issue on Higher-order Statistics)

    Article  MATH  Google Scholar 

  9. Comon, P.: Tensor decompositions. In: McWhirter, J.G., Proudler, I.K. (eds.) Mathematics in Signal Processing V, pp. 1–24. Clarendon, Oxford (2002)

    Google Scholar 

  10. De Lathauwer, L.: A link between the canonical decomposition in multilinear algebra and simultaneous matrix diagonalization. SIAM J. Matrix Anal. Appl. 28(3), 642–666 (2006)

    Article  MATH  MathSciNet  Google Scholar 

  11. De Lathauwer, L.: Decompositions of a higher-order tensor in block terms—part I: lemmas for partitioned matrices. SIAM J. Matrix Anal. Appl. 30(3), 1022–1032 (2008) (Special Issue on Tensor Decompositions and Applications)

    Article  MathSciNet  Google Scholar 

  12. De Lathauwer, L.: Decompositions of a higher-order tensor in block terms—part II: definitions and uniqueness. SIAM J. Matrix Anal. Appl. 30(3), 1033–1066 (2008) (Special Issue on Tensor Decompositions and Applications)

    Article  MathSciNet  Google Scholar 

  13. De Lathauwer, L., Castaing, J.: Tensor-based techniques for the blind separation of DS-CDMA signals. Signal Process. 87(2), 322–336 (2007) (Special Issue on Tensor Signal Processing)

    Article  Google Scholar 

  14. De Lathauwer, L., De Moor, B., Vandewalle, J.: A multilinear singular value decomposition. SIAM J. Matrix Anal. Appl. 21(4), 1253–1278 (2000)

    Article  MATH  MathSciNet  Google Scholar 

  15. De Lathauwer, L., De Moor, B., Vandewalle, J.: On the best rank-1 and rank-(R 1, R 2,...,R N ) approximation of higher-order tensors. SIAM J. Matrix Anal. Appl. 21(4), 1324–1342 (2000)

    Article  MATH  MathSciNet  Google Scholar 

  16. De Lathauwer, L., De Moor, B., Vandewalle, J.: Independent component analysis and (simultaneous) third-order tensor diagonalization. IEEE Trans. Signal Process. 49(10), 2262–2271 (2001)

    Article  Google Scholar 

  17. De Lathauwer, L., De Moor, B., Vandewalle, J.: Computation of the canonical decomposition by means of a simultaneous generalized Schur decomposition. SIAM J. Matrix Anal. Appl. 26(2), 295–327 (2004)

    Article  MATH  MathSciNet  Google Scholar 

  18. De Lathauwer, L., Nion, D.: Decompositions of a higher-order tensor in block terms—part III: alternating least squares algorithms. SIAM J. Matrix Anal. Appl. 30(3), 1067–1083 (2008) (Special Issue on Tensor Decompositions and Applications)

    Article  MathSciNet  Google Scholar 

  19. De Lathauwer, L., Vandewalle, J.: Dimensionality reduction in higher-order signal processing and rank-(R 1, R 2,...,R N ) reduction in multilinear algebra. Linear Algebra Appl. 391, 31–55 (2004) (Special Issue on Linear Algebra in Signal and Image Processing)

    Article  MATH  MathSciNet  Google Scholar 

  20. De Vos, M., De Lathauwer, L., Vanrumste, B., Van Huffel, S., Van Paesschen, W.: Canonical decomposition of ictal scalp EEG and accurate source localization: principles and simulation study. J. Comput. Intell. Neurosci. 2007(Article ID 58253), 1–10 (2007) (Special Issue on EEG/MEG Signal Processing)

    Article  Google Scholar 

  21. De Vos, M., Vergult, A., De Lathauwer, L., De Clercq, W., Van Huffel, S., Dupont, P., Palmini, A., Van Paesschen, W.: Canonical decomposition of ictal scalp EEG reliably detects the seizure onset zone. NeuroImage 37(3), 844–854 (2007)

    Article  Google Scholar 

  22. Dedieu, J.P., Priouret, P., Malajovich, G.: Newton’s method on Riemannian manifolds: covariant alpha theory. IMA J. Numer. Anal. 23(3), 395–419 (2003)

    Article  MATH  MathSciNet  Google Scholar 

  23. Edelman, A., Arias, T.A., Smith, S.T.: The geometry of algorithms with orthogonality constraints. SIAM J. Matrix Anal. Appl. 20, 303–353 (1998)

    Article  MATH  MathSciNet  Google Scholar 

  24. Elad, M., Milanfar, P., Golub, G.H.: Shape from moments—an estimation theory perspective. IEEE Trans. Signal Process. 52, 1814–1829 (2004)

    Article  MathSciNet  Google Scholar 

  25. Eldén, L., Savas, B.: A Newton–Grassmann method for computing the best multi-linear rank-(r 1, r 2, r 3) approximation of a tensor. Tech. Rep. LITH-MAT-R-2007-6-SE, Department of Mathematics, Linköpings Universitet (2007)

  26. Golub, G.H., Van Loan, C.F.: Matrix Computations, 3rd edn. Johns Hopkins University Press, Baltimore (1996)

    MATH  Google Scholar 

  27. Harshman, R.A.: Foundations of the PARAFAC procedure: model and conditions for an “explanatory” multi-mode factor analysis. UCLA Work. Pap. Phon. 16(1), 1–84 (1970)

    Google Scholar 

  28. Helmke, U., Moore, J.B.: Optimization and Dynamical Systems. Springer, New York (1994)

    Google Scholar 

  29. Hitchcock, F.L.: The expression of a tensor or a polyadic as a sum of products. J. Math. Phys. 6(1), 164–189 (1927)

    Google Scholar 

  30. Hitchcock, F.L.: Multiple invariants and generalized rank of a p-way matrix or tensor. J. Math. Phys. 7(1), 39–79 (1927)

    Google Scholar 

  31. Khoromskij, B.N., Khoromskaia, V.: Low rank Tucker-type tensor approximation to classical potentials. Cent. Eur. J. Math. 5(3), 523–550 (2007)

    Article  MATH  MathSciNet  Google Scholar 

  32. Kolda, T.: Orthogonal tensor decompositions. SIAM J. Matrix Anal. Appl. 23, 243–255 (2001)

    Article  MATH  MathSciNet  Google Scholar 

  33. Kroonenberg, P.M.: Applied Multiway Data Analysis. Wiley, New York (2008)

    Book  MATH  Google Scholar 

  34. Kroonenberg, P.M., de Leeuw, J.: Principal component analysis of three-mode data by means of alternating least squares algorithms. Psychometrika 45(1), 69–97 (1980)

    Article  MATH  MathSciNet  Google Scholar 

  35. Li, C., Wang, J.: Newton’s method on Riemannian manifolds: Smale’s point estimate theory under the γ-condition. IMA J. Numer. Anal. 26(2), 228–251 (2006)

    Article  MathSciNet  Google Scholar 

  36. Moravitz Martin, C.D., Van Loan, C.F.: A Jacobi-type method for computing orthogonal tensor decompositions. SIAM J. Matrix Anal. Appl. 30, 1219–1232 (2008)

    Article  MathSciNet  Google Scholar 

  37. Papy, J.M., De Lathauwer, L., Van Huffel, S.: Exponential data fitting using multilinear algebra: the decimative case. Technical report 04-70, ESAT-SISTA, K.U.Leuven (2004)

  38. Papy, J.M., De Lathauwer, L., Van Huffel, S.: Exponential data fitting using multilinear algebra: the single-channel and the multichannel case. Numer. Linear Algebra Appl. 12(8), 809–826 (2005)

    Article  MATH  MathSciNet  Google Scholar 

  39. Savas, B., Lim, L.H.: Best multilinear rank approximation of tensors with quasi-Newton methods on Grassmannians. Tech. Rep. LITH-MAT-R-2008-01-SE, Department of Mathematics, Linköpings Universitet (2008)

  40. Sidiropoulos, N., Bro, R., Giannakis, G.: Parallel factor analysis in sensor array processing. IEEE Trans. Signal Process. 48, 2377–2388 (2000)

    Article  Google Scholar 

  41. Smilde, A., Bro, R., Geladi, P.: Multi-way Analysis. Applications in the Chemical Sciences. Wiley, Chichester (2004)

    Book  Google Scholar 

  42. Tucker, L.R.: The extension of factor analysis to three-dimensional matrices. In: Gulliksen, H., Frederiksen, N. (eds.) Contributions to Mathematical Psychology, pp. 109–127. Holt, Rinehart & Winston, New York (1964)

    Google Scholar 

  43. Tucker, L.R.: Some mathematical notes on three-mode factor analysis. Psychometrika 31, 279–311 (1966)

    Article  MathSciNet  Google Scholar 

  44. Udrişte, C.: Convex Functions and Optimization Methods on Riemannian Manifolds. Kluwer, Dordrecht (1994)

    MATH  Google Scholar 

  45. Vasilescu, M.A.O., Terzopoulos, D.: Multilinear subspace analysis for image ensembles. In: Proc. Computer Vision and Pattern Recognition Conf. (CVPR ’03), vol. 2, pp. 93–99. CVPR, Madison (2003)

    Google Scholar 

  46. Zhang, T., Golub, G.H.: Rank-one approximation to high order tensors. SIAM J. Matrix Anal. Appl. 23, 534–550 (2001)

    Article  MATH  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. ESAT/SCD, Katholieke Universiteit Leuven, Kasteelpark Arenberg 10, B-3001, Leuven, Belgium

    Mariya Ishteva, Lieven De Lathauwer & Sabine Van Huffel

  2. Subfaculty Science and Technology, Katholieke Universiteit Leuven Campus Kortrijk, B-8500, Kortrijk, Belgium

    Lieven De Lathauwer

  3. Department of Mathematical Engineering, Université catholique de Louvain, B-1348, Louvain-la-Neuve, Belgium

    P.-A. Absil

Authors
  1. Mariya Ishteva
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lieven De Lathauwer
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. P.-A. Absil
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sabine Van Huffel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mariya Ishteva.

Additional information

This paper presents research results of the Belgian Network DYSCO (Dynamical Systems, Control, and Optimization), funded by the Interuniversity Attraction Poles Programme, initiated by the Belgian State, Science Policy Office. The scientific responsibility rests with its authors. Research supported by: (1) Research Council K.U.Leuven: GOA-Ambiorics, CoE EF/05/006 Optimization in Engineering (OPTEC), (2) F.W.O.: (a) project G.0321.06, (b) Research Communities ICCoS, ANMMM and MLDM, (3) the Belgian Federal Science Policy Office: IUAP P6/04 (DYSCO, “Dynamical systems, control and optimization”, 2007–2011), (4) EU: ERNSI. M. Ishteva is supported by a K.U.Leuven doctoral scholarship (OE/06/25, OE/07/17, OE/08/007), L. De Lathauwer is supported by “Impulsfinanciering Campus Kortrijk (2007–2012)(CIF1)” and STRT1/08/023.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ishteva, M., De Lathauwer, L., Absil, PA. et al. Differential-geometric Newton method for the best rank-(R 1, R 2, R 3) approximation of tensors. Numer Algor 51, 179–194 (2009). https://doi.org/10.1007/s11075-008-9251-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11075-008-9251-2

Keywords

Search

Navigation