Numerical Algorithms

, Volume 51, Issue 2, pp 179–194 | Cite as

Differential-geometric Newton method for the best rank-(R1, R2, R3) approximation of tensors

  • Mariya Ishteva
  • Lieven De Lathauwer
  • P.-A. Absil
  • Sabine Van Huffel
Original Paper

Abstract

An increasing number of applications are based on the manipulation of higher-order tensors. In this paper, we derive a differential-geometric Newton method for computing the best rank-(R1, R2, R3) approximation of a third-order tensor. The generalization to tensors of order higher than three is straightforward. We illustrate the fast quadratic convergence of the algorithm in a neighborhood of the solution and compare it with the known higher-order orthogonal iteration (De Lathauwer et al., SIAM J Matrix Anal Appl 21(4):1324–1342, 2000). This kind of algorithms are useful for many problems.

Keywords

Multilinear algebra Higher-order tensor Higher-order singular value decomposition Rank-(R1, R2, R3) reduction Quotient manifold Differential-geometric optimization Newton’s method Tucker compression 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Absil, P.A., Ishteva, M., De Lathauwer, L., Van Huffel, S.: A geometric Newton method for Oja’s vector field. (Accepted for publication in Neural Computation). doi:10.1162/neco.2008.04-08-749 (2008)
  2. 2.
    Absil, P.A., Mahony, R., Sepulchre, R.: Optimization Algorithms on Matrix Manifolds. Princeton University Press, Princeton (2008)MATHGoogle Scholar
  3. 3.
    Acar, E., Bingol, C.A., Bingol, H., Bro, R., Yener, B.: Multiway analysis of epilepsy tensors. Bioinformatics 23(13), i10–i18 (2007) (ISMB 2007 Conference Proceedings)CrossRefGoogle Scholar
  4. 4.
    Adler, R.L., Dedieu, J.P., Margulies, J.Y., Martens, M., Shub, M.: Newton’s method on Riemannian manifolds and a geometric model for the human spine. IMA J. Numer. Anal. 22(3), 359–390 (2002)MATHCrossRefMathSciNetGoogle Scholar
  5. 5.
    Alvarez, F., Bolte, J., Munier, J.: A unifying local convergence result for Newton’s method in Riemannian manifolds. Found. Comput. Math. 8(2), 197–226 (2008)MATHCrossRefMathSciNetGoogle Scholar
  6. 6.
    Baranyi, P., Varkonyi-Koczy, A.R., Yam, Y., Patton, R.J.: Adaptation of TS fuzzy models without complexity expansion: HOSVD-based approach. IEEE Trans. Inst. Meas. 54(1), 52–60 (2005)CrossRefGoogle Scholar
  7. 7.
    Carroll, J., Chang, J.: Analysis of individual differences in multidimensional scaling via an N-way generalization of “Eckart-Young” decomposition. Psychometrika 35(3), 283–319 (1970)MATHCrossRefGoogle Scholar
  8. 8.
    Comon, P.: Independent component analysis, a new concept? Signal Process. 36(3), 287–314 (1994) (Special Issue on Higher-order Statistics)MATHCrossRefGoogle Scholar
  9. 9.
    Comon, P.: Tensor decompositions. In: McWhirter, J.G., Proudler, I.K. (eds.) Mathematics in Signal Processing V, pp. 1–24. Clarendon, Oxford (2002)Google Scholar
  10. 10.
    De Lathauwer, L.: A link between the canonical decomposition in multilinear algebra and simultaneous matrix diagonalization. SIAM J. Matrix Anal. Appl. 28(3), 642–666 (2006)MATHCrossRefMathSciNetGoogle Scholar
  11. 11.
    De Lathauwer, L.: Decompositions of a higher-order tensor in block terms—part I: lemmas for partitioned matrices. SIAM J. Matrix Anal. Appl. 30(3), 1022–1032 (2008) (Special Issue on Tensor Decompositions and Applications)CrossRefMathSciNetGoogle Scholar
  12. 12.
    De Lathauwer, L.: Decompositions of a higher-order tensor in block terms—part II: definitions and uniqueness. SIAM J. Matrix Anal. Appl. 30(3), 1033–1066 (2008) (Special Issue on Tensor Decompositions and Applications)CrossRefMathSciNetGoogle Scholar
  13. 13.
    De Lathauwer, L., Castaing, J.: Tensor-based techniques for the blind separation of DS-CDMA signals. Signal Process. 87(2), 322–336 (2007) (Special Issue on Tensor Signal Processing)CrossRefGoogle Scholar
  14. 14.
    De Lathauwer, L., De Moor, B., Vandewalle, J.: A multilinear singular value decomposition. SIAM J. Matrix Anal. Appl. 21(4), 1253–1278 (2000)MATHCrossRefMathSciNetGoogle Scholar
  15. 15.
    De Lathauwer, L., De Moor, B., Vandewalle, J.: On the best rank-1 and rank-(R 1, R 2,...,R N) approximation of higher-order tensors. SIAM J. Matrix Anal. Appl. 21(4), 1324–1342 (2000)MATHCrossRefMathSciNetGoogle Scholar
  16. 16.
    De Lathauwer, L., De Moor, B., Vandewalle, J.: Independent component analysis and (simultaneous) third-order tensor diagonalization. IEEE Trans. Signal Process. 49(10), 2262–2271 (2001)CrossRefGoogle Scholar
  17. 17.
    De Lathauwer, L., De Moor, B., Vandewalle, J.: Computation of the canonical decomposition by means of a simultaneous generalized Schur decomposition. SIAM J. Matrix Anal. Appl. 26(2), 295–327 (2004)MATHCrossRefMathSciNetGoogle Scholar
  18. 18.
    De Lathauwer, L., Nion, D.: Decompositions of a higher-order tensor in block terms—part III: alternating least squares algorithms. SIAM J. Matrix Anal. Appl. 30(3), 1067–1083 (2008) (Special Issue on Tensor Decompositions and Applications)CrossRefMathSciNetGoogle Scholar
  19. 19.
    De Lathauwer, L., Vandewalle, J.: Dimensionality reduction in higher-order signal processing and rank-(R 1, R 2,...,R N) reduction in multilinear algebra. Linear Algebra Appl. 391, 31–55 (2004) (Special Issue on Linear Algebra in Signal and Image Processing)MATHCrossRefMathSciNetGoogle Scholar
  20. 20.
    De Vos, M., De Lathauwer, L., Vanrumste, B., Van Huffel, S., Van Paesschen, W.: Canonical decomposition of ictal scalp EEG and accurate source localization: principles and simulation study. J. Comput. Intell. Neurosci. 2007(Article ID 58253), 1–10 (2007) (Special Issue on EEG/MEG Signal Processing)CrossRefGoogle Scholar
  21. 21.
    De Vos, M., Vergult, A., De Lathauwer, L., De Clercq, W., Van Huffel, S., Dupont, P., Palmini, A., Van Paesschen, W.: Canonical decomposition of ictal scalp EEG reliably detects the seizure onset zone. NeuroImage 37(3), 844–854 (2007)CrossRefGoogle Scholar
  22. 22.
    Dedieu, J.P., Priouret, P., Malajovich, G.: Newton’s method on Riemannian manifolds: covariant alpha theory. IMA J. Numer. Anal. 23(3), 395–419 (2003)MATHCrossRefMathSciNetGoogle Scholar
  23. 23.
    Edelman, A., Arias, T.A., Smith, S.T.: The geometry of algorithms with orthogonality constraints. SIAM J. Matrix Anal. Appl. 20, 303–353 (1998)MATHCrossRefMathSciNetGoogle Scholar
  24. 24.
    Elad, M., Milanfar, P., Golub, G.H.: Shape from moments—an estimation theory perspective. IEEE Trans. Signal Process. 52, 1814–1829 (2004)CrossRefMathSciNetGoogle Scholar
  25. 25.
    Eldén, L., Savas, B.: A Newton–Grassmann method for computing the best multi-linear rank-(r 1, r 2, r 3) approximation of a tensor. Tech. Rep. LITH-MAT-R-2007-6-SE, Department of Mathematics, Linköpings Universitet (2007)Google Scholar
  26. 26.
    Golub, G.H., Van Loan, C.F.: Matrix Computations, 3rd edn. Johns Hopkins University Press, Baltimore (1996)MATHGoogle Scholar
  27. 27.
    Harshman, R.A.: Foundations of the PARAFAC procedure: model and conditions for an “explanatory” multi-mode factor analysis. UCLA Work. Pap. Phon. 16(1), 1–84 (1970)Google Scholar
  28. 28.
    Helmke, U., Moore, J.B.: Optimization and Dynamical Systems. Springer, New York (1994)Google Scholar
  29. 29.
    Hitchcock, F.L.: The expression of a tensor or a polyadic as a sum of products. J. Math. Phys. 6(1), 164–189 (1927)Google Scholar
  30. 30.
    Hitchcock, F.L.: Multiple invariants and generalized rank of a p-way matrix or tensor. J. Math. Phys. 7(1), 39–79 (1927)Google Scholar
  31. 31.
    Khoromskij, B.N., Khoromskaia, V.: Low rank Tucker-type tensor approximation to classical potentials. Cent. Eur. J. Math. 5(3), 523–550 (2007)MATHCrossRefMathSciNetGoogle Scholar
  32. 32.
    Kolda, T.: Orthogonal tensor decompositions. SIAM J. Matrix Anal. Appl. 23, 243–255 (2001)MATHCrossRefMathSciNetGoogle Scholar
  33. 33.
    Kroonenberg, P.M.: Applied Multiway Data Analysis. Wiley, New York (2008)MATHCrossRefGoogle Scholar
  34. 34.
    Kroonenberg, P.M., de Leeuw, J.: Principal component analysis of three-mode data by means of alternating least squares algorithms. Psychometrika 45(1), 69–97 (1980)MATHCrossRefMathSciNetGoogle Scholar
  35. 35.
    Li, C., Wang, J.: Newton’s method on Riemannian manifolds: Smale’s point estimate theory under the γ-condition. IMA J. Numer. Anal. 26(2), 228–251 (2006)CrossRefMathSciNetGoogle Scholar
  36. 36.
    Moravitz Martin, C.D., Van Loan, C.F.: A Jacobi-type method for computing orthogonal tensor decompositions. SIAM J. Matrix Anal. Appl. 30, 1219–1232 (2008)CrossRefMathSciNetGoogle Scholar
  37. 37.
    Papy, J.M., De Lathauwer, L., Van Huffel, S.: Exponential data fitting using multilinear algebra: the decimative case. Technical report 04-70, ESAT-SISTA, K.U.Leuven (2004)Google Scholar
  38. 38.
    Papy, J.M., De Lathauwer, L., Van Huffel, S.: Exponential data fitting using multilinear algebra: the single-channel and the multichannel case. Numer. Linear Algebra Appl. 12(8), 809–826 (2005)MATHCrossRefMathSciNetGoogle Scholar
  39. 39.
    Savas, B., Lim, L.H.: Best multilinear rank approximation of tensors with quasi-Newton methods on Grassmannians. Tech. Rep. LITH-MAT-R-2008-01-SE, Department of Mathematics, Linköpings Universitet (2008)Google Scholar
  40. 40.
    Sidiropoulos, N., Bro, R., Giannakis, G.: Parallel factor analysis in sensor array processing. IEEE Trans. Signal Process. 48, 2377–2388 (2000)CrossRefGoogle Scholar
  41. 41.
    Smilde, A., Bro, R., Geladi, P.: Multi-way Analysis. Applications in the Chemical Sciences. Wiley, Chichester (2004)CrossRefGoogle Scholar
  42. 42.
    Tucker, L.R.: The extension of factor analysis to three-dimensional matrices. In: Gulliksen, H., Frederiksen, N. (eds.) Contributions to Mathematical Psychology, pp. 109–127. Holt, Rinehart & Winston, New York (1964)Google Scholar
  43. 43.
    Tucker, L.R.: Some mathematical notes on three-mode factor analysis. Psychometrika 31, 279–311 (1966)CrossRefMathSciNetGoogle Scholar
  44. 44.
    Udrişte, C.: Convex Functions and Optimization Methods on Riemannian Manifolds. Kluwer, Dordrecht (1994)MATHGoogle Scholar
  45. 45.
    Vasilescu, M.A.O., Terzopoulos, D.: Multilinear subspace analysis for image ensembles. In: Proc. Computer Vision and Pattern Recognition Conf. (CVPR ’03), vol. 2, pp. 93–99. CVPR, Madison (2003)Google Scholar
  46. 46.
    Zhang, T., Golub, G.H.: Rank-one approximation to high order tensors. SIAM J. Matrix Anal. Appl. 23, 534–550 (2001)MATHCrossRefMathSciNetGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC. 2008

Authors and Affiliations

  • Mariya Ishteva
    • 1
  • Lieven De Lathauwer
    • 1
    • 2
  • P.-A. Absil
    • 3
  • Sabine Van Huffel
    • 1
  1. 1.ESAT/SCDKatholieke Universiteit LeuvenLeuvenBelgium
  2. 2.Subfaculty Science and TechnologyKatholieke Universiteit Leuven Campus KortrijkKortrijkBelgium
  3. 3.Department of Mathematical EngineeringUniversité catholique de LouvainLouvain-la-NeuveBelgium

Personalised recommendations