Skip to main content
SpringerLink
Log in

On the Best Low Multilinear Rank Approximation of Higher-order Tensors*

  • Conference paper
  • First Online:
Recent Advances in Optimization and its Applications in Engineering

Summary

This paper deals with the best low multilinear rank approximation of higher-order tensors. Given a tensor, we are looking for another tensor, as close as possible to the given one and with bounded multilinear rank. Higher-order tensors are used in higher-order statistics, signal processing, telecommunications and many other fields. In particular, the best low multilinear rank approximation is used as a tool for dimensionality reduction and signal subspace estimation.

Computing the best low multilinear rank approximation is a nontrivial task. Higher-order generalizations of the singular value decomposition lead to suboptimal solutions. The higher-order orthogonal iteration is a widely used linearly convergent algorithm for further refinement. We aim for conceptually faster algorithms. However, applying standard optimization algorithms directly is not a good idea since there are infinitely many equivalent solutions. Nice convergence properties are observed when the solutions are isolated. The present invariance can be removed by working on quotient manifolds. We discuss three algorithms, based on Newton’s method, the trust-region scheme and conjugate gradients. We also comment on the local minima of the problem.

* Research supported by: (1) The Belgian Federal Science Policy Office: IUAP P6/04 (DYSCO, “Dynamical systems, control and optimization”, 2007–2011), (2) Communauté française de Belgique - Actions de Recherche Concertées, (3) Research Council K.U.Leuven: GOA-AMBioRICS, GOA-MaNet, CoE EF/05/006 Optimization in Engineering (OPTEC), (4) F.W.O. project G.0321.06, “Numerical tensor methods for spectral analysis”. (5) “Impulsfinanciering Campus Kortrijk (2007–2012)(CIF1)” and STRT1/08/023. Part of this research was carried out while M. Ishteva was with K.U.Leuven, supported by OE/06/25, OE/07/17, OE/08/007, OE/09/004. The scientific responsibility rests with the authors.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 259.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 329.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 329.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. P.-A. Absil, C. G. Baker, K. A. Gallivan. Trust-region methods on Riemannian manifolds. Found. Comput. Math., 7(3):303–330, 2007.

    Article  MATH  MathSciNet  Google Scholar 

  2. P.-A. Absil, M. Ishteva, L. De Lathauwer, S. Van Huffel. A geometric Newton method for Oja’s vector field. Neural Comput., 21(5):1415–1433, 2009.

    Article  MATH  MathSciNet  Google Scholar 

  3. P.-A. Absil, R. Mahony, R. Sepulchre. Optimization Algorithms on Matrix Manifolds. Princeton University Press, Princeton, NJ, 2008.

    MATH  Google Scholar 

  4. E. Acar, C. A. Bingol, H. Bingol, R. Bro, B. Yener. Multiway analysis of epilepsy tensors. ISMB 2007 Conference Proc., Bioinformatics, 23(13):i10–i18, 2007.

    Google Scholar 

  5. R. L. Adler, J.-P. Dedieu, J. Y. Margulies, M. Martens, M. Shub. Newton’s method on Riemannian manifolds and a geometric model for the human spine. IMA J. Numer. Anal., 22(3):359–390, 2002.

    Article  MATH  MathSciNet  Google Scholar 

  6. C. A. Andersson, R. Bro. The N-way toolbox for matlab. Chemometrics and Intelligent Laboratory Systems, 52 (1):1–4, 2000. See also http://www.models.kvl.dk/source/nwaytoolbox/.

  7. R. Badeau, R. Boyer. Fast multilinear singular value decomposition for structured tensors. SIAM J. Matrix Anal. Appl., 30(3):1008–1021, 2008.

    Article  MathSciNet  Google Scholar 

  8. C. Caiafa, A. Cichocki. Reconstructing matrices and tensors from few rows and columns. In Proc. of 2009 International Symposium on Nonlinear Theory and its Applications, 2009. In press.

    Google Scholar 

  9. J. Carroll, J. Chang. Analysis of individual differences in multidimensional scaling via an N-way generalization of “Eckart-Young” decomposition. Psychometrika, 35(3):283–319, 1970.

    Article  MATH  Google Scholar 

  10. J. Chen, Y. Saad. On the tensor SVD and the optimal low rank orthogonal approximation of tensors. SIAM J. Matrix Anal. Appl., 30(4):1709–1734, 2009.

    Article  MATH  MathSciNet  Google Scholar 

  11. A. Cichocki, R. Zdunek, A. H. Phan, and S.-I. Amari. Nonnegative Matrix and Tensor Factorizations. Wiley, 2009.

    Book  Google Scholar 

  12. P. Comon. Independent component analysis, a new concept? Signal Processing, 36(3):287–314, 1994.

    Article  MATH  Google Scholar 

  13. P. Comon. Tensor decompositions. In J. G. McWhirter, I. K. Proudler (eds), Mathematics in Signal Processing V, pp. 1–24. Clarendon Press, Oxford, 2002.

    Google Scholar 

  14. P. Comon, G. Golub, L.-H. Lim, B. Mourrain. Symmetric tensors and symmetric tensor rank. SIAM J. Matrix Anal. Appl., 30(3):1254–1279, 2008.

    Article  MathSciNet  Google Scholar 

  15. A. R. Conn, N. I. M. Gould, P. L. Toint. Trust-Region Methods. MPS-SIAM Series on Optimization. SIAM, Philadelphia, PA, 2000.

    MATH  Google Scholar 

  16. L. De Lathauwer. Signal Processing Based on Multilinear Algebra. PhD thesis, Dept. of Electrical Engineering, Katholieke Universiteit Leuven, 1997.

    Google Scholar 

  17. L. De Lathauwer. A link between the canonical decomposition in multilinear algebra and simultaneous matrix diagonalization. SIAM J. Matrix Anal. Appl., 28(3):642–666, 2006.

    Article  MATH  MathSciNet  Google Scholar 

  18. L. De Lathauwer. Decompositions of a higher-order tensor in block terms — Part I: Lemmas for partitioned matrices. SIAM J. Matrix Anal. Appl., 30(3):1022–1032, 2008.

    Article  MATH  MathSciNet  Google Scholar 

  19. L. De Lathauwer. Decompositions of a higher-order tensor in block terms — Part II: Definitions and uniqueness. SIAM J. Matrix Anal. Appl., 30(3):1033– 1066, 2008.

    Article  MATH  MathSciNet  Google Scholar 

  20. L. De Lathauwer. A survey of tensor methods. In Proc. of the 2009 IEEE International Symposium on Circuits and Systems (ISCAS 2009), pp. 2773– 2776, Taipei, Taiwan, 2009.

    Google Scholar 

  21. L. De Lathauwer, J. Castaing. Tensor-based techniques for the blind separation of DS-CDMA signals. Signal Processing, 87(2):322–336, 2007.

    Article  MATH  Google Scholar 

  22. L. De Lathauwer, B. De Moor, J. Vandewalle. A multilinear singular value decomposition. SIAM J. Matrix Anal. Appl., 21(4):1253–1278, 2000.

    Article  MATH  MathSciNet  Google Scholar 

  23. L. De Lathauwer, B. De Moor, J. Vandewalle. On the best rank-1 and rank-(R 1,R 2,…,R N ) approximation of higher-order tensors. SIAM J. Matrix Anal. Appl., 21 (4):1324–1342, 2000.

    Article  MATH  MathSciNet  Google Scholar 

  24. L. De Lathauwer, B. De Moor, J. Vandewalle. Independent component analysis and (simultaneous) third-order tensor diagonalization. IEEE Trans. Signal Process., 49(10):2262–2271, 2001.

    Article  Google Scholar 

  25. L. De Lathauwer, B. De Moor, J. Vandewalle. Computation of the canonical decomposition by means of a simultaneous generalized Schur decomposition.SIAM J. Matrix Anal. Appl., 26(2):295–327, 2004.

    Article  MATH  MathSciNet  Google Scholar 

  26. L. De Lathauwer, D. Nion. Decompositions of a higher-order tensor in block terms — Part III: Alternating least squares algorithms. SIAM J. Matrix Anal. Appl., 30(3):1067–1083, 2008.

    Article  MATH  MathSciNet  Google Scholar 

  27. L. De Lathauwer, J. Vandewalle. Dimensionality reduction in higher-order signal processing and rank-(R 1,R 2,…,R N ) reduction in multilinear algebra. Linear Algebra Appl., 391:31–55, 2004.

    Article  MATH  MathSciNet  Google Scholar 

  28. V. de Silva, L.-H. Lim. Tensor rank and the ill-posedness of the best low-rank approximation problem. SIAM J. Matrix Anal. Appl., 30(3):1084–1127, 2008.

    Article  MathSciNet  Google Scholar 

  29. M. De Vos, L. De Lathauwer, B. Vanrumste, S. Van Huffel, W. Van Paesschen.Canonical decomposition of ictal scalp EEG and accurate source localization:Principles and simulation study. Journal of Computational Intelligence and Neuroscience, 2007(Article ID 58253):1–10, 2007.

    Google Scholar 

  30. M. De Vos, A. Vergult, L. De Lathauwer, W. De Clercq, S. Van Huffel, P. Dupont, A. Palmini, W. Van Paesschen. Canonical decomposition of ictal scalp EEG reliably detects the seizure onset zone. NeuroImage, 37(3):844–854, 2007.

    Article  Google Scholar 

  31. M. Elad, P. Milanfar, G. H. Golub. Shape from moments — an estimation theory perspective. IEEE Trans. on Signal Processing, 52(7):1814–1829, 2004.

    Article  MathSciNet  Google Scholar 

  32. L. Eldén, B. Savas. A Newton–Grassmann method for computing the best multi-linear rank-(r 1,r 2,…,r 3) approximation of a tensor. SIAM J. Matrix Anal. Appl., 31(2):248–271, 2009.

    Article  MATH  MathSciNet  Google Scholar 

  33. R. Fletcher, C. M. Reeves. Function minimization by conjugate gradients. Comput. J., 7:149–154, 1964.

    Article  MATH  MathSciNet  Google Scholar 

  34. G. H. Golub, C. F. Van Loan. Matrix Computations. Johns Hopkins University Press, Baltimore, Maryland, 3rd edition, 1996.

    MATH  Google Scholar 

  35. M. Haardt, F. Roemer, G. Del Galdo. Higher-order SVD-based subspace estimation to improve the parameter estimation accuracy in multidimensional harmonic retrieval problems. IEEE Trans. on Signal Processing, 56(7):3198– 3213, 2008.

    Article  Google Scholar 

  36. W. W. Hager, H. Zhang. A new conjugate gradient method with guaranteed descent and an efficient line search. SIAM Journal on Optimization, 16(1):170– 192, 2005.

    Article  MATH  MathSciNet  Google Scholar 

  37. R. A. Harshman. Foundations of the PARAFAC procedure: Model and conditions for an “explanatory” multi-mode factor analysis. UCLA Working Papers in Phonetics, 16(1):1–84, 1970.

    Google Scholar 

  38. U. Helmke, J. B. Moore. Optimization and Dynamical Systems. Springer-Verlag, 1993.

    Google Scholar 

  39. M. R. Hestenes, E. Stiefel. Methods of conjugate gradients for solving linear systems. J. Research Nat. Bur. Standards, 49:409–436 (1953), 1952.

    MathSciNet  Google Scholar 

  40. F. L. Hitchcock. The expression of a tensor or a polyadic as a sum of products. Journal of Mathematical Physics, 6(1):164–189, 1927.

    Google Scholar 

  41. F. L. Hitchcock. Multiple invariants and generalized rank of a p-way matrix or tensor. Journal of Mathematical Physics, 7(1):39–79, 1927.

    Google Scholar 

  42. M. Ishteva. Numerical methods for the best low multilinear rank approximation of higher-order tensors. PhD thesis, Dept. of Electrical Engineering, Katholieke Universiteit Leuven, 2009.

    Google Scholar 

  43. M. Ishteva, P.-A. Absil, S. Van Huffel, L. De Lathauwer. Best low multilinear rank approximation with conjugate gradients. Tech. Rep. 09-246, ESAT-SISTA, K.U.Leuven, Belgium, 2009.

    Google Scholar 

  44. M. Ishteva, P.-A. Absil, S. Van Huffel, L. De Lathauwer. Tucker compression and local optima. Tech. Rep. UCL-INMA-2010.012, Université catholique de Louvain and 09-247, ESAT-SISTA, K.U.Leuven, Belgium, 2010.

    Google Scholar 

  45. M. Ishteva, L. De Lathauwer, P.-A. Absil, S. Van Huffel. Dimensionality reduction for higher-order tensors: algorithms and applications. International Journal of Pure and Applied Mathematics, 42(3):337–343, 2008.

    MathSciNet  Google Scholar 

  46. M. Ishteva, L. De Lathauwer, P.-A. Absil, S. Van Huffel. Best low multilinear rank approximation of higher-order tensors, based on the Riemannian trustregion scheme. Tech. Rep. 09-142, ESAT-SISTA, K.U.Leuven, Belgium, 2009.

    Google Scholar 

  47. M. Ishteva, L. De Lathauwer, P.-A. Absil, S. Van Huffel. Differential-geometric Newton method for the best rank-(R 1, R 2, R 3) approximation of tensors. Numerical Algorithms, 51(2):179–194, 2009.

    Article  MATH  MathSciNet  Google Scholar 

  48. M. Ishteva, L. De Lathauwer, S. Van Huffel. Comparison of the performance of matrix and tensor based multi-channel harmonic analysis. In 7th International Conf. on Mathematics in Signal Processing, Cirencester, UK, pp. 77–80, 2006.

    Google Scholar 

  49. E. Kofidis, P. A. Regalia. On the best rank-1 approximation of higher-order supersymmetric tensors. SIAM J. Matrix Anal. Appl, 23(3):863–884, 2002.

    Article  MATH  MathSciNet  Google Scholar 

  50. T. Kolda. Orthogonal tensor decompositions. SIAM J. Matrix Anal. Appl., 23:243–255, 2001.

    Article  MATH  MathSciNet  Google Scholar 

  51. T. G. Kolda, B. W. Bader. Tensor decompositions and applications. SIAM Review, 51(3):455–500, 2009.

    Article  MATH  MathSciNet  Google Scholar 

  52. P. M. Kroonenberg. Applied Multiway Data Analysis. Wiley, 2008.

    Google Scholar 

  53. P. M. Kroonenberg, J. de Leeuw. Principal component analysis of three-mode data by means of alternating least squares algorithms. Psychometrika, 45(1):69– 97, 1980.

    Article  MATH  MathSciNet  Google Scholar 

  54. M. W. Mahoney, M. Maggioni, P. Drineas. Tensor-CUR decompositions for tensor-based data. SIAM J. Matrix Anal. Appl., 30(3):957–987, 2008.

    Article  MathSciNet  Google Scholar 

  55. J. H. Manton. Optimization algorithms exploiting unitary constraints. IEEE Trans. Signal Process., 50(3):635–650, 2002.

    Article  MathSciNet  Google Scholar 

  56. C. D. Moravitz Martin, C. F. Van Loan. A Jacobi-type method for computing orthogonal tensor decompositions. SIAM J. Matrix Anal. Appl., 30(3):1219– 1232, 2008.

    Article  MathSciNet  Google Scholar 

  57. J. J. Moré, D. C. Sorensen. Computing a trust region step. SIAM J. Sci. Statist. Comput., 4(3):553–572, 1983.

    MATH  MathSciNet  Google Scholar 

  58. J. Nocedal, S. J. Wright. Numerical Optimization. Springer Verlag, New York, 2nd edition, 2006. Springer Series in Operations Research.

    MATH  Google Scholar 

  59. I. V. Oseledets, D. V. Savostianov, E. E. Tyrtyshnikov. Tucker dimensionality reduction of three-dimensional arrays in linear time. SIAM J. Matrix Anal. Appl., 30(3):939–956, 2008.

    Article  MathSciNet  Google Scholar 

  60. J.-M. Papy, L. De Lathauwer, S. Van Huffel. Exponential data fitting using multilinear algebra: The single-channel and the multichannel case. Numer. Linear Algebra Appl., 12(8):809–826, 2005.

    Article  MATH  MathSciNet  Google Scholar 

  61. J.-M. Papy, L. De Lathauwer, S. Van Huffel. Exponential data fitting using multilinear algebra: The decimative case. J. Chemometrics, 23(7–8):341–351, 2009.

    Article  Google Scholar 

  62. E. Polak, G. Ribière. Note sur la convergence de méthodes de directions conjugu ées. Rev. Française Informat. Recherche Opérationnelle, 3(16):35–43, 1969.

    MATH  Google Scholar 

  63. B. Savas, L. Eldén. Krylov subspace methods for tensor computations. Tech. Rep. LITH-MAT-R-2009-02-SE, Dept. of Mathematics, Link¨oping University, 2009.

    Google Scholar 

  64. B. Savas, L.-H. Lim. Best multilinear rank approximation of tensors with quasi- Newton methods on Grassmannians. Tech. Rep. LITH-MAT-R-2008-01-SE, Dept. of Mathematics, Link¨oping University, 2008.

    Google Scholar 

  65. M. Shub. Some remarks on dynamical systems and numerical analysis. In L. Lara-Carrero, J. Lewowicz (eds), Proc. VII ELAM., pp. 69–92. Equinoccio, U. Sim´on Bol´ıvar, Caracas, 1986.

    Google Scholar 

  66. N. Sidiropoulos, R. Bro, G. Giannakis. Parallel factor analysis in sensor array processing. IEEE Trans. Signal Process., 48:2377–2388, 2000.

    Article  Google Scholar 

  67. A. Smilde, R. Bro, P. Geladi. Multi-way Analysis. Applications in the Chemical Sciences. John Wiley and Sons, Chichester, U.K., 2004.

    Book  Google Scholar 

  68. S. T. Smith. Geometric Optimization Methods for Adaptive Filtering. PhD thesis, Division of Applied Sciences, Harvard University, Cambridge, MA, 1993.

    Google Scholar 

  69. S. T. Smith. Optimization techniques on Riemannian manifolds. In A. Bloch (ed), Hamiltonian and gradient flows, algorithms and control, volume 3 of Fields Inst. Commun., pp. 113–136. Amer. Math. Soc., Providence, RI, 1994.

    Google Scholar 

  70. T. Steihaug. The conjugate gradient method and trust regions in large scale optimization. SIAM J. Numer. Anal., 20(3):626–637, 1983.

    Article  MATH  MathSciNet  Google Scholar 

  71. P. L. Toint. Towards an efficient sparsity exploiting Newton method for minimization. In I. S. Duff (ed), Sparse Matrices and Their Uses, pp. 57–88. Academic Press, London, 1981.

    Google Scholar 

  72. L. R. Tucker. The extension of factor analysis to three-dimensional matrices. In H. Gulliksen, N. Frederiksen (eds), Contributions to mathematical psychology, pp. 109–127. Holt, Rinehart & Winston, NY, 1964.

    Google Scholar 

  73. L. R. Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika, 31:279–311, 1966.

    Article  MathSciNet  Google Scholar 

  74. L. Vanhamme, S. Van Huffel. Multichannel quantification of biomedical magnetic resonance spectroscopy signals. In F. Luk (ed), Proc. of SPIE, Advanced Signal Processing Algorithms, Architectures, and Implementations VIII, volume 3461, pp. 237–248, San Diego, California, 1998.

    Google Scholar 

  75. T. Zhang, G. H. Golub. Rank-one approximation to high order tensors. SIAM J. Matrix Anal. Appl., 23:534–550, 2001.

    Article  MATH  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mathematical Engineering, Université catholique de Louvain, Bâtiment Euler - P13, Av. Georges Lemaître 4, 1348, Louvain-la-Neuve, Belgium

    Mariya Ishteva & P.-A. Absil

  2. Department of Electrical Engineering - ESAT, SCD, K.U.Leuven, Kasteelpark Arenberg 10/2446, 3001, Leuven, Belgium

    Sabine Van Huffel & Lieven De Lathauwer

  3. Group Science, Engineering and Technology, K.U.Leuven Campus Kortrijk, E. Sabbelaan 53, 8500, Kortrijk, Belgium

    Lieven De Lathauwer

Authors
  1. Mariya Ishteva
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. P.-A. Absil
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sabine Van Huffel
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lieven De Lathauwer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mariya Ishteva .

Editor information

Editors and Affiliations

  1. Dept. Electrical Engineering (ESAT), Katholieke Universiteit Leuven, Kasteelpark Arenberg 10, Leuven, 3001, Belgium

    Moritz Diehl

  2. CORE-UCL, Voie du Roman Pays 34, Louvain-la-Neuve, 1348, Belgium

    Francois Glineur

  3. , Department Computerwetenschappen, Katholieke Universiteit Leuven, Leuven, 3001, Belgium

    Elias Jarlebring

  4. Dept. Computer Science, Katholieke Universiteit Leuven, Celesijnenlaan 200a, Leuven, 3001, Belgium

    Wim Michiels

Rights and permissions

Reprints and permissions

Copyright information

© 2010 Springer -Verlag Berlin Heidelberg

About this paper

Cite this paper

Ishteva, M., Absil, PA., Van Huffel, S., De Lathauwer, L. (2010). On the Best Low Multilinear Rank Approximation of Higher-order Tensors*. In: Diehl, M., Glineur, F., Jarlebring, E., Michiels, W. (eds) Recent Advances in Optimization and its Applications in Engineering. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-12598-0_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-12598-0_13

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-12597-3

  • Online ISBN: 978-3-642-12598-0

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation