Frontiers of Mathematics in China

, Volume 2, Issue 4, pp 501–526 | Cite as

Numerical multilinear algebra and its applications

  • Qi Liqun 
  • Sun Wenyu 
  • Wang Yiju 
Survey Article


Numerical multilinear algebra (or called tensor computation), in which instead of matrices and vectors the higher-order tensors are considered in numerical viewpoint, is a new branch of computational mathematics. Although it is an extension of numerical linear algebra, it has many essential differences from numerical linear algebra and more difficulties than it. In this paper, we present a survey on the state of the art knowledge on this topic, which is incomplete, and indicate some new trends for further research. Our survey also contains a detailed bibliography as its important part. We hope that this new area will be receiving more attention of more scholars.


Numerical multilinear algebra higher order tensor tensor decomposition lower rank approximation of tensor multi-way data analysis 


15A18 15A69 65F30 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Alon N, de la Vega W F, Kannan R, et al. Random sampling and approximation of max-csps. J Comput System Sci, 2003, 67: 212–243zbMATHCrossRefMathSciNetGoogle Scholar
  2. 2.
    Bergman G M. Ranks of tensors and change of base field. J Algebra, 1969, 11: 613–621zbMATHCrossRefMathSciNetGoogle Scholar
  3. 3.
    Bienvenu G, Kopp L. Optimality of high-resolution array processing using the eigen-system approach. IEEE Trans ASSP, 1983, 31: 1235–1248CrossRefGoogle Scholar
  4. 4.
    Cao X R, Liu R W. General approach to blind source separation. IEEE Trans Signal Processing, 1996, 44: 562–570CrossRefGoogle Scholar
  5. 5.
    Cardoso J F. Super-symmetric decomposition of the forth-order cumulant tensor. Blind identification of more sources than sensors. In: Proceedings of the IEEE International Conference on Acoust, Speech, and Signal Processing (ICASSP’91), Toronto, Canada, 1991Google Scholar
  6. 6.
    Cardoso J F. High-order contrasts for independent component analysis. Neural Computation, 1999, 11: 157–192CrossRefGoogle Scholar
  7. 7.
    Caroll J D, Chang J J. Analysis of individual differences in multidimensional scaling via n-way generalization of Eckart-Young decomposition. Psychometrika, 1970, 35: 283–319CrossRefGoogle Scholar
  8. 8.
    Comon P. Tensor diagonalization, a useful tool in signal processing. In: Blanke M, Soderstrom T, eds. IFAC-SYSID, 10th IFAC Symposium on System Identification (Copenhagen, Denmark, July 4–6, 1994. Invited Session). Vol. 1, 77–82Google Scholar
  9. 9.
    Comon P. Independent component analysis, a new concept? Signal Processing, Special Issue on Higher-Order Statistics, 1994, 36: 287–314zbMATHGoogle Scholar
  10. 10.
    Comon P, Mourrain B. Decomposition of quantics in sums of powers of linear forms. Signal Processing, Special Issue on Higher-Order Statistics, 1996, 53: 96–107Google Scholar
  11. 11.
    Comon P. Block methods for channel identification and source separation. In: IEEE Symposium on Adaptive Systems for Signal Process, Commun Control (Lake Louise, Alberta, Canada, Oct 1–4, 2000. Invited Plenary). 87–92Google Scholar
  12. 12.
    Comon P, Chevalier P. Blind source separation: Models, concepts, algorithms, and performance. In: Haykin S, ed. Unsupervised Adaptive Filtering, Vol 1. New York: John Wiley, 2000Google Scholar
  13. 13.
    Comon P. Tensor decompositions: State of the art and applications. In: McWhirter J G, Proundler I K, eds. Mathematics in Signal Processing, V. Oxford: Oxford University Press, 2002Google Scholar
  14. 14.
    Comon P. Canonical tensor decompositions. In: ARCC Workshop on Tensor Decompositions, Americal Institute of Mathematics (AIM), Palo Alto, California, USA, July 19–23, 2004Google Scholar
  15. 15.
    Comon P, Golub G, Lim L H, et al. Symmetric tensors and symmetric tensor rank. SIAM J Matrix and Applications, 2007 (in press)Google Scholar
  16. 16.
    Coppi R, Bolasco S, eds. Multiway Data Analysis. Amsterdam: Elsevier, 1989zbMATHGoogle Scholar
  17. 17.
    de la Vega W F, Kannan R, Karpinski M, et al. Tensor Decomposition and Approximation Algorithms for Constraint Satisfaction Problems. New York: ACM Press, 2005, 747–754Google Scholar
  18. 18.
    Defant A, Floret K. Tensor Norms and Operator Ideals. North-Holland Mathematics Studies, No 176. Amsterdam: North-Holland, 1993Google Scholar
  19. 19.
    Ferrier C. Hilbert’s 17th problem and best dual bounds in quadatic minimization. Cybernet Systems Anal, 1998, 34: 696–709zbMATHCrossRefMathSciNetGoogle Scholar
  20. 20.
    Golub G, Van Loan C F. Matrix Computations. 3rd ed. Baltimore: Johns Hopkins University Press, 1996zbMATHGoogle Scholar
  21. 21.
    Golub G, Kolda T G, Nagy J, et al. Workshop on Tensor Decompositions, American Institute of Mathematics, Palo Alto, California, 19–23 July, 2004.
  22. 22.
    Golub G, Mahoney M, Drinears P, et al. Workshop for Modern Massive Data Sets, 21–24 June, 2006.
  23. 23.
    Golub G, Mahoney M, Drinears P, et al. Bridge the gap between numerical linear algebra, theoretical computer science, and data applications. SIAM News, 2006, 39(8).
  24. 24.
    Greub W H. Multilinear Algebra. Berlin: Springer-Verlag, 1967zbMATHGoogle Scholar
  25. 25.
    Harshman R A. Determination and proof of minimum uniqueness conditions of PARAFAC. UCLA Working Papers in Phonetics, 1972, 22: 111–117Google Scholar
  26. 26.
    He X, Sun W. Introduction to Generalized Inverses of Matrices. Nanjing: Jiangsu Sci & Tech Publishing House, 1990 (in Chinese)Google Scholar
  27. 27.
    Hitchcock F L. The expression of a tensor or a polyadic as a sum of products. J Math Physics, 1927, 6: 164–189Google Scholar
  28. 28.
    Hitchcock F L. Multiple invariants and generalized rank of a p-way matrix or tensor. J Math Physics, 1927, 7: 39–79Google Scholar
  29. 29.
    Kilmer M E, Martin C D M. Decomposing a tensor. SIAM News, 2004, 37.
  30. 30.
    Kofidis E, Regalia P A. Tensor approximation and signal processing applications. In: Olshevsky V, ed. Structured Matrices in Mathematics, Computer Science and Engineering, Vol. I. Contemporary Mathematics, 280. Providence: AMS, 2001Google Scholar
  31. 31.
    Kofidis E, Regalia P A. On the best rank-1 approximation of higher-order supersymmetric tensors. SIAM J Matrix Anal Appl, 2002, 23: 863–884zbMATHCrossRefMathSciNetGoogle Scholar
  32. 32.
    Kolda T G. Orthogonal tensor decompositions. SIAM J Matrix Anal Appl, 2001, 22: 243–255CrossRefMathSciNetGoogle Scholar
  33. 33.
    Kroonenberg P M. Singular value decompositions of interactions in three-way contigency tables. In: Coppi R, Bolasco S, eds. Multiway Data Analysis. North Holland: Elsevier Science Publishers, 1989: 169–184Google Scholar
  34. 34.
    Kruskal J B. Three-way arrays: rank and uniqueness of trilinear decompositions, with application to arithmetic complexity and statistics. Linear Algebra and Applications, 1977, 18: 95–138zbMATHCrossRefMathSciNetGoogle Scholar
  35. 35.
    Lasserre J B. Global optimization with polynomials and the problem of moments. SIAM J Optimization, 2001, 11: 796–817zbMATHCrossRefMathSciNetGoogle Scholar
  36. 36.
    Lasserre J B. Semidefinite programming vs. LP relaxations for polynomial programming. Mathematics of Operations Research, 2002, 27: 347–360zbMATHCrossRefMathSciNetGoogle Scholar
  37. 37.
    Lasserre J B. Polynomial programming: LP-relaxations also converge. SIAM J Optimization, 2004, 15: 383–393zbMATHCrossRefMathSciNetGoogle Scholar
  38. 38.
    Lasserre J B. A sum of squares approximation of nonnegative polynomials. SIAM J Optimization, 2006, 16: 751–765zbMATHCrossRefMathSciNetGoogle Scholar
  39. 39.
    De Lathauwer L, Comon P, De Moor B, et al. Higher-order power method—application in independent component analysis. In: Proceedings of the International Symposium on Nonlinear Theory and Its Applications (NOLTA’95), Las Vegas, NV. 1995, 91–96Google Scholar
  40. 40.
    De Lathauwer L, De Moor B. From matrix to tensor: Multilinear algebra and signal processing. In: Mathematics in Signal Processing, IMA Conference Series (Warwick, Dec 17–19, 1996), Oxford: Oxford University Press, 1996Google Scholar
  41. 41.
    De Lathauwer L, De Moor B, Vandewalle J. A multilinear singular value decomposition. SIAM J Matrix Anal Appl, 2000, 21: 1253–1278zbMATHCrossRefMathSciNetGoogle Scholar
  42. 42.
    De Lathauwer L, De Moor B, Vandewalle J. On the rank-1 and rank-(R 1, R 2, ..., R N) approximation of higher-order tensors. SIAM J Matrix Anal Appl, 2000, 21: 1324–1342zbMATHCrossRefMathSciNetGoogle Scholar
  43. 43.
    De Lathauwer L. First-order perturbation analysis of the best rank-(R 1, R 2, R 3) approximation in multilinear algebra. J Chemometrics, 2004, 18: 2–11CrossRefGoogle Scholar
  44. 44.
    De Lathauwer L, De Moor B, Vandewalle J. Computation of the canonical decomposition by means of a simultaneous generalized Schur decomposition. SIAM J Matrix Anal Appl, 2004/05, 26: 295–327zbMATHCrossRefMathSciNetGoogle Scholar
  45. 45.
    De Lathauwer L, Comon P. Workshop on Tensor Decompositions and Applications, Luminy, Marseille, France, August 29–September 2, 2005.
  46. 46.
    Leon S. Workshop Report: Algorithms for Modern Massive Data Sets (MMDS). NA Digest, 2006, Vol 6, No 27Google Scholar
  47. 47.
    Leurgans S E, Ross R T, Abel R B. A decomposition for three-way arrays. SIAM J Matrix Anal Appl, 1993, 14: 1064–1083zbMATHCrossRefMathSciNetGoogle Scholar
  48. 48.
    Li C, Sun W. A nonsmooth Newton-type method for nonlinear semidefinite programs. Technical Report, School of Mathematics and Computer Science, Nanjing Normal University, March, 2007Google Scholar
  49. 49.
    Li C, Sun W. Filter-Successive Linearization Methods for Nonlinear Semidefinite Programs. Technical Report, School of Mathematics and Computer Science, Nanjing Normal University, December, 2006Google Scholar
  50. 50.
    Lim L H. Singular values and eigenvalues of tensors: A variational approach. In: Proceedings of the 1st IEEE International Workshop on Computational Advances of multi-sensor Adaptive Processing (CAMSAP), December 13–15, 2005. 2005, 129–132Google Scholar
  51. 51.
    Liu X, Sidiropoulos N D. Cramer-Rao lower bounds for low-rank decomposition of multidimensional arrays. IEEE Trans on Signal Processing, 2001, 49: 2074–2086CrossRefGoogle Scholar
  52. 52.
    Luo Z Q, Lu J. On blind source separation using mutual information criterion. Mathematical Programming, Ser B, 2003, 97: 587–603zbMATHCrossRefMathSciNetGoogle Scholar
  53. 53.
    McCullagh P. Tensor Methods in Statistics. Monographs in Statistics and Applied Probability. London: Chapman and Hall, 1987Google Scholar
  54. 54.
    Merris R. Multilinear Algebra. The Netherland: Gordon and Breach Science Publishers, 1997zbMATHGoogle Scholar
  55. 55.
    Moré J J. The Levenberg-Marquadt algorithm: implementation and theory. In: Watson G A, ed. Numerical Analysis. Lecture Notes in Math, Vol 630, Berlin: Springer-Verlag, 1977, 105–116CrossRefGoogle Scholar
  56. 56.
    Nesterov Y. Squared functional systems and optimization problems. In: Frenk H, Roos K, Terlaky T, et al, eds. High Performance Optimization, Dordrecht: Kluwer, 2000, 405–440Google Scholar
  57. 57.
    Ni G, Qi L, Wang F, et al. The degree of the E-characteristic polynomial of an even order tensor. Journal of Mathematical Analysis and Applications, 2007, 329: 1218–1229CrossRefMathSciNetzbMATHGoogle Scholar
  58. 58.
    Ni G Y, Wang Y J. On the best rank-1 approximation to higher-order symmetric tensors. Mathematical and Computer Modeling, 2007, 46: 1345–1352CrossRefzbMATHMathSciNetGoogle Scholar
  59. 59.
    Northcoot D G. Multilinear Algebra. Cambridge: Cambridge University Press, 1984Google Scholar
  60. 60.
    Paatero P. A weighted non-negative least squares algorithm for three-way “PARAFAC” factor analysis. Chemometrics Intell Lab Syst, 1997, 38: 223–242CrossRefGoogle Scholar
  61. 61.
    Parrilo P A. Semidefinite programming relaxation for semialgebraic problems. Mathematical Programming, 2003, 96: 293–320zbMATHCrossRefMathSciNetGoogle Scholar
  62. 62.
    Qi L. Eigenvalues of a real supersymmetric tensor. Journal of Symbolic Computation, 2005, 40: 1302–1324CrossRefMathSciNetzbMATHGoogle Scholar
  63. 63.
    Qi L. Rank and eigenvalues of a supersymmetric tensor, a multivariate homogeneous polynomial and an algebraic surface defined by them. Journal of Symbolic Computation, 2006, 41: 1309–1327CrossRefMathSciNetzbMATHGoogle Scholar
  64. 64.
    Qi L. Eigenvalues and invariants of tensors. Journal of Mathematical Analysis and Applications, 2007, 325: 1363–1377zbMATHCrossRefMathSciNetGoogle Scholar
  65. 65.
    Qi L, Wang F, Wang Y. Z-eigenvalue methods for a global polynomial optimization problem. Mathematical Programming, 2008 (in press)Google Scholar
  66. 66.
    Schnabel R B. Conic methods for unconstrained optimization and tensor methods for nonlinear equations. In: Bachem A, Grotschel M, Korte B, eds. Mathematical Programming, The State of the Art. Berlin: Springer-Verlag, 1983, 417–438Google Scholar
  67. 67.
    Schnabel R B, Frank P D. Tensor methods for nonlinear equations. SIAM J Numerical Analysis, 1984, 21: 815–843zbMATHCrossRefMathSciNetGoogle Scholar
  68. 68.
    Schnabel R B, Chow T T. Tensor methods for unconstrained optimization using second derivatives. SIAM J Optimization, 1991, 1: 293–315zbMATHCrossRefMathSciNetGoogle Scholar
  69. 69.
    Schott J R. Matrix Analysis for Statistics. 2nd ed. New York: John Wiley & Sons, Inc, 2005zbMATHGoogle Scholar
  70. 70.
    Schweighofer M. Optimization of polynomials on compact semialgebraic sets. SIAM J Optimization, 2005, 15: 805–825zbMATHCrossRefMathSciNetGoogle Scholar
  71. 71.
    Shor N Z. Class of global minimum bounds of polynomial functions. Cybernetics, 1987, 23: 731–734zbMATHCrossRefGoogle Scholar
  72. 72.
    Shor N Z. Nondifferentiable Optimization and Polynomial Problems. Dordrecht: Kluwer, 1998zbMATHGoogle Scholar
  73. 73.
    Sidiropoulos N, Giannakis G, Bro R. Blind PARAFAC receivers for DS-CDMA systems. IEEE Trans Signal Process, 2000, 48: 810–823CrossRefGoogle Scholar
  74. 74.
    Sidiropoulos N, Bro R. On the uniqueness of multilinear decomposition of N-way arrays, J Chemometrics, 2000, 14: 229–239CrossRefGoogle Scholar
  75. 75.
    Sun W. Quasi-Newton methods for nonlinear matrix equations. Technical Report, School of Mathematics and Computer Science, Nanjing Normal University, 2006Google Scholar
  76. 76.
    Sun W, Du Q, Chen J. Computational Methods. Beijing: Science Press, 2007 (in Chinese)Google Scholar
  77. 77.
    Sun W, Yuan Y. Optimization Theory and Methods: Nonlinear Programming. New York: Springer, 2006zbMATHGoogle Scholar
  78. 78.
    Talwar S, Vilberg M, Paulraj A. Blind estimation of multiple cochannel digital signals arriving at an antenna array: Part I, algorithms. IEEE Trans Signal Process, 1996, 44: 1184–1197CrossRefGoogle Scholar
  79. 79.
    Tucker L R. Some mathematical notes on the three-mode factor analysis. Psychometrika, 1966, 31: 279–311CrossRefMathSciNetGoogle Scholar
  80. 80.
    Van Der Veen A-J, Paulraj A. An analytical constant modulus algorithm. IEEE Trans Signal Processing, 1996, 44: 1136–1155CrossRefGoogle Scholar
  81. 81.
    Vorobyov S A, Rong Y, Sidiropoulos N D, et al. Robust iterative fitting of multilinear models. IEEE Trans on Signal Processing, 2005, 53: 2678–2689CrossRefMathSciNetGoogle Scholar
  82. 82.
    Wang Y J, Qi L Q. On the successive supersymmetric rank-1 decomposition of higher-order supersymmetric tensors. Numerical Linear Algebra with Applications, 2007, 14(6): 503–519CrossRefMathSciNetGoogle Scholar
  83. 83.
    Zhang T, Golub G H. Rank-one approximation to high order tensors. SIAM J Matrix Anal Appl, 2001, 23: 534–550zbMATHCrossRefMathSciNetGoogle Scholar

Copyright information

© Higher Education Press and Springer-Verlag 2007

Authors and Affiliations

  1. 1.Department of Applied MathematicsThe Hong Kong Polytechnic UniversityHong KongChina
  2. 2.School of Mathematics and Computer ScienceNanjing Normal UniversityNanjingChina
  3. 3.School of Operations Research and Management ScienceQufu Normal UniversityRizhaoChina

Personalised recommendations