Skip to main content
SpringerLink
Log in

Structure of the Hessian matrix and an economical implementation of Newton’s method in the problem of canonical approximation of tensors

  • Published:
Computational Mathematics and Mathematical Physics Aims and scope Submit manuscript

Abstract

A tensor given by its canonical decomposition is approximated by another tensor (again, in the canonical decomposition) of fixed lower rank. For this problem, the structure of the Hessian matrix of the objective function is analyzed. It is shown that all the auxiliary matrices needed for constructing the quadratic model can be calculated so that the computational effort is a quadratic function of the tensor dimensionality (rather than a cubic function as in earlier publications). An economical version of the trust region Newton method is proposed in which the structure of the Hessian matrix is efficiently used for multiplying this matrix by vectors and for scaling the trust region. At each step, the subproblem of minimizing the quadratic model in the trust region is solved using the preconditioned conjugate gradient method, which is terminated if a negative curvature direction is detected for the Hessian matrix.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. R. Bellman, Adaptive Control Processes: A Guided Tour (Princeton Univ. Press, Princeton, NJ, 1961).

    MATH  Google Scholar 

  2. G. Beylkin and M. J. Mohlenkamp, “Algorithms for Numerical Analysis in High Dimensions,” SIAM J. Sci. Comput. 26, 2133–2159 (2005).

    Article  MATH  MathSciNet  Google Scholar 

  3. A. Evrim and V. Bülent, “Unsupervised Multiway Data Analysis: A Literature Survey,” IEEE Trans. Knowledge Data Eng. 21 (1), 6–20 (2009).

    Article  Google Scholar 

  4. T. Zhang and G. H. Golub, “Rank-One Approximation to High Order Tensors,” SIAM J. Matrix Anal. Appl. 23, 534–550 (2001).

    Article  MATH  MathSciNet  Google Scholar 

  5. I. V. Oseledets and D. V. Savost’yanov, “Minimization Methods for Approximating Tensors and Their Comparison,” Zh. Vychisl. Mat. Mat. Fiz. 46, 1725–1734 (2006) [Comput. Math. Math. Phys. 46, 1641-1650 (2006)].

    MathSciNet  Google Scholar 

  6. V. A. Kazeev, “On the Problem of Canonical Tensor Decomposition and an Algorithm for Its Solution as Applied to Search for Fast Matrix Algorithms,” Proceedings of the 51st MFTI Scientific Conference on Modern Problems in Fundamental Sciences and Applications, part VIII: Problems im Modern Physics (MFTI, Moscow, 2008), pp. 225-228.

  7. M. Espig, PhD Thesis (Univ. Leipzig, Leipzig, 2008).

    Google Scholar 

  8. J. D. Caroll and J. J. Chang, “Analysis of Individual Differences in Multidimensional Scaling via an N-Way Generalization of Eckart-Young Decomposition,” Psychometrika 35, 283–319 (1970).

    Article  Google Scholar 

  9. R. A. Harshman, “Foundations of the PARAFAC Procedure: Models and Conditions for an Explanatory Multimodal Factor Analysis,” UCLA Working Papers in Phonetics 16, 1–84 (1970).

    Google Scholar 

  10. V. de Silva and L.–H. Lim, “Tensor Rank and the Ill-Posedness of the Best Low-Rank Approximation Problem,” SIAM J. Matrix Anal. Appl. 30, 1084–1127 (2008).

    Article  MathSciNet  Google Scholar 

  11. E. E. Tyrtyshnikov, Matrix Analysis and Linear Algebra (Fizmatlit, Moscow, 2007) [in Russian].

    Google Scholar 

  12. M. J. Mohlenkamp and L. Monzon, “Trigonometric Identities and Sums of Separable Functions,” Math. Intelligencer 27 (2), 65–69 (2005).

    Article  MathSciNet  Google Scholar 

  13. T. Steihaug, “The Conjugate Gradient Method and Trust Regions in Large Scale Optimization,” SIAM J. Numer. Anal. 20, 626–637 (1983).

    Google Scholar 

  14. J. Nocedal and S. Wright, Numerical Optimization (Springer-Verlag, New York, 1999).

    Book  MATH  Google Scholar 

  15. G. A. Shultz, R. B. Schnabel, and R. H. Byrd, “A Family of Trust-Region-Based Algorithms for Unconstrained Minimization with Strong Global Convergence Properties,” SIAM J. Numer. Anal. 22, 47–67 (1985).

    Article  MATH  MathSciNet  Google Scholar 

  16. R. H. Byrd and G. A. Shultz, “A Trust Region Algorithm for Nonlinearly Constrained Optimization,” SIAM J. Numer. Anal. 24, 1152–1170 (1987).

    Article  MATH  MathSciNet  Google Scholar 

  17. R. H. Byrd, R. B. Schnabel, and G. A. Shultz, “Approximate Solution of the Trust Regions Problem by Minimization over Two-Dimensional Subspaces,” Math. Program. 40, 247–263 (1988).

    Article  MATH  MathSciNet  Google Scholar 

  18. D. M. Gay, “Computing Optimal Locally Constrained Steps,” SIAM J. Sci. Stat. Comput. 2 (2), 186–197 (1981).

    Article  MATH  MathSciNet  Google Scholar 

  19. D. C. Sorensen, “Newton’s Method with a Model Trust Modification,” SIAM J. Numer. Anal. 19, 409–426 (1982).

    Article  MATH  MathSciNet  Google Scholar 

  20. D. Ž. Djoković, “On the Hadamard Product of Matrices,” Math. Z. 86 (5), 395 (1965).

    Article  MATH  MathSciNet  Google Scholar 

  21. R. A. Horn and C. R. Johnson, Matrix Analysis (Cambridge Univ. Press, Cambridge, 1985).

    MATH  Google Scholar 

  22. V. Strassen, “Gaussian Elimination Is Not Optimal,” Numer. Math. 13, 354–356 (1969).

  23. J._M. Landsberg, “The Border Rank of the Multiplication of 2 × 2 Matrices in Seven,” J. Am. Math. Soc. 19, 447–459 (2005).

    Google Scholar 

  24. H.-J. Flad, B. N. Khoromskij, D. V. Savostyanov, and E. E. Tyrtyshnikov, “Verification of the Cross 3D Algorithm on Quantum Chemistry Data,” Russ. J. Numer. Anal. Math. Model. 23 (4), 210–220 (2008).

    MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Numerical Mathematics, Russian Academy of Sciences, ul. Gubkina 8, 119333, Moscow, Russia

    V. A. Kazeev & E. E. Tyrtyshnikov

Authors
  1. V. A. Kazeev
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. E. E. Tyrtyshnikov
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to V. A. Kazeev.

Additional information

Original Russian Text © V.A. Kazeev, E.E. Tyrtyshnikov, 2010, published in Zhurnal Vychislitel’noi Matematiki i Matematicheskoi Fiziki, 2010, Vol. 50, No. 6, pp. 979–998.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Kazeev, V.A., Tyrtyshnikov, E.E. Structure of the Hessian matrix and an economical implementation of Newton’s method in the problem of canonical approximation of tensors. Comput. Math. and Math. Phys. 50, 927–945 (2010). https://doi.org/10.1134/S0965542510060011

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1134/S0965542510060011

Search

Navigation