Exact line and plane search for tensor optimization

Abstract

Line and plane searches are used as accelerators and globalization strategies in many optimization algorithms. We introduce a class of optimization problems called tensor optimization, which comprises applications ranging from tensor decompositions to least squares support tensor machines. We develop algorithms to efficiently compute the global minimizers of their line and plane search subproblems. Furthermore, we introduce scaled line and plane search, which compute an optimal scaling of the solution simultaneously with the optimal line or plane search step, and show that this scaling can be computed at almost no additional cost. Obtaining the global minimizers of (scaled) line and plane search problems often requires solving a bivariate or polyanalytic polynomial system. We show how to compute the isolated real solutions of bivariate polynomial systems and the isolated complex solutions of polyanalytic polynomial systems using a single generalized eigenvalue decomposition. Finally, we apply block term decompositions to the problem of blind multi-user detection-estimation in DS-CDMA communication to demonstrate that exact line and plane search can significantly reduce computation time of the workhorse tensor decomposition algorithm alternating least squares.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Notes

  1. 1.

    The homogeneity requirement, which is satisfied by most tensor decompositions, is only necessary for scaled line and plane search.

References

  1. 1.

    Abatzoglou, T.J.: Tensor-based techniques for the blind separation of DS-CDMA signals. Signal Process. 87(2), 322–336 (2007)

    MathSciNet  Article  Google Scholar 

  2. 2.

    Acar, E., Rasmussen, M.A., Savorani, F., Næs, T., Bro, R.: Understanding data fusion within the framework of coupled matrix and tensor factorizations. Chemom. Intell. Lab. Syst. 129, 53–63 (2013)

    Article  Google Scholar 

  3. 3.

    Berberich, E., Emeliyanenko, P., Sagraloff, M.: An elimination method for solving bivariate polynomial systems: Eliminating the usual drawbacks. In: Müller-Hannemann, M., Werneck, R. (eds.) Proceedings of the Thirteenth Workshop on Algorithm Engineering and Experiments (ALENEX), pp. 35–47. SIAM, San Francisco, CA, USA (2011)

    Google Scholar 

  4. 4.

    Bro, R.: Multi-way analysis in the food industry: models, algorithms, and applications. Ph.D. Thesis, University of Amsterdam (1998)

  5. 5.

    Buchberger, B.: Gröbner bases and systems theory. Multidim. Syst. Signal Process. 12, 223–251 (2001)

    MATH  MathSciNet  Article  Google Scholar 

  6. 6.

    Busé, L., Khalil, H., Mourrain, B.: Resultant-based methods for plane curves intersection problems. In: Proceedings of the 8th International Conference on Computer Algebra in Scientific Computing. Lecture Notes in Computer Science, vol. 3718, pp. 75–92. Springer-Verlag, Heidelberg (2005)

  7. 7.

    Byrd, R.H., Schnabel, R.B., Schultz, G.A.: Approximate solution of the trust region problem by minimization over two-dimensional subspaces. Math. Program. 40(1), 247–263 (1988)

    MATH  Article  Google Scholar 

  8. 8.

    Carroll, J.D., Chang, J.J.: Analysis of individual differences in multidimensional scaling via an n-way generalization of “Eckart-Young” decomposition. Psychometrika 35(3), 283–319 (1970)

    MATH  Article  Google Scholar 

  9. 9.

    Chen, Y., Han, D., Qi, L.: New ALS methods with extrapolating search directions and optimal step size for complex-valued tensor decompositions. IEEE Trans. Signal Process. 59(12), 5888–5898 (2011)

    MathSciNet  Article  Google Scholar 

  10. 10.

    Cichocki, A., Mandic, D., Caiafa, C., Phan, A.H., Zhou, G., Zhao, Q., De Lathauwer, L.: Tensor decompositions for signal processing applications. From two-way to multiway component analysis. IEEE Signal Process. Mag. 32(2), 145–163 (2015)

    Article  Google Scholar 

  11. 11.

    Comon, P.: Independent component analysis, a new concept? Signal Process. 36(3), 287–314 (1994)

    MATH  Article  Google Scholar 

  12. 12.

    Comon, P., Jutten, C.: Handbook of Blind Source Separation: Independent Component Analysis and Applications. Academic Press, New York (2010)

    Google Scholar 

  13. 13.

    Corless, R.M., Gianni, P.M., Trager, B.M.: A reordered Schur factorization method for zero-dimensional polynomial systems with multiple roots. In: Proceedings of the International Symposium on Symbolic and Algebraic Computation, pp. 133–140. ACM Press, New York (1997)

  14. 14.

    De Lathauwer, L.: Decompositions of a higher-order tensor in block terms—Part I: lemmas for partitioned matrices. SIAM J. Matrix Anal. Appl. 30(3), 1022–1032 (2008)

    MATH  MathSciNet  Article  Google Scholar 

  15. 15.

    De Lathauwer, L.: Decompositions of a higher-order tensor in block terms—Part II: definitions and uniqueness. SIAM J. Matrix Anal. Appl. 30(3), 1033–1066 (2008)

    MATH  MathSciNet  Article  Google Scholar 

  16. 16.

    De Lathauwer, L.: Blind separation of exponential polynomials and the decomposition of a tensor in rank-\(({L}_r,{L}_r,1)\) terms. SIAM J. Matrix Anal. Appl. 32(4), 1451–1474 (2011)

    MATH  MathSciNet  Article  Google Scholar 

  17. 17.

    De Lathauwer, L.: A short introduction to tensor-based methods for factor analysis and blind source separation. In: Proceeding of the 7th International Symposium on Image and Signal Processing and Analysis (ISPA 2011), pp. 558–563 (2011)

  18. 18.

    De Lathauwer, L.: Block component analysis, a new concept for source separation and factor analysis. In: Latent Variable Analysis and Signal Separation. Lecture Notes in Computer Science, vol. 7191, pp. 1–8. Proceedings of 10th International Conference, LVA/ICA 2012, Tel Aviv, Israel, March 12–15, 2012. (2012)

  19. 19.

    De Lathauwer, L., de Baynast, A.: Blind deconvolution of DS-CDMA signals by means of decomposition in rank-\((1, {L}, {L})\) terms. IEEE Trans. Signal Process. 56(4), 1562–1571 (2008)

    MathSciNet  Article  Google Scholar 

  20. 20.

    De Lathauwer, L., Nion, D.: Decompositions of a higher-order tensor in block terms—Part III: alternating least squares algorithms. SIAM J. Matrix Anal. Appl. 30(3), 1067–1083 (2008)

    MATH  MathSciNet  Article  Google Scholar 

  21. 21.

    De Lathauwer, L., Vandewalle, J.: Dimensionality reduction in higher-order signal processing and rank-\(({R}_1,{R}_2,\ldots,{R}_{N})\) reduction in multilinear algebra. Linear Algebra Appl. 391, 31–55 (2004)

    MATH  MathSciNet  Article  Google Scholar 

  22. 22.

    Ding, Z., Johnson, C.R., Kennedy, R.A.: On the (non)existence of undesirable equilibria of godard blind equalizers. IEEE Trans. Signal Process. 40(10), 2425–2432 (1992)

    MATH  Article  Google Scholar 

  23. 23.

    Diochnos, D.I., Emiris, I.Z., Tsigaridas, E.P.: On the asymptotic and practical complexity of solving bivariate systems over the reals. J. Symb. Comput. 44(7), 818–835 (2009)

    MATH  MathSciNet  Article  Google Scholar 

  24. 24.

    Domanov, I., De Lathauwer, L.: On the uniqueness of the canonical polyadic decomposition of third-order tensors—Part I: basic results and uniqueness of one factor matrix. SIAM J. Matrix Anal. Appl. 34, 855–875 (2013)

    MATH  MathSciNet  Article  Google Scholar 

  25. 25.

    Domanov, I., De Lathauwer, L.: On the uniqueness of the canonical polyadic decomposition of third-order tensors—Part II: overall uniqueness. SIAM J. Matrix Anal. Appl. 34, 876–903 (2013)

    MATH  MathSciNet  Article  Google Scholar 

  26. 26.

    Emiris, I.Z., Mourrain, B.: Matrices in elimination theory. J. Symb. Comput. 28(1–2), 3–43 (1999)

    MATH  MathSciNet  Article  Google Scholar 

  27. 27.

    Faugère, J.C.: A new efficient algorithm for computing Gröbner bases without reduction to zero (F5). In: Proceedings of the 2002 International Symposium on Symbolic and Algebraic Computation, ISSAC ’02, pp. 75–83. ACM, New York (2002)

  28. 28.

    Godard, D.N.: Self-recovering equalization and carrier tracking in two-dimensional data communication systems. IEEE Trans. Commun. 28(11), 1867–1875 (1980)

    Article  Google Scholar 

  29. 29.

    Gohberg, I.C., Lancaster, P., Rodman, L.: Matrix Polynomials. Academic Press, New York (1982)

    Google Scholar 

  30. 30.

    Haardt, M., Roemer, F., Del Galdo, G.: Higher-order SVD-based subspace estimation to improve the parameter estimation accuracy in multidimensional harmonic retrieval problems. IEEE Trans. Signal Process. 56(7), 3198–3213 (2008)

    MathSciNet  Article  Google Scholar 

  31. 31.

    Harshman, R.A.: Foundations of the PARAFAC procedure: models and conditions for an “explanatory” multi-modal factor analysis. UCLA Working Papers in Phonetics, vol. 16, pp. 1–84 (1970)

  32. 32.

    Karami, A., Yazdi, M., Zolghadre Asli, A.: Noise reduction of hyperspectral images using kernel non-negative Tucker decomposition. IEEE J. Sel. Top. Signal Process. 5(3), 487–493 (2011)

  33. 33.

    Kolda, T.G., Bader, B.W.: Tensor decompositions and applications. SIAM Rev. 51(3), 455–500 (2009)

    MATH  MathSciNet  Article  Google Scholar 

  34. 34.

    Kruskal, J.B.: Three-way arrays: rank and uniqueness of trilinear decompositions, with application to arithmetic complexity and statistics. Linear Algebra Appl. 18(2), 95–138 (1977)

    MATH  MathSciNet  Article  Google Scholar 

  35. 35.

    Kruskal, J.B.: Rank, Decomposition, and Uniqueness for 3-way and \({N}\)-way Arrays, pp. 7–18. Elsevier Science Publishers B.V., Amsterdam (1989)

  36. 36.

    Kruskal, J.B., Harshman, R.A., Lundy, M.E.: How 3-MFA data can cause degenerate PARAFAC solutions, among other relationships. In: Coppi, R., Bolasco, S. (eds.) Multiway Data Analysis, pp. 115–121. North-Holland Publishing Co., Amsterdam (1989)

    Google Scholar 

  37. 37.

    Li, T.Y.: Numerical solution of multivariate polynomial systems by homotopy continuation methods. Acta Numer. 6, 399–436 (1997)

    Article  Google Scholar 

  38. 38.

    Lu, H., Plataniotis, K.N., Venetsanopoulos, A.N.: A survey of multilinear subspace learning for tensor data. Pattern Recognit. 44(7), 1540–1551 (2011)

    MATH  Article  Google Scholar 

  39. 39.

    Manocha, D., Demmel, J.W.: Algorithms for intersecting parametric and algebraic curves II: multiple intersections. Graph. Models Image Process. 57(2), 81–100 (1995)

    Article  Google Scholar 

  40. 40.

    Mitchell, B.C., Burdick, D.S.: Slowly converging PARAFAC sequences: swamps and two-factor degeneracies. J. Chem. 8(2), 155–168 (1994)

    Article  Google Scholar 

  41. 41.

    Moore, R.E.: Methods and Applications of Interval Analysis. SIAM, Philadelphia (1987)

  42. 42.

    Nion, D., De Lathauwer, L.: Block component model based blind DS-CDMA receivers. IEEE Trans. Signal Process. 56(11), 5567–5579 (2008)

    MathSciNet  Article  Google Scholar 

  43. 43.

    Nion, D., De Lathauwer, L.: An enhanced line search scheme for complex-valued tensor decompositions. Appl. DS-CDMA Signal Process. 88(3), 749–755 (2008)

    MATH  Article  Google Scholar 

  44. 44.

    Nion, D., De Lathauwer, L.: Tensor algebra and multidimensional harmonic retrieval in signal processing for MIMO radar. IEEE Trans. Signal Process. 58(11), 5693–5705 (2010)

    MathSciNet  Article  Google Scholar 

  45. 45.

    Nocedal, J., Wright, S.J.: Numerical Optimization. Springer Series in Operations Research, 2nd edn. Springer, New York (2006)

    Google Scholar 

  46. 46.

    Rajih, M., Comon, P., Harshman, R.A.: Enhanced line search: a novel method to accelerate PARAFAC. SIAM J. Matrix Anal. Appl. 30(3), 1128–1147 (2008)

    MathSciNet  Article  Google Scholar 

  47. 47.

    Šebek, M., Pejchová, S., Henrion, D., Kwakernaak, H.: Numerical methods for zeros and determinant of polynomial matrix. In: Proceedings of the IEEE Mediterranean Symposium on New Directions in Control and Automation, pp. 488–491. Chania, Crete, Greece (1996)

  48. 48.

    Seidel, R., Wolpert, N.: On the exact computation of the topology of real algebraic curves. In: Proceedings of the Twenty-first Annual Symposium on Computational Geometry, SCG 2005, pp. 107–115. ACM, New York (2005)

  49. 49.

    Sidiropoulos, N.D., Bro, R., Giannakis, G.B.: Parallel factor analysis in sensor array processing. IEEE Trans. Signal Process. 48(8), 2377–2388 (2000)

    Article  Google Scholar 

  50. 50.

    Sidiropoulos, N.D., Dimić, G.Z.: Blind multiuser detection in W-CDMA systems with large delay spread. IEEE Signal Process. Lett. 8(3), 87–89 (2001)

    Article  Google Scholar 

  51. 51.

    Sidiropoulos, N.D., Giannakis, G.B., Bro, R.: Blind PARAFAC receivers for DS-CDMA systems. IEEE Trans. Signal Process. 48(3), 810–823 (2000)

    Article  Google Scholar 

  52. 52.

    Sorber, L., Van Barel, M., De Lathauwer, L.: Unconstrained optimization of real functions in complex variables. SIAM J. Optim. 22(3), 879–898 (2012)

    MATH  MathSciNet  Article  Google Scholar 

  53. 53.

    Sorber, L., Van Barel, M., De Lathauwer, L.: Numerical solution of bivariate and polyanalytic polynomial systems. SIAM J. Numer. Anal. 52(4), 1551–1572 (2014)

    MATH  MathSciNet  Article  Google Scholar 

  54. 54.

    Sorber, L., Van Barel, M., De Lathauwer, L.: Tensorlab v2.0. http://www.tensorlab.net/. Accessed January 2014

  55. 55.

    Tao, D., Li, X., Wu, X., Hu, W., Maybank, J.: Supervised tensor learning. Knowl. Inform. Syst. 13(1), 1–42 (2007)

    Article  Google Scholar 

  56. 56.

    Tucker, L.R.: Some mathematical notes on three-mode factor analysis. Psychometrika 31(3), 279–311 (1966)

    MathSciNet  Article  Google Scholar 

  57. 57.

    Vasilescu, M.A.O., Terzopoulos, D.: Multilinear analysis of image ensembles: tensorfaces. In: Proceedings 7th European Conference on Computer Vision. Lecture Notes in Computer Science, vol. 2350, pp. 447–460. Springer-Verlag, Heidelberg (2002)

  58. 58.

    Verschelde, J., Verlinden, J., Cools, R.: Homotopies exploiting newton polytopes for solving sparse polynomial systems. SIAM J. Numer. Anal. 31, 915–930 (1994)

    MATH  MathSciNet  Article  Google Scholar 

  59. 59.

    Wilkinson, J.H.: The evaluation of the zeros of ill-conditioned polynomials. Part I. Numer. Math. 1(1), 150–166 (1959)

    MATH  MathSciNet  Article  Google Scholar 

  60. 60.

    Zarzoso, V., Comon, P.: Optimal step-size constant modulus algorithm. IEEE Trans. Commun. 56(1), 10–13 (2008)

    Article  Google Scholar 

  61. 61.

    Zhang, X., Ling, C., Qi, L.: The best rank-1 approximation of a symmetric tensor and related spherical optimization problems. SIAM J. Matrix Anal. Appl. 33(3), 806–821 (2012)

    MATH  MathSciNet  Article  Google Scholar 

Download references

Acknowledgments

L. Sorber was supported by a doctoral fellowship of the Flanders agency for Innovation by Science and Technology (IWT). I. Domanov and L. De Lathauwer were supported by the Research Council KU Leuven: GOA/10/09 MaNet, CoE PFV/10/002 (OPTEC), PDM postdoc Grant; F.W.O.: project G.0830.14N, G.0881.14N; the Belgian Federal Science Policy Office: IUAP P7 (DYSCO II, Dynamical systems, control and optimization, 2012-2017); EU: The research leading to these results has received funding from the European Research Council under the European Union’s Seventh Framework Programme (FP7/2007-2013) / ERC Advanced Grant: BIOTENSORS (no. 339804). This paper reflects only the authors’ views and the Union is not liable for any use that may be made of the contained information. M. Van Barel was supported by the Research Council KU Leuven: OT/10/038, CoE PF/10/002 (OPTEC); F.W.O.: project G.0828.14N; by the Belgian Network DYSCO (Dynamical Systems, Control, and Optimization), funded by the Interuniversity Attraction Poles Programme, initiated by the Belgian State, Science Policy Office.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Ignat Domanov.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Sorber, L., Domanov, I., Van Barel, M. et al. Exact line and plane search for tensor optimization. Comput Optim Appl 63, 121–142 (2016). https://doi.org/10.1007/s10589-015-9761-5

Download citation

Keywords

  • Exact line search
  • Exact plane search
  • Tensor decomposition
  • Tensor optimization
  • Bivariate polynomial system