A nonlinear kernel support matrix machine for matrix learning

  • Yunfei YeEmail author
Original Article


In many problems of supervised tensor learning, real world data such as face images or MRI scans are naturally represented as matrices, which are also called as second order tensors. Most existing classifiers based on tensor representation, such as support tensor machine and kernelized support tensor machine need to solve iteratively which occupy much time and may suffer from local minima. In this paper, we present a kernel support matrix machine which performs a matrix-form inner product with maximum margin classifier. Specifically, the matrix inner product is introduced to leverage the inherent structural information within matrix data. Further, matrix kernel functions are applied to detect the nonlinear relationships. We analyze a unifying optimization problem for which we propose an asymptotically convergent algorithm. Theoretical analysis for the generalization bounds is derived based on Rademacher complexity with respect to a probability distribution. We demonstrate the merits of the proposed method by exhaustive experiments on both simulation study and a number of real-word datasets from a variety of application domains.


Kernel support matrix machine Supervised tensor learning Reproducing kernel matrix Hilbert space Kernel functions 



The work is supported by National Natural Science Foundations of China under Grant 11531001 and National Program on Key Basic Research Project under Grant 2015CB856004. We are grateful to Dong Han for our discussions.


  1. 1.
    Cai D, He X, Han J (2006) Learning with tensor representation. Technical report, Computer Science Department, UIUC, UIUCDCS-R-2006-2716.
  2. 2.
    Chen Y, Wang K, Zhong P (2016) One-class support tensor machine. Knowl Based Syst 96:14–28CrossRefGoogle Scholar
  3. 3.
    Chu C, Kim SK, Lin YA, Yu Y, Bradski G, Ng AY, Olukotun K (2007) Map-reduce for machine learning on multicore. In: Proceedings of the 2006 conference on advances in neural information processing systems, vol 19. MIT Press, Cambridge, pp 281–288Google Scholar
  4. 4.
    Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273–297zbMATHGoogle Scholar
  5. 5.
    Erfani SM, Baktashmotlagh M, Rajasegarar S, Nguyen V, Leckie C, Bailey J, Ramamohanarao K (2016) R1stm: One-class support tensor machine with randomised kernel. In: Proceedings of the 2016 SIAM international conference on data mining. SIAM, pp 198–206.
  6. 6.
    Evgeniou T, Micchelli CA, Pontil M (2005) Learning multiple tasks with kernel methods. J Mach Learn Res 6:615–637MathSciNetzbMATHGoogle Scholar
  7. 7.
    Gao X, Fan L, Xu H (2018) Multiple rank multi-linear kernel support vector machine for matrix data classification. Int J Mach Learn Cybernet 9(2):251–261CrossRefGoogle Scholar
  8. 8.
    Hao Z, He L, Chen B, Yang X (2013) A linear support higher-order tensor machine for classification. IEEE Trans Image Process 22(7):2911–2920CrossRefGoogle Scholar
  9. 9.
    He L, Kong X, Yu PS, Yang X, Ragin AB, Hao Z (2014) Dusk: a dual structure-preserving kernel for supervised tensor learning with applications to neuroimages. In: Proceedings of the 2014 SIAM international conference on data mining. SIAM, pp 127–135.
  10. 10.
    He L, Lu CT, Ma G, Wang S, Shen L, Philip SY, Ragin AB (2017) Kernelized support tensor machines. In: Proceedings of the 34th international conference on machine learning, vol 70, pp 1442–1451Google Scholar
  11. 11.
    Horn RA (1990) The hadamard product. In: Proceedings of symposia in applied mathematics, vol 40. American Mathematical Society, Providence, pp 87–169Google Scholar
  12. 12.
    Joachims T (1999) Transductive inference for text classification using support vector machines. In: Proceedings of the 16th international conference on machine learning, vol 99, pp 200–209Google Scholar
  13. 13.
    Kadri H, Duflos E, Preux P, Rakotomamonjy A, Audiffren J (2016) Operator-valued kernels for learning from functional response data. J Mach Learn Res 17(1):613–666MathSciNetzbMATHGoogle Scholar
  14. 14.
    Khemchandani R, Chandra S et al (2007) Twin support vector machines for pattern classification. IEEE Trans Pattern Anal Mach Intell 29(5):905–910zbMATHCrossRefGoogle Scholar
  15. 15.
    Le Gall F (2014) Powers of tensors and fast matrix multiplication. In: Proceedings of the 39th international symposium on symbolic and algebraic computation. ACM, pp 296–303.
  16. 16.
    Luo L, Xie Y, Zhang Z, Li WJ (2015) Support matrix machines. In: Proceedings of the 32nd international conference on machine learning, vol 37, pp 938–947Google Scholar
  17. 17.
    Micchelli CA, Pontil MA (2005) On learning vector-valued functions. Neural Comput 17(1):177–204MathSciNetzbMATHCrossRefGoogle Scholar
  18. 18.
    Nene SA, Nayar SK, Murase H (1996) Columbia object image library (coil-20). Technical report, Columbia University, CUCS-005-96.
  19. 19.
    Platt JC (1999) Fast training of support vector machines using sequential minimal optimization. In: Schölkopf B, Burges CJC, Smola AJ (eds) Advances in kernel methods—support vector learning. MIT Press, Cambridge, pp 185–208Google Scholar
  20. 20.
    Reisert M, Burkhardt H (2007) Learning equivariant functions with matrix valued kernels. J Mach Learn Res 8:385–408MathSciNetzbMATHGoogle Scholar
  21. 21.
    Samaria FS, Harter AC (1994) Parameterisation of a stochastic model for human face identification. In: Proceedings of 1994 IEEE workshop on applications of computer vision. IEEE, pp 138–142.
  22. 22.
    Schölkopf B, Sung KK, Burges CJ, Girosi F, Niyogi P, Poggio T, Vapnik V (1997) Comparing support vector machines with gaussian kernels to radial basis function classifiers. IEEE Trans Signal Process 45(11):2758–2765CrossRefGoogle Scholar
  23. 23.
    Schölkopf B, Smola AJ, Williamson RC, Bartlett PL (2000) New support vector algorithms. Neural Comput 12(5):1207–1245CrossRefGoogle Scholar
  24. 24.
    Schölkopf B, Platt JC, Shawe-Taylor J, Smola AJ, Williamson RC (2001) Estimating the support of a high-dimensional distribution. Neural Comput 13(7):1443–1471zbMATHCrossRefGoogle Scholar
  25. 25.
    Shalev-Shwartz S, Ben-David S (2014) Understanding machine learning: from theory to algorithms. Cambridge University Press, New YorkzbMATHCrossRefGoogle Scholar
  26. 26.
    Signoretto M, De Lathauwer L, Suykens JA (2011) A kernel-based framework to tensorial data analysis. Neural Netw 24(8):861–874zbMATHCrossRefGoogle Scholar
  27. 27.
    Stitson MO, Gammerman A, Vapnik V, Vovk V, Watkins C, Weston J (1997) Support vector regression with anova decomposition kernels. In: Schölkopf B, Burges CJC, Smola AJ (eds) Advances in kernel methods—support vector learning. MIT Press, Cambridge, pp 285–292Google Scholar
  28. 28.
    Suykens JA, Vandewalle J (1999) Least squares support vector machine classifiers. Neural Process Lett 9(3):293–300CrossRefGoogle Scholar
  29. 29.
    Tao D, Li X, Hu W, Maybank S, Wu X (2007) Supervised tensor learning. Knowl Inf Syst 13(1):1–42CrossRefGoogle Scholar
  30. 30.
    Vapnik V (1995) The nature of statistical learning theory. Springer, New YorkzbMATHCrossRefGoogle Scholar
  31. 31.
    Weston J, Gammerman A, Stitson M, Vapnik V, Vovk V, Watkins C (1997) Density estimation using support vector machines. In: Schölkopf B, Burges CJC, Smola AJ (eds) Advances in kernel methods—support vector learning. MIT Press, Cambridge, pp 293–306Google Scholar
  32. 32.
    Wong WK, Lai Z, Xu Y, Wen J, Ho CP (2015) Joint tensor feature analysis for visual object recognition. IEEE Trans Cybern 45(11):2425–2436CrossRefGoogle Scholar
  33. 33.
    Yang Y, Liu X (1999) A re-examination of text categorization methods. In: Proceedings of the 22nd annual international ACM SIGIR conference on research and development in information retrieval. ACM, pp 42–49.
  34. 34.
    Ye Y (2017) The Matrix Hilbert space and its application to matrix learning. arXiv:1706.08110
  35. 35.
    Zhou H, Li L (2014) Regularized matrix regression. J R Stat Soc B 76(2):463–483MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Mathematical SciencesShanghai Jiao Tong UniversityShanghaiChina

Personalised recommendations