Skip to main content
SpringerLink
Log in

Approximate kernel partial least squares

  • Published:
Annals of Mathematics and Artificial Intelligence Aims and scope Submit manuscript

Abstract

As an extension of partial least squares (PLS), kernel partial least squares (KPLS) is an very important methods to find nonlinear patterns from data. However, the application of KPLS to large-scale problems remains a big challenge, due to storage and computation issues in the number of examples. To address this limitation, we utilize randomness to design scalable new variants of the kernel matrix to solve KPLS. Specifically, we consider the spectral properties of low-rank kernel matrices constructed as sums of random feature dot-products and present a new method called randomized kernel partial least squares (RKPLS) to approximate KPLS. RKPLS can alleviate the computation requirements of approximate KPLS with linear space and computation in the sample size. Theoretical analysis and experimental results show that the solution of our algorithm converges to exact kernel matrix in expectation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Wold, S., Ruhe, A., Wold, H., Dunn, W.J.: The collinearity problem in linear regression. The partial least squares (PLS) approach to generalized inverse. SIAM J Sci Stat. Comput. 5, 735–743 (1984)

    Article  Google Scholar 

  2. Helland, S.: On the structure of partial least squares regression. Commun Stat-Simul Comput. 17(2), 581–607 (1988)

    Article  MathSciNet  Google Scholar 

  3. Pan, W.T.: A newer equal part linear regression model : a case study of the influence of educational input on gross national income. EURASIA J Math SciTechn Educ. 13(8), 5765–5773 (2017)

    Google Scholar 

  4. Fang, X., Xu, Y., Li, X., Lai, Z., Wong, W.K., Fang, B.: Regularized label relaxation linear regression. IEEE Trans Neural Netw Learn Syst. 29(4), 1006–1018 (2018)

    Article  Google Scholar 

  5. Dong, Y., Tao, D., Li, X., Ma, J., Pu, J.: Texture classification and retrieval using shearlets and linear regression. IEEE Trans Cybern. 45(3), 358–369 (2015)

    Article  Google Scholar 

  6. Frank, I., Friedman, J.: A statistical view of some chemometrics regression tools. Technometrics. 35, 109–148 (1993)

    Article  Google Scholar 

  7. Krämer, N. and Braun, M.: Kernelizing PLS, degrees of freedom, and efficient model selection. In Proceedings of the 24th International Conference on Machine Learning, pp.441–448 (2007)

  8. Rosipal, R., Krämer, N.: Overview and recent advances in partial least squares. Lect. Notes Comput. Sci. 3940, 34–51 (2006)

    Article  Google Scholar 

  9. Rosipal, R., Trejo, L.J.: Kernel partial least squares regression in reproducing kernel Hilbert space. J. Mach. Learn. Res. 2, 97–123 (2001)

    MATH  Google Scholar 

  10. Schölkopf, B., Smola, A.J.: Learning with Kernels, 1st edn. MIT Press, Cambridge (2001)

  11. Rosipal, R.: Kernel partial least squares for nonlinear regression and discrimination. Neural Network World. 13, 291–300 (2003)

    Google Scholar 

  12. Talukdar, U., Hazarika, S.M., Gan, J.Q.: A kernel partial least square based feature selection method. Pattern Recogn. 83, 91–106 (2018)

    Article  Google Scholar 

  13. Blanchard, G., Krämer, N.: Kernel partial least squares is universally consistent. In proceedings of the 13th international conference on artificial intelligence and statistics. JMLR Workshop Conf Proc. 9, 57–64 (2010)

    Google Scholar 

  14. Rahimi, A. and Recht, B.: Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. In Advances in Neural Information Processing Systems, pp.1313–1320 (2008)

  15. Le, Q., Sarlós, T., and Smola, A.: Fastfood-approximating kernel expansions in loglinear time. In Proceedings of the 30th international conference on machine learning, Atlanta, USA, JMLR Workshop and Conference Proceedings, vol.28 (2013)

  16. Mackey, L., Jordan, M.I., Chen, R.Y., Farrell, B., Tropp, J.A.: Matrix concentration inequalities via the method of exchangeable pairs. Ann. Probab. (2014)

  17. Lopez-Paz, D., Sra, S., Smola, A., Ghahramani, Z., and Schölkopf, B. : Randomized nonlinear component analysis. In Proceedings of the 31st International Conference on Machine Learning, pp.1359–1367 (2014)

  18. De Bie, T., Cristianini, N., and Rosipal, R. : Eigenproblems in pattern recognition. In Handbook of Geometric Computing, pp.129–170 (2005)

  19. Bennett, K. and Embrechts, M. : An optimization perspective on kernel partial least squares. In Advances in Learning Theory: Methods, Models and Applications, NATO Science Series III: Computer and Systems Sciences, vol.190, IOS Press Amsterdam, pp. 227–250 (2003)

  20. Boyd. S and Vandenberghe. L. Convex Optimization. Cambridge University Press ( 2004)

  21. Momma, M. and Bennett, K.P.: Sparse kernel partial least squares regression. In Proceedings of the Sixteenth Annual Conference on Learning Theory, pp.216–230 (2003)

  22. Aronszajn, N.: Theory of reproducing kernels. Trans. Am. Math. Soc. 68, 337–404 (1950)

    Article  MathSciNet  Google Scholar 

  23. Loosli, G., Canu, S., and Bottou, L. : Training invariant support vector machines using selective sampling. In Large Scale Kernel Machines, pp.301–320. MIT Press (2007)

  24. Rahimi, A. and Recht, B. : Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems (NIPS), vol. 20, pp.1177–1184 (2007)

  25. Williams, C. and Seeger, M. : Using the Nyström method to speed up kernel machines. In Adavances in Neural Infromation Processing Systems, pp.682–688, Cambridge, MA (2001)

  26. Si, S., Hsieh, C.-J., Dhillon, I.: Memory efficient kernel approximation. J. Mach. Learn. Res. 18(20), 1–32 (2017)

    MathSciNet  MATH  Google Scholar 

  27. Singer, M., Krivobokova, T., Munk, A.: Kernel partial least squares for stationary data. J. Mach. Learn. Res. 18(123), 1–41 (2017)

    MathSciNet  MATH  Google Scholar 

  28. Yang, T., Li, Y.-F., Mahdavi, M., Jin, R., and Zhou, Z.-H.: Nyström method vs random Fourier features: A theoretical and empirical comparison. In Advances in Neural Information Processing Systems, pp.485–493 (2012)

  29. Durand, A., Hannula, M., Kontinen, J., Meier, A., and Virtema, J.: Approximation and dependence via multiteam semantics. Annals of Mathematics and Artificial Intelligence, pp.1–24 (2018)

  30. Lin,S. and Zhou, D. :Optimal learning rates for kernel partial least squares. Journal of Fourier Analysis and Applications, 24(3):908–933 (2017)

  31. UCI machine learning repository. [Online] Available: http://archive.ics.uci.edu/ml

  32. LeCun, Y. and Cortes, C. : The MNIST database of handwritten digits. [Online] Available: http://yann.lecun.com/exdb/mnist/

Download references

Acknowledgements

The authors would like to thank the anonymous reviewers for their considerations and suggestions. We would also give thanks to the National Natural Science Foundation of China under grans no.61772020 for supporting our research work.

Author information

Authors and Affiliations

  1. School of Mathematics and Statistics, Xidian University, Xi’an, 710071, Shaanxi, China

    Xiling Liu & Shuisheng Zhou

  2. Department of Basic Science, College of Information and Business, Zhongyuan Technology University, Zhengzhou, 450007, Henan, China

    Xiling Liu

Authors
  1. Xiling Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shuisheng Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shuisheng Zhou.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, X., Zhou, S. Approximate kernel partial least squares. Ann Math Artif Intell 88, 973–986 (2020). https://doi.org/10.1007/s10472-020-09694-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10472-020-09694-3

Keywords

Search

Navigation