Abstract
We consider the problem of fast approximate kernel regression. Since kernels can map input features into the infinite dimensional space, kernel trick is used to make the algorithms tractable. However on large data set time complexity of \(O(n^2)\) is prohibitive. Therefore, various approximation methods are employed, such as randomization. A Nyström method (based on a random selection of columns) is usually employed. Main advantage of this algorithm is its time complexity which is reduced to \(O(n m^2 + m^3)\). Space complexity is also reduced to O(nm) because it does not require the computation of the entire matrix. An arbitrary number \(m \ll n\) represents both the size of a random subset of an input set and the dimension of random feature vectors. A Nyström method can be extended with the randomized SVD so that l (where \(l > m\)) randomly selected columns of a kernel matrix without replacement are used for a construction of m-dimensional random feature vectors while keeping time complexity linear in n. Approximated matrix computed in this way is a better approximation than the matrix computed via the Nyström method. We will prove here that the expected error of the approximated kernel predictor derived via this method is approximately the same in expectation as the error of the error of kernel predictor. Furthermore, we will empirically show that using the l randomly selected columns of a kernel matrix for a construction of m-dimensional random feature vectors produces smaller error on a regression problem, than using m randomly selected columns.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Bach, F.: Sharp analysis of low-rank kernel matrix approximations. In: 2013 Proceedings of Machine Learning Research, 26th Annual Conference on Learning Theory (COLT), vol. 30, pp. 185–209, Princeton (2013)
Gross, D., Nesme, V.: Note on sampling without replacing from a finite collection of matrices. arXiv preprint arXiv:1001.2738 (2010)
Halko, N., Martinsson, P.-G., Tropp, J.A.: Finding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions. SIAM Rev. 53(2), 217–288 (2011)
Le, Q., Sarlós, T., Smola, A.: Fastfood-approximating kernel expansions in loglinear time. In: 2013 Proceedings of the 30th International Conference on Machine Learning, JMLR: W&CP, vol. 28, Atlanta, Georgia, USA (2013)
Li, M., Bi, W., Kwok, J.T., Lu, B.-L.: Large-scale Nyström kernel matrix approximation using randomized SVD. IEEE Trans. Neural Netw. Learn. Syst. 26(1), 152–164 (2015)
Mackey, L., Jordan, M.I., Chen, R.Y., Farrell, B., Tropp, J.A., et al.: Matrix concentration inequalities via the method of exchangeable pairs. Ann. Probab. 42(3), 906–945 (2014)
McWilliams, B., Balduzzi, D., Buhmann, J.M.: Correlated random features for fast semi-supervised learning. In: Advances in Neural Information Processing Systems (NIPS 2013), vol. 26, pp. 440–448 (2013)
Rahimi, A., Recht, B.: Random features for large-scale kernel machines. In: Advances in Neural Information Processing Systems (NIPS 2007), vol. 20, pp. 1177–1184 (2008)
Rahimi, A., Recht, B.: Weighted sums of random kitchen sinks: replacing minimization with randomization in learning. In: Advances in Neural Information Processing Systems (NIPS 2008), vol. 21, pp. 1313–1320 (2009)
Williams, C., Seeger, M.: Using the Nyström method to speed up kernel machines. In: Advances in Neural Information Processing Systems (NIPS 2000), vol. 13, pp. 682–688 (2001)
Yang, T., Li, Y.-F., Mahdavi, M., Jin, R., Zhou, Z.-H.: Nyström method vs random fourier features: a theoretical andempirical comparison. In: Advances in Neural Information Processing Systems, vol. 25 (NIPS 2012), pp. 476–484 (2012)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Trokicić, A., Todorović, B. (2019). Randomized Nyström Features for Fast Regression: An Error Analysis. In: Ćirić, M., Droste, M., Pin, JÉ. (eds) Algebraic Informatics. CAI 2019. Lecture Notes in Computer Science(), vol 11545. Springer, Cham. https://doi.org/10.1007/978-3-030-21363-3_21
Download citation
DOI: https://doi.org/10.1007/978-3-030-21363-3_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-21362-6
Online ISBN: 978-3-030-21363-3
eBook Packages: Computer ScienceComputer Science (R0)