Skip to main content
SpringerLink
Log in

Fast randomized matrix and tensor interpolative decomposition using CountSketch

  • Published:
Advances in Computational Mathematics Aims and scope Submit manuscript

Abstract

We propose a new fast randomized algorithm for interpolative decomposition of matrices which utilizes CountSketch. We then extend this approach to the tensor interpolative decomposition problem introduced by Biagioni et al. (J. Comput. Phys. 281(C), 116–134 (2015)). Theoretical performance guarantees are provided for both the matrix and tensor settings. Numerical experiments on both synthetic and real data demonstrate that our algorithms maintain the accuracy of competing methods, while running in less time, achieving at least an order of magnitude speedup on large matrices and tensors.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Ailon, N., Chazelle, B.: The fast Johnson–Lindenstrauss transform and approximate nearest neighbors. SIAM J. Comput. 39(1), 302–322 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  2. Atkinson, K., Han, W.: Theoretical Numerical Analysis: a Functional Analysis Framework. Number 39 in Texts in Applied Mathematics, 3rd edn. Springer, New York (2009). ISBN 978-1-4419-0457-7

    Google Scholar 

  3. Avron, H., Maymounkov, P., Toledo, S.: Blendenpik: supercharging LAPACK’s least-squares solver. SIAM J. Sci. Comput. 32(3), 1217–1236 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  4. Avron, H., Nguyen, H.L., Woodruff, D.P.: Subspace embeddings for the polynomial kernel. In: Proceedings of the 27th International Conference on Neural Information Processing Systems, vol. 2, pp 2258–2266. MIT Press (2014)

  5. Battaglino, C., Ballard, G., Kolda, T.G.: A practical randomized CP tensor decomposition. SIAM Journal on Matrix Analysis and Applications 39 (2), 876–901 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  6. Beylkin, G., Mohlenkamp, M.J.: Numerical operator calculus in higher dimensions. Proc. Natl. Acad. Sci. 99(16), 10246–10251 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  7. Beylkin, G., Mohlenkamp, M.J.: Algorithms for numerical analysis in high dimensions. SIAM J. Sci. Comput. 26(6), 2133–2159 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  8. Biagioni, D.J., Beylkin, D., Beylkin, G.: Randomized interpolative decomposition of separated representations. J. Comput. Phys. 281(C), 116–134 (2015). ISSN 0021-9991. https://doi.org/10.1016/j.jcp.2014.10.009

    Article  MathSciNet  MATH  Google Scholar 

  9. Bien, J., Xu, Y., Mahoney, M.W.: CUR from a sparse optimization viewpoint. In: Advances in Neural Information Processing Systems, pp 217–225 (2010)

  10. Boutsidis, C., Drineas, P., Magdon-Ismail, M.: Near-optimal column-based matrix reconstruction. SIAM J. Comput. 43(2), 687–717 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  11. Boutsidis, C., Mahoney, M.W., Drineas, P.: An improved approximation algorithm for the column subset selection problem. In: Proceedings of the Twentieth Annual ACM-SIAM Symposium on Discrete Algorithms, pp 968–977. SIAM (2009)

  12. Boutsidis, C., Woodruff, D.P.: Optimal CUR matrix decompositions. SIAM J. Comput. 46(2), 543–589 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  13. Caiafa, C.F., Cichocki, A.: Generalizing the column–row matrix decomposition to multi-way arrays. Linear Algebra Appl. 433(3), 557–573 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  14. Charikar, M., Chen, K., Farach-Colton, M.: Finding frequent items in data streams. Theoretical Computer Science 312(1), 3–15 (2004). ISSN 0304-3975. https://doi.org/10.1016/S0304-3975(03)00400-6

    Article  MathSciNet  MATH  Google Scholar 

  15. Cheng, H., Gimbutas, Z., Martinsson, P.-G., Rokhlin, V.: On the compression of low rank matrices. SIAM Journal on Scientific Computing 26(4), 1389–1404 (2005). ISSN 1064-8275. https://doi.org/10.1137/030602678

    Article  MathSciNet  MATH  Google Scholar 

  16. Clarkson, K.L., Woodruff, D.P.: Low-rank approximation and regression in input sparsity time. Journal of the ACM 63(6), 54:1–54:45 (2017). ISSN 0004-5411. https://doi.org/10.1145/3019134

    Article  MathSciNet  MATH  Google Scholar 

  17. da Costa, M.N., Lopes, R.R., Romano, J.M.T.: Randomized methods for higher-order subspace separation. In: 2016 24th European Signal Processing Conference (EUSIPCO), pp 215–219. IEEE (2016)

  18. Deshpande, A., Rademacher, L.: Efficient volume sampling for row/column subset selection. In: 2010 IEEE 51st Annual Symposium on Foundations of Computer Science, pp 329–338. IEEE (2010)

  19. Deshpande, A., Rademacher, L., Vempala, S., Wang, G.: Matrix approximation and projective clustering via volume sampling. In: Proceedings of the Seventeenth Annual ACM-SIAM Symposium on Discrete Algorithm. Society for Industrial and Applied Mathematics, pp 1117–1126 (2006)

  20. Deshpande, A., Vempala, S.: Adaptive sampling and fast low-rank matrix approximation. In: Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques, pp 292–303. Springer (2006)

  21. Diao, H., Song, Z., Sun, W., Woodruff, D.: Sketching for Kronecker product regression and P-splines. In: Proceedings of the 21st International Conference on Artificial Intelligence and Statistics, pp 1299–1308 (2018)

  22. Drineas, P., Kannan, R.: Pass efficient algorithms for approximating large matrices. In: Proceedings of the Fourteenth Annual ACM-SIAM Symposium on Discrete Algorithms, pp 223–232. Society for Industrial and Applied Mathematics, Baltimore (2003)

  23. Drineas, P., Kannan, R., Mahoney, M.W.: Fast Monte Carlo algorithms for matrices III Computing a compressed approximate matrix decomposition. SIAM J. Comput. 36(1), 184–206 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  24. Drineas, P., Mahoney, M.W.: A randomized algorithm for a tensor-based generalization of the singular value decomposition. Linear Algebra and its Applications 420(2-3), 553–571 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  25. Drineas, P., Mahoney, M.W., Muthukrishnan, S.: Relative-error CUR matrix decompositions. SIAM Journal on Matrix Analysis and Applications 30 (2), 844–881 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  26. Friedland, S., Mehrmann, V., Miedlar, A., Nkengla, M.: Fast low rank approximations of matrices and tensors. Electronic Journal of Linear Algebra 22, 1031–1048 (2011). ISSN 1081-3810. https://doi.org/10.13001/1081-3810.1489

    Article  MathSciNet  MATH  Google Scholar 

  27. Frieze, A., Kannan, R., Vempala, S.: Fast Monte-Carlo algorithms for finding low-rank approximations. Journal of the ACM (JACM) 51(6), 1025–1041 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  28. Golub, G.H., Van Loan, C.F.: Matrix Computation, 4th edn. Johns Hopkins University Press, Baltimore (2013). ISBN 978-1-4214-0794-4

    MATH  Google Scholar 

  29. Goreinov, S.A., Tyrtyshnikov, E.E., Zamarashkin, N.L.: Pseudo-skeleton approximations by matrices of maximal volume. Mathematical Notes 62(4), 515–519 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  30. Goreinov, S.A., Tyrtyshnikov, E.E., Zamarashkin, N.L.: A theory of pseudoskeleton approximations. Linear Algebra Appl. 261(1-3), 1–21 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  31. Gu, M., Eisenstat, S.C.: Efficient algorithms for computing a strong rank-revealing QR factorization. SIAM Journal on Scientific Computing 17(4), 848–869 (1996). ISSN 1064-8275. https://doi.org/10.1137/0917055

    Article  MathSciNet  MATH  Google Scholar 

  32. Guruswami, V., Sinop, A.K.: Optimal column-based low-rank matrix reconstruction. In: Proceedings of the Twenty-Third Annual ACM-SIAM Symposium on Discrete Algorithms, pp 1207–1214. SIAM (2012)

  33. Halko, N., Martinsson, P.-G., Tropp, J.A.: Finding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions. SIAM Rev. 53(2), 217–288 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  34. Iwen, M.A., Needell, D., Rebrova, E., Zare, A.: Lower memory oblivious (tensor) subspace embeddings with fewer random bits. Modewise Methods for Least Squares. arXiv:1912.08294 (2019)

  35. Jin, R., Kolda, T.G., Ward, R.: Faster Johnson-Lindenstrauss transforms via Kronecker Products. arXiv:1909.04801 (2019)

  36. Kolda, T.G., Bader, B.W.: Tensor decompositions and applications. SIAM Review 51(3), 455–500 (2009). ISSN 0036-1445. https://doi.org/10.1137/07070111X

    Article  MathSciNet  MATH  Google Scholar 

  37. Liberty, E., Woolfe, F., Martinsson, P.-G., Rokhlin, V., Tygert, M.: Randomized algorithms for the low-rank approximation of matrices. Proceedings of the National Academy of Sciences 104(51), 20167–20172 (2007). ISSN 0027-8424, 1091-6490. https://doi.org/10.1073/pnas.0709640104

    Article  MathSciNet  MATH  Google Scholar 

  38. Mahoney, M.W., Drineas, P.: CUR matrix decompositions for improved data analysis. Proc. Natl. Acad. Sci. 106(3), 697–702 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  39. Mahoney, M.W., Maggioni, M., Drineas, P.: Tensor-CUR decompositions for tensor-based data. SIAM Journal on Matrix Analysis and Applications 30(3), 957–987 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  40. Malik, O.A., Becker, S.: Low-rank Tucker decomposition of large tensors using TensorSketch. In: Advances in Neural Information Processing Systems, pp 10096–10106 (2018)

  41. Malik, O.A., Becker, S.: Guarantees for the Kronecker fast Johnson–Lindenstrauss transform using a coherence and sampling argument. Linear Algebra and its Applications 602, 120–137 (2020). ISSN 0024-3795. https://doi.org/10.1016/j.laa.2020.05.004

    Article  MathSciNet  MATH  Google Scholar 

  42. Martinsson, P.-G., Rokhlin, V., Tygert, M.: A randomized algorithm for the decomposition of matrices. Applied and Computational Harmonic Analysis 30(1), 47–68 (2011). ISSN 1063-5203. https://doi.org/10.1016/j.acha.2010.02.003

    Article  MathSciNet  MATH  Google Scholar 

  43. Oseledets, I.V., Savostianov, D.V., Tyrtyshnikov, E.E.: Tucker dimensionality reduction of three-dimensional arrays in linear time. SIAM Journal on Matrix Analysis and Applications 30(3), 939–956 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  44. Pagh, R.: Compressed matrix multiplication. ACM Transactions on Computation Theory 5(3), 9:1–9:17 (2013). ISSN 1942-3454. https://doi.org/10.1145/2493252.2493254

    Article  MathSciNet  MATH  Google Scholar 

  45. Pham, N., Pagh, R.: Fast and scalable polynomial kernels via explicit feature maps. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’13. ISBN 978-1-4503-2174-7. https://doi.org/10.1145/2487575.2487591, pp 239–247. ACM, New York (2013)

  46. Rakhshan, B.T., Rabusseau, G.: Tensorized random projections. arXiv:2003.05101 (2020)

  47. Reynolds, M.J., Beylkin, G., Doostan, A.: Optimization via separated representations and the canonical tensor decomposition. J. Comput. Phys. 348(C), 220–230 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  48. Reynolds, M.J., Doostan, A., Beylkin, G.: Randomized alternating least squares for canonical tensor decompositions application to a PDE with random data. SIAM J. Sci. Comput. 38(5), A2634–A2664 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  49. Smith, S., Huang, K., Sidiropoulos, N.D., Karypis, G.: Streaming tensor factorization for infinite data sources. In: Proceedings of the SIAM International Conference on Data Mining, p 2018. SIAM (2018)

  50. Sun, Y., Guo, Y., Tropp, J.A., Udell, M.: Tensor random projection for low memory dimension reduction. In: NeurIPS Workshop on Relational Representation Learning (2018)

  51. Tarzanagh, D.A., Michailidis, G.: Fast randomized algorithms for t-product based tensor operations and decompositions with applications to imaging data. SIAM Journal on Imaging Sciences 11(4), 2629–2664 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  52. Tsourakakis, C.E.: Mach: fast randomized tensor decompositions. In: Proceedings of the 2010 SIAM International Conference on Data Mining, pp 689–700. SIAM (2010)

  53. Tyrtyshnikov, E.E.: Incomplete cross approximation in the Mosaic-Skeleton method. Computing 64, 367–380 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  54. Voronin, S., Martinsson, P.-G.: Efficient algorithms for CUR and interpolative matrix decompositions. Adv. Comput. Math. 43(3), 495–516 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  55. Wang, S., Zhang, Z.: Improving CUR matrix decomposition and the nyström approximation via adaptive sampling. The Journal of Machine Learning Research 14(1), 2729–2769 (2013)

    MATH  Google Scholar 

  56. Wang, Y., Tung, H.-Y., Smola, A.J., Anandkumar, A.: Fast and guaranteed tensor decomposition via sketching. In: Advances in Neural Information Processing Systems, pp 991–999 (2015)

  57. Woolfe, F., Liberty, E., Rokhlin, V., Tygert, M.: A fast randomized algorithm for the approximation of matrices. Applied and Computational Harmonic Analysis 25(3), 335–366 (2008). ISSN 1063-5203. https://doi.org/10.1016/j.acha.2007.12.002

    Article  MathSciNet  MATH  Google Scholar 

  58. Yang, B., Zamzam, A., Sidiropoulos, N.D.: ParaSketch: parallel tensor factorization via sketching. In: Proceedings of the 2018 SIAM International Conference on Data Mining, pp 396–404. SIAM (2018)

  59. Zhang, J., Saibaba, A.K., Kilmer, M.E., Aeron, S.: A randomized tensor singular value decomposition based on the t-product. Numerical Linear Algebra with Applications 25, e2179 (2018)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Applied Mathematics, University of Colorado Boulder, Boulder, CO, USA

    Osman Asif Malik & Stephen Becker

Authors
  1. Osman Asif Malik
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stephen Becker
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Osman Asif Malik.

Additional information

Communicated by: Gunnar J Martinsson

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Malik, O.A., Becker, S. Fast randomized matrix and tensor interpolative decomposition using CountSketch. Adv Comput Math 46, 76 (2020). https://doi.org/10.1007/s10444-020-09816-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10444-020-09816-9

Keywords

Mathematics Subject Classification (2010)

Search

Navigation