Advertisement

Algorithmica

pp 1–17 | Cite as

On Using Toeplitz and Circulant Matrices for Johnson–Lindenstrauss Transforms

  • Casper Benjamin FreksenEmail author
  • Kasper Green Larsen
Article
  • 7 Downloads
Part of the following topical collections:
  1. Special Issue: Algorithms and Computation

Abstract

The Johnson–Lindenstrauss lemma is one of the cornerstone results in dimensionality reduction. A common formulation of it, is that there exists a random linear mapping \(f : {\mathbb {R}}^n \rightarrow {\mathbb {R}}^m\) such that for any vector \(x \in {\mathbb {R}}^n\), f preserves its norm to within \((1 {\pm } \varepsilon )\) with probability \(1 - \delta \) if \(m = \varTheta (\varepsilon ^{-2} \lg (1/\delta ))\). Much effort has gone into developing fast embedding algorithms, with the Fast Johnson–Lindenstrauss transform of Ailon and Chazelle being one of the most well-known techniques. The current fastest algorithm that yields the optimal \(m = {\mathcal {O}}(\varepsilon ^{-2}\lg (1/\delta ))\) dimensions has an embedding time of \({\mathcal {O}}(n \lg n + \varepsilon ^{-2} \lg ^3 (1/\delta ))\). An exciting approach towards improving this, due to Hinrichs and Vybíral, is to use a random \(m \times n\) Toeplitz matrix for the embedding. Using Fast Fourier Transform, the embedding of a vector can then be computed in \({\mathcal {O}}(n \lg m)\) time. The big question is of course whether \(m = {\mathcal {O}}(\varepsilon ^{-2} \lg (1/\delta ))\) dimensions suffice for this technique. If so, this would end a decades long quest to obtain faster and faster Johnson–Lindenstrauss transforms. The current best analysis of the embedding of Hinrichs and Vybíral shows that \(m = {\mathcal {O}}(\varepsilon ^{-2}\lg ^2 (1/\delta ))\) dimensions suffice. The main result of this paper, is a proof that this analysis unfortunately cannot be tightened any further, i.e., there exist vectors requiring \(m = \varOmega (\varepsilon ^{-2} \lg ^2 (1/\delta ))\) for the Toeplitz approach to work.

Keywords

Dimensionality reduction Johnson–Lindenstrauss Toeplitz matrices 

Mathematics Subject Classification

68Q25 68W40 

Notes

References

  1. 1.
    Achlioptas, D.: Database-friendly random projections: Johnson–Lindenstrauss with binary coins. J. Comput. Syst. Sci. 66(4), 671–687 (2003).  https://doi.org/10.1016/S0022-0000(03)00025-4 MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Ailon, N., Chazelle, B.: The fast Johnson–Lindenstrauss transform and approximate nearest neighbors. SIAM J. Comput. 39(1), 302–322 (2009).  https://doi.org/10.1137/060673096 MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Ailon, N., Liberty, E.: Fast dimension reduction using Rademacher series on dual BCH codes. Discrete Comput. Geom. 42(4), 615–630 (2009).  https://doi.org/10.1007/s00454-008-9110-x MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Ailon, N., Liberty, E.: An almost optimal unrestricted fast Johnson–Lindenstrauss transform. ACM Trans. Algorithms 9(3), 21:1–21:12 (2013).  https://doi.org/10.1145/2483699.2483701 MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Blocki, J., Blum, A., Datta, A., Sheffet, O.: The Johnson–Lindenstrauss transform itself preserves differential privacy. In: Proceedings of the 2012 IEEE 53rd Annual Symposium on Foundations of Computer Science, FOCS ’12, pp. 410–419. IEEE Computer Society, Washington, DC (2012).  https://doi.org/10.1109/FOCS.2012.67
  6. 6.
    Boutsidis, C., Zouzias, A., Mahoney, M.W., Drineas, P.: Randomized dimensionality reduction for \(k\)-means clustering. IEEE Trans. Inf. Theory 61(2), 1045–1062 (2015).  https://doi.org/10.1109/TIT.2014.2375327 MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Candes, E.J., Romberg, J., Tao, T.: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52(2), 489–509 (2006).  https://doi.org/10.1109/TIT.2005.862083 MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Cohen, M.B., Elder, S., Musco, C., Musco, C., Persu, M.: Dimensionality reduction for \(k\)-means clustering and low rank approximation. In: Proceedings of the Forty-Seventh Annual ACM Symposium on Theory of Computing, STOC ’15, pp. 163–172. ACM, New York (2015).  https://doi.org/10.1145/2746539.2746569
  9. 9.
    Dasgupta, A., Kumar, R., Sarlos, T.: A sparse Johnson–Lindenstrauss transform. In: Proceedings of the Forty-Second ACM Symposium on Theory of Computing, STOC ’10, pp. 341–350. ACM, New York (2010).  https://doi.org/10.1145/1806689.1806737
  10. 10.
    Dasgupta, S., Gupta, A.: An elementary proof of a theorem of Johnson and Lindenstrauss. Random Struct. Algorithms 22(1), 60–65 (2003).  https://doi.org/10.1002/rsa.10073 MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Donoho, D.L.: Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006).  https://doi.org/10.1109/TIT.2006.871582 MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Har-Peled, S., Indyk, P., Motwani, R.: Approximate nearest neighbor: towards removing the curse of dimensionality. Theory Comput 8(14), 321–350 (2012).  https://doi.org/10.4086/toc.2012.v008a014 MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Hinrichs, A., Vybíral, J.: Johnson–Lindenstrauss lemma for circulant matrices. Random Struct. Algorithms 39(3), 391–398 (2011).  https://doi.org/10.1002/rsa.20360 MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Indyk, P.: Algorithmic applications of low-distortion geometric embeddings. In: Proceedings of the 42nd IEEE Symposium on Foundations of Computer Science, FOCS ’01, pp. 10–33. IEEE Computer Society, Washington, DC (2001).  https://doi.org/10.1109/SFCS.2001.959878
  15. 15.
    Johnson, W.B., Lindenstrauss, J.: Extensions of Lipschitz mappings into a Hilbert space. Contemp. Math. 26, 189–206 (1984).  https://doi.org/10.1090/conm/026/737400 MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Kane, D.M., Nelson, J.: Sparser Johnson–Lindenstrauss transforms. J. ACM 61(1), 4:1–4:23 (2014).  https://doi.org/10.1145/2559902 MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Krahmer, F., Ward, R.: New and improved Johnson–Lindenstrauss embeddings via the restricted isometry property. SIAM J. Math. Anal. 43(3), 1269–1281 (2011)MathSciNetCrossRefGoogle Scholar
  18. 18.
    Larsen, K.G., Nelson, J.: Optimality of the Johnson–Lindenstrauss lemma. In: Proceedings of the 2017 IEEE 58th Annual Symposium on Foundations of Computer Science, FOCS ’17. IEEE Computer Society, Washington, DC (2017)Google Scholar
  19. 19.
    Muthukrishnan, S.: Data streams: algorithms and applications. In: Foundations and Trends™ in Theoretical Computer Science, vol. 1, no 2. Publishers Inc., Hanover (2005).  https://doi.org/10.1561/0400000002 MathSciNetCrossRefGoogle Scholar
  20. 20.
    Spielman, D.A., Srivastava, N.: Graph sparsification by effective resistances. SIAM J. Comput. 40(6), 1913–1926 (2011).  https://doi.org/10.1137/080734029 MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Vempala, S.S.: The random projection method. In: DIMACS—Series in Discrete Mathematics and Theoretical Computer Science, vol. 65. American Mathematical Society, Providence (2004).  https://doi.org/10.1007/978-1-4615-0013-1_16 CrossRefGoogle Scholar
  22. 22.
    Vu, K., Poirion, P.L., Liberti, L.: Using the Johnson–Lindenstrauss lemma in linear and integer programming. ArXiv e-prints (2015)Google Scholar
  23. 23.
    Vybíral, J.: A variant of the Johnson–Lindenstrauss lemma for circulant matrices. J. Funct. Anal. 260(4), 1096–1105 (2011).  https://doi.org/10.1016/j.jfa.2010.11.014 MathSciNetCrossRefzbMATHGoogle Scholar
  24. 24.
    Weinberger, K., Dasgupta, A., Langford, J., Smola, A., Attenberg, J.: Feature hashing for large scale multitask learning. In: Proceedings of the 26th Annual International Conference on Machine Learning, ICML ’09, pp. 1113–1120. ACM, New York (2009).  https://doi.org/10.1145/1553374.1553516
  25. 25.
    Woodruff, D.P.: Sketching as a tool for numerical linear algebra. In: Foundations and Trends™ in Theoretical Computer Science, vol. 10, no 1–2. Publishers Inc., Hanover (2014).  https://doi.org/10.1561/0400000060 CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Aarhus UniversityAarhus NDenmark

Personalised recommendations