Dimensionality reduction via the Johnson–Lindenstrauss Lemma: theoretical and empirical bounds on embedding dimension

  • John Fedoruk
  • Byron Schmuland
  • Julia Johnson
  • Giseon Heo
Article
  • 13 Downloads

Abstract

The Johnson–Lindenstrauss (JL) lemma has led to the development of tools for dealing with datasets in high dimensions. The lemma asserts that a set of high-dimensional points can be projected into lower dimensions, while approximately preserving the pairwise distance structure. Significant improvements of the JL lemma since its inception are summarized. Particular focus is placed on reproving Matoušek’s versions of the lemma (Random Struct Algorithms 33(2):142–156, 2008) first using subgaussian projection coefficients and then using sparse projection coefficients. The results of the lemma are illustrated using simulated data. The simulation suggests a projection that is more effective in terms of dimensionality reduction than is borne out by the theory. This more effective projection was applied to a very large natural, rather than simulated, dataset thus further strengthening empirical evidence of the existence of a better than the proven optimal lower bound on the embedding dimension. Additionally, we provide comparisons with other commonly used data reduction and simplification techniques.

Keywords

Johnson–Lindenstrauss Dimensionality reduction Random projection Subgaussian tail 

Notes

Acknowledgements

Giseon Heo acknowledges funding support provided by McIntyre Memorial Fund, Orthodontic Division, the University of Alberta and Natural Sciences and Engineering Research Council of Canada, Discovery Grant 293180. Thanks to Linglong Kong (University of Alberta) who did preprocessing, data cleaning, and outlier removal of the NYU site resting-status fMRI data used in this study. Thanks to Hongjia Zhang (Laurentian University) who provided visualizations of the results of executing JLT, PCA, and LLE on the ADHD data.

References

  1. 1.
    Achlioptas D (2003) Database-friendly random projections: Johnson–Lindenstrauss with binary coins. J. Comput. Syst. Sci. 66(4):671–687MathSciNetCrossRefMATHGoogle Scholar
  2. 2.
    Ailon N, Chazelle B (2006) Approximate nearest neighbors and the fast Johnson–Lindenstrauss transform. In: Proceedings of the Thirty-Eighth Annual ACM Symposium on Theory of Computing. ACM, pp 557–563Google Scholar
  3. 3.
    Bamberger S, Krahmer F (2017) Optimal fast Johnson–Lindenstrauss embeddings for large data sets. arXiv:1712.01774 [cs.DS]
  4. 4.
    Cannings TI, Samworth RJ (2015) Random-projection ensemble classification. arXiv:1504.04595v2
  5. 5.
    Cohen MB, Jayram TS, Nelson J (2018) simple analyses of the sparse Johnson–Lindenstrauss transform. In: 1st Symposium on Simplicity in Algorithms (SOSA 2018), vol 61(15), pp 1–9Google Scholar
  6. 6.
    Cunningham JP, Yu BM (2014) Dimensionality reduction for large-scale neural recordings. Nat Neurosci 17:1500CrossRefGoogle Scholar
  7. 7.
    Dasgupta S, Gupta A (1999) An elementary proof of the Johnson–Lindenstrauss lemma. Technical Report 99-006, UC BerkeleyGoogle Scholar
  8. 8.
    Dasgupta S (2013) Experiments with random projection. arXiv:1301.3849v1
  9. 9.
    Donoho D (2000) Aide-memoire. high dimensional data analysis: the curses and blessings of dimensionality. http://statweb.stanford.edu/~donoho/Lectures/AMS2000/Curses.pdf
  10. 10.
    Fedoruk J (2016) Dimensionality reduction via the Johnson and Lindenstrauss lemma: mathematical and computational improvements. Master thesis University of AlbertaGoogle Scholar
  11. 11.
    Fedoruk J , Schmuland B, Johnson J, Heo G (2016) Dimensionality reduction via the Johnson–Lindenstrauss lemma. In: Proceedings International Conference on Advances in Big Data Analytics (ABDA’16), Las Vegas, Nevada. CSREA Press, pp 134–139Google Scholar
  12. 12.
    Fern X, Brodley CE (2003) Random projection for high dimensional data clustering: a cluster ensemble approach. In: Proceedings of the 20th International Conference on Machine Learning (ICML-03), pp 186–193Google Scholar
  13. 13.
    Frankl P, Maehara H (1988) The Johnson–Lindenstrauss lemma and the sphericity of some graphs. J Combin Theory Ser B 44(3):355–362MathSciNetCrossRefMATHGoogle Scholar
  14. 14.
    Freksen CB, Larsen KG On using Toeplitz and Circulant matrices for Johnson–Lindenstrauss transforms. arXiv:1706.10110 [math.FA]
  15. 15.
    Gyllensten AC, Sahlgren M (2015) Navigating the semantic horizon using relative neighborhood graphs. CoRR, arXiv:abs/1501.02670
  16. 16.
    Hastie T, Tibshirani R, Friedman J (2009) The elements of statistical learning. Springer Science+Business Media, New YorkCrossRefMATHGoogle Scholar
  17. 17.
    Indyk P, Motwani R (1998) Approximate nearest neighbors: towards removing the curse of dimensionality. In: 30th Annual ACM Symposium on Theory of Computing, Dallas, TX. ACM, New York, pp 604–613Google Scholar
  18. 18.
    Jaques N (2018) Fast Johnson–Lindenstrauss transform for classification of high dimensional data.http://www.cs.ubc.ca/~jaquesn/MachineLearningTheory.pdf. Accessed 14 May 2018
  19. 19.
    Kane DM, Nelson J (2014) Sparser Johnson–Lindenstrauss transforms. J ACM (JACM) 61(1):4MathSciNetCrossRefMATHGoogle Scholar
  20. 20.
    Roweis S, Saul L (2018) LLE Algorithm Pseudocode. https://cs.nyu.edu/~roweis/lle/algorithm.html. Accessed April 2018
  21. 21.
    Larsen CG, Nelson J (2017) Optimality of the Johnson–Lindenstrauss lemma. arXiv:1609.02094v2 [cs.IT]
  22. 22.
    Larsen CG, Nelson J (2014) The Johnson–Lindenstrauss lemma is optimal for linear dimensionality reduction. arXiv:1411.2404v1
  23. 23.
    Johnson WB, Lindenstrauss J (1984) Extensions of Lipschitz mappings into a Hilbert space. In: Conference in Modern Analysis and Probability, New Haven, CI, 1982. American Mathematical Society, Providence, RI, pp 189–206Google Scholar
  24. 24.
    Matoušek J (2008) On variants of the Johnson–Lindenstrauss lemma. Random Struct Algorithms 33(2):142–156MathSciNetCrossRefMATHGoogle Scholar
  25. 25.
    Pierson E, Yau C (2015) ZIFA: dimensionality reduction for zero-inflated single-cell gene expression analysis. Genome Biol 16:241CrossRefGoogle Scholar
  26. 26.
    Rojo J, Nguye TS (2010) Improving the Johnson–Lindenstrauss lemma. arXiv:1005.1440v1
  27. 27.
    Roweis ST, Saul LK (2000) Nonlinear dimensionality reduction by locally linear embedding. Science 290(5500):2323–2326CrossRefGoogle Scholar
  28. 28.
    Bellec P, Chu C, Chouinard-Decorte F, Benhajali Y, Margulies DS, Craddock RC (2017) The Neuro Bureau ADHD-200 Preprocessed repository. NeuroImage 144(Part B):275–286.  https://doi.org/10.1016/j.neuroimage.2016.06.034 CrossRefGoogle Scholar
  29. 29.
    Wang J (2011) Classical multidimensional scaling. Geometric structure of high dimensional data and dimensionality reduction. Springer, Heidelberg, pp 115–129CrossRefGoogle Scholar
  30. 30.
    You L, Knoll F, Mao Y, Gao S (2017) Practical Johnson–Lindenstrauss transforms via algebraic geometry codes. In: International Conference on Control, Artificial Intelligence, Robotics Optimization (ICCAIRO), pp 171–176Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Mathematical SciencesMacEwan UniversityEdmontonCanada
  2. 2.Department of Mathematical and Statistical SciencesUniversity of AlbertaEdmontonCanada
  3. 3.Department of Mathematics and Computer ScienceLaurentian UniversitySudburyCanada
  4. 4.Department of DentistryUniversity of AlbertaEdmontonCanada

Personalised recommendations