Skip to main content
SpringerLink
Log in

The Role of Local Intrinsic Dimensionality in Benchmarking Nearest Neighbor Search

  • Conference paper
  • First Online:
Similarity Search and Applications (SISAP 2019)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 11807))

Included in the following conference series:

Abstract

This paper reconsiders common benchmarking approaches to nearest neighbor search. It is shown that the concept of local intrinsic dimensionality (LID) allows to choose query sets of a wide range of difficulty for real-world datasets. Moreover, the effect of different LID distributions on the running time performance of implementations is empirically studied. To this end, different visualization concepts are introduced that allow to get a more fine-grained overview of the inner workings of nearest neighbor search principles. The paper closes with remarks about the diversity of datasets commonly used for nearest neighbor search benchmarking. It is shown that such real-world datasets are not diverse: results on a single dataset predict results on all other datasets well.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 74.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    We thank the authors of the implementations for their help and responsiveness in adding this feature to their library.

  2. 2.

    https://github.com/facebookresearch/faiss.

  3. 3.

    We note that IVF counts the initial comparisons to find the closest centroids as distance computations, whereas Annoy did not count the inner product computations during tree traversal.

  4. 4.

    In order not to clutter the plots, we fixed parameters as follows: IVF | number of lists 8192; Annoy | number of trees 100; HNSW | efConstruction 500, M 8; ONNG | edge 100, outdegree 10, indegree 120.

References

  1. Alman, J., Williams, R.: Probabilistic polynomials and hamming nearest neighbors. In: FOCS 2015, pp. 136–150 (2015)

    Google Scholar 

  2. Amsaleg, L., et al.: Estimating local intrinsic dimensionality. In: KDD 2015, pp. 29–38. ACM (2015)

    Google Scholar 

  3. Amsaleg, L., Chelly, O., Houle, M.E., Kawarabayashi, K.I., Radovanović, M., Treeratanajaru, W.: Intrinsic dimensionality estimation within tight localities. In: Proceedings of the 2019 SIAM International Conference on Data Mining, pp. 181–189. SIAM (2019)

    Chapter  Google Scholar 

  4. Aumüller, M., Bernhardsson, E., Faithfull, A.: ANN-benchmarks: a benchmarking tool for approximate nearest neighbor algorithms. In: Beecks, C., Borutta, F., Kröger, P., Seidl, T. (eds.) SISAP 2017. LNCS, vol. 10609, pp. 34–49. Springer, Heidelberg (2017). https://doi.org/10.1007/978-3-319-68474-1_3

    Chapter  Google Scholar 

  5. Aumüller, M., Ceccarello, M.: Benchmarking nearest neighbor search: influence of local intrinsic dimensionality and result diversity in real-world datasets. In: 1st Workshop on Evaluation and Experimental Design in Data Mining and Machine Learning (EDML 2019) (2019). https://imada.sdu.dk/Research/EDML/

  6. Bernhardsson, E.: Annoy. https://github.com/spotify/annoy

  7. Casanova, G., et al.: Dimensional testing for reverse k-nearest neighbor search. PVLDB 10(7), 769–780 (2017)

    Google Scholar 

  8. Chávez, E., Navarro, G., Baeza-Yates, R., Marroquín, J.L.: Searching in metric spaces. ACM Comput. Surv. 33(3), 273–321 (2001). https://doi.org/10.1145/502807.502808

    Article  Google Scholar 

  9. Curtin, R.R., et al.: MLPACK: a scalable C++ machine learning library. J. Mach. Learn. Res. 14, 801–805 (2013)

    MathSciNet  MATH  Google Scholar 

  10. Edel, M., Soni, A., Curtin, R.R.: An automatic benchmarking system. In: NIPS 2014 Workshop on Software Engineering for Machine Learning (2014)

    Google Scholar 

  11. Houle, M.E.: Dimensionality, discriminability, density and distance distributions. In: Data Mining Workshops (ICDMW), pp. 468–473. IEEE (2013)

    Google Scholar 

  12. Houle, M.E., Schubert, E., Zimek, A.: On the correlation between local intrinsic dimensionality and outlierness. In: Marchand-Maillet, S., Silva, Y.N., Chávez, E. (eds.) SISAP 2018. LNCS, vol. 11223, pp. 177–191. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-02224-2_14

    Chapter  Google Scholar 

  13. Iwasaki, M., Miyazaki, D.: Optimization of Indexing Based on k-Nearest Neighbor Graph for Proximity Search in High-dimensional Data. ArXiv e-prints, October 2018

    Google Scholar 

  14. Jégou, H., Douze, M., Schmid, C.: Product quantization for nearest neighbor search. IEEE Trans. Pattern Anal. Mach. Intell. 33(1), 117–128 (2011). https://doi.org/10.1109/TPAMI.2010.57

    Article  Google Scholar 

  15. Johnson, J., Douze, M., Jégou, H.: Billion-scale similarity search with GPUs. CoRR abs/1702.08734 (2017)

    Google Scholar 

  16. Johnson, W.B., Lindenstrauss, J., Schechtman, G.: Extensions of Lipschitz maps into Banach spaces. Israel J. Math. 54(2), 129–138 (1986)

    Article  MathSciNet  Google Scholar 

  17. Jolliffe, I.: Principal Component Analysis. Springer, Berlin (2011)

    MATH  Google Scholar 

  18. Kriegel, H., Schubert, E., Zimek, A.: The (black) art of runtime evaluation: are we comparing algorithms or implementations? Knowl. Inf. Syst. 52(2), 341–378 (2017)

    Article  Google Scholar 

  19. Levina, E., Bickel, P.J.: Maximum likelihood estimation of intrinsic dimension. In: NIPS, pp. 777–784 (2005)

    Google Scholar 

  20. Li, W., Zhang, Y., Sun, Y., Wang, W., Zhang, W., Lin, X.: Approximate nearest neighbor search on high dimensional data - experiments, analyses, and improvement (v1.0). CoRR abs/1610.02455 (2016)

    Google Scholar 

  21. Malkov, Y.A., Yashunin, D.A.: Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs. ArXiv e-prints, March 2016

    Google Scholar 

  22. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. CoRR abs/1301.3781 (2013)

    Google Scholar 

  23. Pennington, J., Socher, R., Manning, C.D.: Glove: global vectors for word representation. In: Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014)

    Google Scholar 

  24. Smith-Miles, K., Baatar, D., Wreford, B., Lewis, R.: Towards objective measures of algorithm performance across instance space. Comput. Oper. Res. 45, 12–24 (2014)

    Article  MathSciNet  Google Scholar 

  25. Spring, R., Shrivastava, A.: Scalable and sustainable deep learning via randomized hashing. In: KDD 2017, pp. 445–454 (2017). https://doi.org/10.1145/3097983.3098035

  26. Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. CoRR abs/1708.07747 (2017)

    Google Scholar 

Download references

Acknowledgements

The authors would like to thank the anonymous reviewers for their useful suggestions, which helped to improve the presentation of the paper. The research leading to these results has received funding from the European Research Council under the European Union’s 7th Framework Programme (FP7/2007-2013)/ERC grant agreement no. 614331.

Author information

Authors and Affiliations

  1. IT University of Copenhagen, Copenhagen, Denmark

    Martin Aumüller & Matteo Ceccarello

Authors
  1. Martin Aumüller
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Matteo Ceccarello
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Martin Aumüller .

Editor information

Editors and Affiliations

  1. ISTI-CNR, Pisa, Italy

    Giuseppe Amato

  2. ISTI-CNR, Pisa, Italy

    Claudio Gennaro

  3. New Jersey Institute of Technology, Newark, NJ, USA

    Vincent Oria

  4. University of Novi Sad, Novi Sad, Serbia

    Miloš Radovanović

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Aumüller, M., Ceccarello, M. (2019). The Role of Local Intrinsic Dimensionality in Benchmarking Nearest Neighbor Search. In: Amato, G., Gennaro, C., Oria, V., Radovanović , M. (eds) Similarity Search and Applications. SISAP 2019. Lecture Notes in Computer Science(), vol 11807. Springer, Cham. https://doi.org/10.1007/978-3-030-32047-8_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-32047-8_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-32046-1

  • Online ISBN: 978-3-030-32047-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation