When Is “Nearest Neighbor” Meaningful?

  • Kevin Beyer
  • Jonathan Goldstein
  • Raghu Ramakrishnan
  • Uri Shaft
Conference paper

DOI: 10.1007/3-540-49257-7_15

Volume 1540 of the book series Lecture Notes in Computer Science (LNCS)
Cite this paper as:
Beyer K., Goldstein J., Ramakrishnan R., Shaft U. (1999) When Is “Nearest Neighbor” Meaningful?. In: Beeri C., Buneman P. (eds) Database Theory — ICDT’99. ICDT 1999. Lecture Notes in Computer Science, vol 1540. Springer, Berlin, Heidelberg

Abstract

We explore the effect of dimensionality on the “nearest neighbor” problem. We show that under a broad set of conditions (much broader than independent and identically distributed dimensions), as dimensionality increases, the distance to the nearest data point approaches the distance to the farthest data point. To provide a practical perspective, we present empirical results on both real and synthetic data sets that demonstrate that this effect can occur for as few as 10–15 dimensions.

These results should not be interpreted to mean that high-dimensional indexing is never meaningful; we illustrate this point by identifying some high-dimensional workloads for which this effect does not occur. However, our results do emphasize that the methodology used almost universally in the database literature to evaluate high-dimensional indexing techniques is flawed, and should be modified. In particular, most such techniques proposed in the literature are not evaluated versus simple linear scan, and are evaluated over workloads for which nearest neighbor is not meaningful. Often, even the reported experiments, when analyzed carefully, show that linear scan would outperform the techniques being proposed on the workloads studied in high (10–15) dimensionality!

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Copyright information

© Springer-Verlag Berlin Heidelberg 1999

Authors and Affiliations

  • Kevin Beyer
    • 1
  • Jonathan Goldstein
    • 1
  • Raghu Ramakrishnan
    • 1
  • Uri Shaft
    • 1
  1. 1.CS Dept.University of Wisconsin-MadisonMadison