Fast and Scalable Outlier Detection with Approximate Nearest Neighbor Ensembles

Conference paper

DOI: 10.1007/978-3-319-18123-3_2

Part of the Lecture Notes in Computer Science book series (LNCS, volume 9050)
Cite this paper as:
Schubert E., Zimek A., Kriegel HP. (2015) Fast and Scalable Outlier Detection with Approximate Nearest Neighbor Ensembles. In: Renz M., Shahabi C., Zhou X., Cheema M. (eds) Database Systems for Advanced Applications. DASFAA 2015. Lecture Notes in Computer Science, vol 9050. Springer, Cham

Abstract

Popular outlier detection methods require the pairwise comparison of objects to compute the nearest neighbors. This inherently quadratic problem is not scalable to large data sets, making multidimensional outlier detection for big data still an open challenge. Existing approximate neighbor search methods are designed to preserve distances as well as possible. In this article, we present a highly scalable approach to compute the nearest neighbors of objects that instead focuses on preserving neighborhoods well using an ensemble of space-filling curves. We show that the method has near-linear complexity, can be distributed to clusters for computation, and preserves neighborhoods—but not distances—better than established methods such as locality sensitive hashing and projection indexed nearest neighbors. Furthermore, we demonstrate that, by preserving neighborhoods, the quality of outlier detection based on local density estimates is not only well retained but sometimes even improved, an effect that can be explained by relating our method to outlier detection ensembles. At the same time, the outlier detection process is accelerated by two orders of magnitude.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Erich Schubert
    • 1
  • Arthur Zimek
    • 1
  • Hans-Peter Kriegel
    • 1
  1. 1.Ludwig-Maximilians-Universität MünchenMünchenGermany

Personalised recommendations