Towards Enabling Outlier Detection in Large, High Dimensional Data Warehouses

  • Konstantinos Georgoulas
  • Yannis Kotidis
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7338)

Abstract

In this work we present a novel framework that permits us to detect outliers in a data warehouse. We extend the commonly used definition of distance-based outliers in order to cope with the large data domains that are typical in dimensional modeling of OLAP datasets. Our techniques utilize a two-level indexing scheme. The first level is based on Locality Sensitivity Hashing (LSH) and allows us to replace range searching, which is very inefficient in high dimensional spaces, with approximate nearest neighbor computations in an intuitive manner. The second level utilizes the Piece-wise Aggregate Approximation (PAA) technique, which substantially reduces the space required for storing the data representations. As will be explained, our method permits incremental updates on the data representation used, which is essential for managing voluminous datasets common in data warehousing applications.

Keywords

Data Item Data Warehouse Range Query Data Cube Indexing Scheme 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Kimball, R.: The Data Warehouse Toolkit. John Wiley & Sons (1996)Google Scholar
  2. 2.
    Subramaniam, S., Palpanas, T., Papadopoulos, D., Kalogeraki, V., Gunopulos, D.: Online Outlier Detection in Sensor Data Using Non-Parametric Models. In: VLDB (2006)Google Scholar
  3. 3.
    Korn, F., Pagel, B.U., Faloutsos, C.: On the ’Dimensionality Curse’ and the ’Self-Similarity Blessing’. IEEE Trans. Knowl. Data Eng. 13(1), 96–111 (2001)CrossRefGoogle Scholar
  4. 4.
    Andoni, A., Indyk, P.: Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions. In: FOCS, pp. 459–468. IEEE Computer Society (2006)Google Scholar
  5. 5.
    Lv, Q., Josephson, W., Wang, Z., Charikar, M., Li, K.: Multi-Probe LSH: Efficient Indexing for High-Dimensional Similarity Search. In: VLDB, pp. 950–961 (2007)Google Scholar
  6. 6.
    Georgoulas, K., Kotidis, Y.: Distributed Similarity Estimation using Derived Dimensions. VLDB J. 21(1), 25–50 (2012)CrossRefGoogle Scholar
  7. 7.
    Keogh, E.J., Chakrabarti, K., Pazzani, M.J., Mehrotra, S.: Dimensionality Reduction for Fast Similarity Search in Large Time Series Databases. Knowl. Inf. Syst. 3(3), 263–286 (2001)MATHCrossRefGoogle Scholar
  8. 8.
    Roussopoulos, N., Kotidis, Y., Roussopoulos, M.: Cubetree: Organization of and Bulk Updates on the Data Cube. In: SIGMOD Conference, pp. 89–99 (1997)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Konstantinos Georgoulas
    • 1
  • Yannis Kotidis
    • 1
  1. 1.Athens University of Economics and BusinessAthensGreece

Personalised recommendations