Advertisement

A Density-Based Backward Approach to Isolate Rare Events in Large-Scale Applications

  • Enikö Székely
  • Pascal Poncelet
  • Florent Masseglia
  • Maguelonne Teisseire
  • Renaud Cezar
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8140)

Abstract

While significant work in data mining has been dedicated to the detection of single outliers in the data, less research has approached the problem of isolating a group of outliers, i.e. rare events representing micro-clusters of less – or significantly less – than 1% of the whole dataset. This research issue is critical for example in medical applications. The problem is difficult to handle as it lies at the frontier between outlier detection and clustering and distinguishes by a clear challenge to avoid missing true positives. We address this challenge and propose a novel two-stage framework, based on a backward approach, to isolate abnormal groups of events in large datasets. The key of our backward approach is to first identify the core of the dense regions and then gradually augments them based on a density-driven condition. The framework outputs a small subset of the dataset containing both rare events and outliers. We tested our framework on a biomedical application to find micro-clusters of pathological cells. The comparison against two common clustering (DBSCAN) and outlier detection (LOF) algorithms show that our approach is a very efficient alternative to the detection of rare events – generally a recall of 100% and a higher precision, positively correlated wih the size of the rare event – while also providing a \(\mathcal{O}(N)\) solution to the existing algorithms dominated by a \(\mathcal{O}(N^2)\) complexity.

Keywords

rare events outlier/anomaly detection large scale k-means 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Aggarwal, C.: Outlier analysis. Springer (2013)Google Scholar
  2. 2.
    Bae, D.-H., Jeong, S., Kim, S.-W., Lee, M.: Outlier detection using centrality and center-proximity. In: Proceedings of CIKM (2012)Google Scholar
  3. 3.
    Breunig, M., Kriegel, H.-P., Ng, R.T., Sander, J.: Lof: identifying density-based local outliers. In: Proceedings of ACM SIGMOD (2000)Google Scholar
  4. 4.
    Chandola, V., Banerjee, A., Kumar, V.: Anomaly detection: a survey. ACM Computing Surveys 41 (2009)Google Scholar
  5. 5.
    Chawla, N.V., Japkowich, N., Kolcz, A.: Editorial: special issue on learning from imbalanced data. SIGKDD Explorations 6 (2004)Google Scholar
  6. 6.
    Dai, H., Zhu, F., Lim, E.-P., Pang, H.: Mining coherent anomaly collections on web data. In: Proceedings of CIKM (2012)Google Scholar
  7. 7.
    Ertoz, L., Steinbach, M., Kumar, V.: Finding clusters of different sizes, shapes and densities in noisy, high-dimensional data. In: Proceedings of SDM (2003)Google Scholar
  8. 8.
    Ester, M., Kriegel, H.-P., Sander, J., Xu, X.: A density-based algorithm for discovering clusters in large spatial databases with noise. In: Proceedings of ACM SIGKDD (1996)Google Scholar
  9. 9.
    Hawkins, D.: Identification of outliers. Chapman and Hall (1980)Google Scholar
  10. 10.
    He, Z., Xu, X., Deng, S.: Discovering cluster-based local outliers. Pattern Recognition Letters 24 (2003)Google Scholar
  11. 11.
    Levina, E., Bickel, P.J.: Maximum likelihood estimation of intrinsic dimension. In: Advances in Neural Information Processing Systems, vol. 17 (2005)Google Scholar
  12. 12.
    Liu, F.T., Ting, K.M., Zhou, Z.-H.: Isolation forest. In: Proceedings of ICDM (2008)Google Scholar
  13. 13.
    Liu, F.T., Ting, K.M., Zhou, Z.-H.: On detecting clustered anomalies using SCiForest. In: Balcázar, J.L., Bonchi, F., Gionis, A., Sebag, M. (eds.) ECML PKDD 2010, Part II. LNCS (LNAI), vol. 6322, pp. 274–290. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  14. 14.
    Liu, F.T., Ting, K.M., Zhou, Z.-H.: Isolation-based anomaly detection. ACM Transactions on Knowledge Discovery from Data 6 (2012)Google Scholar
  15. 15.
    von Luxburg, U.: A tutorial on spectral clustering. Statistics and Computing 17 (2007)Google Scholar
  16. 16.
    MacQueen, J.B.: Some methods for classification and analysis of multivariate observations. In: Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability (1967)Google Scholar
  17. 17.
    Papadimitriou, S., Kitagawa, H., Gribbons, P.B., Faloutsos, C.: Loci: Fast outlier detection using the local correlation integral. In: Proceedings of ICDE (2003)Google Scholar
  18. 18.
    Rocke, D.M., Woodruff, D.L.: Identification of outliers in multivariate data. Journal of the American Statistical Association (1996)Google Scholar
  19. 19.
    Shmueli, G., Burkom, H.: Statistical challenges facing early outbreak detection in biosurveillance. Technometrics, Special Issue on Anomaly Detection (2010)Google Scholar
  20. 20.
    Tang, Y., Zhang, Y.-Q., Chawla, N.V., Krasser, S.: Svms for highly imbalanced classification. IEEE Transactions on Systems, Man and Cybernetics 39 (2009)Google Scholar
  21. 21.
    Xiong, H., Wu, J., Chen, J.: K-means clustering versus validation measures: a data distribution perspective. In: Proceedings of SIGKDD (2006)Google Scholar
  22. 22.
    Zhu, S., Wang, D., Li, T.: Data clustering with size constraints. Knowledge-Based Systems 23 (2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Enikö Székely
    • 1
  • Pascal Poncelet
    • 1
  • Florent Masseglia
    • 2
  • Maguelonne Teisseire
    • 3
  • Renaud Cezar
    • 4
  1. 1.Computer Science DepartmentUniversity of MontpellierMontpellierFrance
  2. 2.INRIAMontpellierFrance
  3. 3.Maison de la TélédetectionMontpellierFrance
  4. 4.Institute for Research in BiotherapyUniversity HospitalMontpellierFrance

Personalised recommendations