Advertisement

Density Biased Sampling with Locality Sensitive Hashing for Outlier Detection

  • Xuyun ZhangEmail author
  • Mahsa Salehi
  • Christopher Leckie
  • Yun Luo
  • Qiang He
  • Rui Zhou
  • Rao Kotagiri
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11234)

Abstract

Outlier or anomaly detection is one of the major challenges in big data analytics since unusual but insightful patterns are often hidden in massive data sets such as sensing data and social networks. Sampling techniques have been a focus for outlier detection to address scalability on big data. The recent study has shown uniform random sampling with ensemble can boost outlier detection performance. However, uniform sampling assumes that all points are of equal importance, which usually fails to hold for outlier detection because some points are more sensitive to sampling than others. Thus, it is necessary and promising to utilise the density information of points to reflect their importance for sampling based detection. In this paper, we formally investigate density biased sampling for outlier detection, and propose a novel density biased sampling approach. To attain scalable density estimation, we use Locality Sensitive Hashing (LSH) for counting the nearest neighbours of a point. Extensive experiments on both synthetic and real-world data sets show that our approach significantly outperforms existing outlier detection methods based on uniform sampling.

Keywords

Outlier/anomaly detection Locality-Sensitive Hashing Density biased sampling Big data Unsupervised learning 

Notes

Acknowledgments

This work was supported in part by the New Zealand Marsden Fund under Grant No. 17-UOA-248, the UoA FRDF under Grant No. 3714668, and the NJU Overseas Open fund under Grant No. KFKT2018A12.

References

  1. 1.
    Aggarwal, C.C.: Outlier ensembles: position paper. ACM SIGKDD Explor. Newsl. 14(2), 49–58 (2013)CrossRefGoogle Scholar
  2. 2.
    Aggarwal, C.C., Sathe, S.: Theoretical foundations and algorithms for outlier ensembles. ACM SIGKDD Explor. Newsl. 17(1), 24–47 (2015)CrossRefGoogle Scholar
  3. 3.
    Andoni, A., Indyk, P.: Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions. In: FOCS, pp. 459–468 (2006)Google Scholar
  4. 4.
    Angiulli, F., Pizzuti, C.: Fast outlier detection in high dimensional spaces. In: Elomaa, T., Mannila, H., Toivonen, H. (eds.) PKDD 2002. LNCS, vol. 2431, pp. 15–27. Springer, Heidelberg (2002).  https://doi.org/10.1007/3-540-45681-3_2CrossRefGoogle Scholar
  5. 5.
    Breunig, M.M., Kriegel, H.P., Ng, R.T., Sander, J.: LOF: identifying density-based local outliers. ACM SIGMOD Rec 29(2), 93–104 (2000)CrossRefGoogle Scholar
  6. 6.
    Chandola, V., Banerjee, A., Kumar, V.: Anomaly detection: a survey. ACM Comput. Surv. (CSUR) 41(3), 15 (2009)CrossRefGoogle Scholar
  7. 7.
    Dong, W., Wang, Z., Josephson, W., Charikar, M., Li, K.: Modeling LSH for performance tuning. In: CIKM, pp. 669–678 (2008)Google Scholar
  8. 8.
    Fawcett, T.: An introduction to ROC analysis. Pattern Recogn. Lett. 27(8), 861–874 (2006)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Fu, P., Hu, X.: Biased-sampling of density-based local outlier detection algorithm. In: ICNC-FSKD, pp. 1246–1253 (2016)Google Scholar
  10. 10.
    Indyk, P., Motwani, R.: Approximate nearest neighbors: towards removing the curse of dimensionality. In: STOC, pp. 604–613 (1998)Google Scholar
  11. 11.
    Jones, M.: Kumaraswamy’s distribution: a beta-type distribution with some tractability advantages. Stat. Methodol. 6(1), 70–81 (2009)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Knox, E.M., Ng, R.T.: Algorithms for mining distance-based outliers in large datasets. In: VLDB, pp. 392–403 (1998)Google Scholar
  13. 13.
    Kollios, G., Gunopulos, D., Koudas, N., Berchtold, S.: Efficient biased sampling for approximate clustering and outlier detection in large data sets. IEEE Trans. Knowl. Data Eng. 15(5), 1170–1187 (2003)CrossRefGoogle Scholar
  14. 14.
    Kriegel, H.P., Zimek, A., et al.: Angle-based outlier detection in high-dimensional data. In: ACM SIGKDD, pp. 444–452 (2008)Google Scholar
  15. 15.
    Liu, F.T., Ting, K.M., Zhou, Z.-H.: On detecting clustered anomalies using SCiForest. In: Balcázar, J.L., Bonchi, F., Gionis, A., Sebag, M. (eds.) ECML PKDD 2010. LNCS (LNAI), vol. 6322, pp. 274–290. Springer, Heidelberg (2010).  https://doi.org/10.1007/978-3-642-15883-4_18CrossRefGoogle Scholar
  16. 16.
    Liu, F.T., Ting, K.M., Zhou, Z.H.: Isolation-based anomaly detection. ACM Trans. Knowl. Discov. Data 6(1), 3 (2012)CrossRefGoogle Scholar
  17. 17.
    Luo, C., Shrivastava, A.: Arrays of (locality-sensitive) count estimators (ACE): anomaly detection on the edge. In: WWW, pp. 1439–1448 (2018)Google Scholar
  18. 18.
    Nanopoulos, A., Manolopoulos, Y., Theodoridis, Y.: An efficient and effective algorithm for density biased sampling. In: CIKM, pp. 398–404 (2002)Google Scholar
  19. 19.
    Pang, G., Cao, L., Chen, L., Lian, D., Liu, H.: Sparse modeling-based sequential ensemble learning for effective outlier detection in high-dimensional numeric data. In: AAAI (2018)Google Scholar
  20. 20.
    Pillutla, M.R., Raval, N., Bansal, P., Srinathan, K., Jawahar, C.: LSH based outlier detection and its application in distributed setting. In: CIKM, pp. 2289–2292 (2011)Google Scholar
  21. 21.
    Rayana, S., Zhong, W., Akoglu, L.: Sequential ensemble learning for outlier detection: a bias-variance perspective. In: ICDM, pp. 1167–1172 (2016)Google Scholar
  22. 22.
    Schubert, E.: Generalized and efficient outlier detection for spatial, temporal, and high-dimensional data mining. Ph.D. thesis (2013)Google Scholar
  23. 23.
    Sugiyama, M., Borgwardt, K.: Rapid distance-based outlier detection via sampling. In: NIPS, pp. 467–475 (2013)Google Scholar
  24. 24.
    Wang, Y., Parthasarathy, S., Tatikonda, S.: Locality sensitive outlier detection: a ranking driven approach. In: ICDE, pp. 410–421 (2011)Google Scholar
  25. 25.
    Wu, M., Jermaine, C.: Outlier detection by sampling with accuracy guarantees. In: ACM SIGKDD, pp. 767–772 (2006)Google Scholar
  26. 26.
    Yang, X., Latecki, L.J., Pokrajac, D.: Outlier detection with globally optimal exemplar-based GMM. In: SDM, pp. 145–154 (2009)CrossRefGoogle Scholar
  27. 27.
    Zhang, X., et al.: LSHiForest: a generic framework for fast tree isolation based ensemble anomaly analysis. In: ICDE, pp. 983–994 (2017)Google Scholar
  28. 28.
    Zimek, A., Campello, R.J., Sander, J.: Ensembles for unsupervised outlier detection: challenges and research questions a position paper. ACM SIGKDD Explor. Newsl. 15(1), 11–22 (2014)CrossRefGoogle Scholar
  29. 29.
    Zimek, A., Gaudet, M., Campello, R.J., Sander, J.: Subsampling for efficient and effective unsupervised outlier detection ensembles. In: ACM SIGKDD, pp. 428–436 (2013)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Xuyun Zhang
    • 1
    Email author
  • Mahsa Salehi
    • 2
  • Christopher Leckie
    • 3
  • Yun Luo
    • 4
  • Qiang He
    • 5
  • Rui Zhou
    • 5
  • Rao Kotagiri
    • 3
  1. 1.Department of Electrical and Computer EngineeringUniversity of AucklandAucklandNew Zealand
  2. 2.Faculty of Information TechnologyMonash UniversityMelbourneAustralia
  3. 3.Department of Computing and Information SystemsUniversity of MelbourneMelbourneAustralia
  4. 4.Faculty of Computer Science and TechnologyGuizhou UniversityGuiyangChina
  5. 5.School of Software and Electrical EngineeringSwinburne University of TechnologyMelbourneAustralia

Personalised recommendations