Advertisement

A Density Estimation Approach for Detecting and Explaining Exceptional Values in Categorical Data

  • Fabrizio Angiulli
  • Fabio Fassetti
  • Luigi Palopoli
  • Cristina SerraoEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11828)

Abstract

In this work we deal with the problem of detecting and explaining exceptional behaving values in categorical datasets. As a first main contribution we provide the notion of frequency occurrence which can be thought as a form of Kernel Density Estimation applied to the domain of frequency values. As a second contribution, we define an outlierness measure for categorical values that, leveraging the cdf of the density described above, decides if the frequency of a certain value is rare if compared to the frequencies associated with the other values. This measure is able to simultaneously identify two kinds of anomalies called lower outliers and upper outliers, namely exceptionally low or high frequent values. The experiments highlight that the method is scalable and able to identify anomalies of different nature from traditional techniques.

Keywords

Outliers Categorical attributes Outlier explanation 

References

  1. 1.
    Angiulli, F., Fassetti, F., Manco, G., Palopoli, L.: Outlying property detection with numerical attributes. Data Min. Knowl. Discov. 31(1), 134–163 (2017)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Angiulli, F., Fassetti, F., Palopoli, L.: Detecting outlying properties of exceptional objects. ACM Trans. Database Syst. (TODS) 34(1), 7 (2009)CrossRefGoogle Scholar
  3. 3.
    Angiulli, F., Fassetti, F., Palopoli, L.: Discovering characterizations of the behavior of anomalous subpopulations. IEEE TKDE 25(6), 1280–1292 (2013)Google Scholar
  4. 4.
    Angiulli, F., Pizzuti, C.: Outlier mining in large high-dimensional data sets. IEEE Trans. Knowl. Data Eng. 17(2), 203–215 (2005)CrossRefGoogle Scholar
  5. 5.
    Breunig, M.M., Kriegel, H.P., Ng, R.T., Sander, J.: LOF: identifying density-based local outliers. In: ACM Sigmod Record, vol. 29, pp. 93–104. ACM (2000)Google Scholar
  6. 6.
    Chandola, V., Banerjee, A., Kumar, V.: Anomaly detection: a survey. ACM Comput. Surv. (CSUR) 41(3), 15 (2009)CrossRefGoogle Scholar
  7. 7.
    Chandola, V., Boriah, S., Kumar, V.: A framework for exploring categorical data. In: SIAM International Conference on Data Mining (SDM), pp. 187–198 (2009)Google Scholar
  8. 8.
    Dang, X.H., Assent, I., Ng, R.T., Zimek, A., Schubert, E.: Discriminative features for identifying and interpreting outliers. In: IEEE ICDE, pp. 88–99 (2014)Google Scholar
  9. 9.
    Das, K., Schneider, J.: Detecting anomalous records in categorical datasets. In: ACM International Conference on Knowledge Discovery and Data Mining (KDD), pp. 220–229 (2007)Google Scholar
  10. 10.
    He, Z., Deng, S., Xu, X.: An optimization model for outlier detection in categorical data. In: Huang, D.-S., Zhang, X.-P., Huang, G.-B. (eds.) ICIC 2005. LNCS, vol. 3644, pp. 400–409. Springer, Heidelberg (2005).  https://doi.org/10.1007/11538059_42CrossRefGoogle Scholar
  11. 11.
    Knorr, E.M., Ng, R.T.: Finding intensional knowledge of distance-based outliers. In: International Conference on Very Large Data Bases, VLDB, pp. 211–222 (1999)Google Scholar
  12. 12.
    Koufakou, A., Secretan, J., Georgiopoulos, M.: Non-derivable itemsets for fast outlier detection in large high-dimensional categorical data. Knowl. Inf. Syst. 29(3), 697–725 (2011)CrossRefGoogle Scholar
  13. 13.
    Micenkovà, B., Ng, R.T., Dang, X., Assent, I.: Explaining outliers by subspace separability. In: IEEE International Conference on Data Mining, pp. 518–527 (2013)Google Scholar
  14. 14.
    Pang, G., Cao, L., Chen, L.: Outlier detection in complex categorical data by modelling the feature value couplings. In: IJCAI, pp. 1902–1908 (2016)Google Scholar
  15. 15.
    Suri, N.R., Murty, M.N., Athithan, G.: An algorithm for mining outliers in categorical data through ranking. In: IEEE International Conference on Hybrid Intelligent Systems (HIS), pp. 247–252 (2012)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Fabrizio Angiulli
    • 1
  • Fabio Fassetti
    • 1
  • Luigi Palopoli
    • 1
  • Cristina Serrao
    • 1
    Email author
  1. 1.DIMESUniversity of CalabriaRendeItaly

Personalised recommendations