Skip to main content
SpringerLink
Log in

A survey of anomaly detection techniques

  • Research Article
  • Published:
Journal of Optics Aims and scope Submit manuscript

Abstract

The phrase "anomaly detection" is often used to describe any technique that looks for samples that differ from expected patterns. Depending on availability of data labels, types of abnormalities and applications, many anomaly detection techniques have been developed. This study aims to give a well-organized and a thorough review of anomaly detection techniques. We think it will aid in a better understanding of the topic of anomaly detection. It also presents the different approaches introduced in the literature for anomaly detection from images as well as other patterns. Despite the common availability of categorical data in practice, anomaly detection from categorical data has received a relatively little attention as compared to that from quantitative data. We divide the anomaly detection research methodologies into distinct categories. We describe the fundamental anomaly detection techniques, as well as their modifications and importantance. In addition, we highlight the merits and demerits of each category. Finally, we discuss the research gaps and limitations encountered, when using anomaly detection techniques for categorical data to solve real-world problems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. L. Ruff, J. Kauffmann, R. Vandermeulen, G. Montavon, W. Samek, M. Kloft, T. Dietterich, K. M ̈uller, A Unifying Review of Deep and Shallow Anomaly Detection (IEEE, 2021)

  2. H. Hojjati, T. Ho, N. Armanfard, Self-Supervised Anomaly Detection: A Survey and Outlook, (IEEE, 2022)

  3. V. Hodge, J. Austin, A survey of outlier detection methodologies. Arti. Int. Rev. 22(10), 85–126 (2004)

    Article  Google Scholar 

  4. R. Feinman, R. Curtin, S. Shintre, A. Gardner, Detecting adversarial samples from artifacts. arX., 0410 (2017)

  5. K. Lee, K. Lee, H. Lee, J. Shin, A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Adv. Neur. Info. Proc. Syst. 31 (2018)

  6. V. Jumutc, J.A. Suykens, Multi-class supervised novelty detection. IEEE Trans. Pattern. Anal. Mach. Intell. 36(12), 2510–2523 (2014)

    Article  Google Scholar 

  7. S. Kim, Y. Choi, M. Lee, Deep learning with support vector data description. Neur. Comput. 165, 111–117 (2015)

    Google Scholar 

  8. T. Ehret, A. Davy, J. M. Morel, M. Delbracio, Image anomalies: a review and synthesis of detection methods. J. Math. Imaging Vis. (2018)

  9. V. Chandola, A. Banerjee, V. Kumar, Outlier detection: a survey, ACM Comp. Surv. 14(15) (2009)

  10. G. Pang, C. Shen, L. Cao, A. V. D. Hengel, Deep learning for anomaly detection: a review, ACM Comp. Surv. 54(2) ( 2021)

  11. R. Chalapathy, S. Chawla, Deep learning for anomaly detection: a survey (2019)

  12. R. Chalapathy, S. Chawla, Deep learning for anomaly detection: a survey, Res. Gate (2019)

  13. H. Peng, R. Marculescu, Multi-scale compositionality: identifying the compositional structures of social dynamics using deep learning. PLoS ONE 10(4), e0118309 (2015)

    Article  Google Scholar 

  14. X. Song, M. Wu, C. Jermaine, S. Ranka, Conditional anomaly detection. IEEE Trans. Knowl. Data Eng. 19, 631–645 (2007)

    Article  Google Scholar 

  15. S. Ramaswamy, R. Rastogi, K. Shim, Efcient algorithms for mining outliers from large datasets, ACM. SIGM. ICMD, pp. 427–438 (2000)

  16. F. Angiulli, C. Pizzuti, Fast outlier detection in high dimensional spaces. ECP. DMKD, pp. 15–26 (2002)

  17. M. Breunig, H. Kriegel, R. Ng, J. Sander, LOF: identifying density-based local outliers, ACM. SIGM. Rec., 29(93–104) (2000)

  18. H. Kriegel, P. Kroger, E. Schubert, A. Zimek, LoOP: local outlier probabilities. ACM. CIKM. 09, 1649–1652 (2009)

    Google Scholar 

  19. H. Ville, I. Karkkainen, P. Franti, Outlier detection using k-nearest neighbour graph. IEEE, ICPR. 3, 330–433 (2004)

    Google Scholar 

  20. H. Huang, K. Mehrotra, C. Mohan, Rank-based outlier detection. J. Stat. Comput. Simlut. 83, 518–531 (2013)

    Article  MathSciNet  Google Scholar 

  21. H. P. Kriegel, P. Kroger, E. Schubert, A. Zimek, Outlier detection in axis-parallel subspaces of high dimensional data, Asia Conf. AKDDM., pp. 831–838 (2009)

  22. J. Zhang, Y. Jiang, K.H. Chang, S. Zhang, J. Cai, L. Hu, A concept lattice based outlier mining method in low dimensional subspaces. Patt. Reco. Lett. 15, 1434–1439 (2009)

    Article  ADS  Google Scholar 

  23. J. Zhang, X. Yu, Y. Li, S. Zhang, Y. Xun, X. Qin, A relevant subspace based contextual outlier mining algorithm. Knowl. Based. Syst. 99(72), 1–9 (2016)

    Article  Google Scholar 

  24. J. Dutta, B. Banerjee, C. Reddy, RODS: rarity based outlier detection in a sparse coding framework. IEEE, TKDE. 28(2), 483–495 (2016)

    Google Scholar 

  25. E. Muller, I. Assent, U. Steinhausen, T. Seidl, OutRank: ranking outliers in high dimensional data. IEEE 24th ICDE., pp. 600–603, (2008)

  26. E. Muller, M. Schifer, T. Seidl, Adaptive outlierness for ¨ subspace outlier ranking, in 19th Int. Conf. Info. CIKM, vol. 10, pp. 1629–1632 (2010)

  27. A. Lazarevic, V. Kumar, Feature bagging for outlier detection, KDD, in 11th ACM. SIGK, pp. 157–166 (2005)

  28. F. Keller, E. Muller, K. ohm, HiCS: High contrast ¨ subspaces for density-based outlier ranking, IEEE 28th, ICDE, pp. 1037–1048 (2012)

  29. B. Stein, M. Leeuwen, T. Back, Local subspacebased outlier detection using global neighbourhoods, in 4th IEEE, ICBD, pp. 1136–1142 (2016)

  30. A. Zimek, M. Gaudet, R. Campello, J. Sander, Subsampling for efcient and efective unsupervised outlier detection ensembles, in 19th ACM, KDD, pp. 428–436 (2013)

  31. J. Pasillas-Diaz, S. Ratte, Bagged subspaces for unsupervised outlier detection. IJCI. 33(3), 507–523 (2017)

    MathSciNet  Google Scholar 

  32. A. Ghoting, M. Otey, S. Parthasarathy, “LOADED: Linkbased outlier and anomaly detection in evolving data sets, Fourth IEEE, ICDM, pp. 387–390 (2004)

  33. A. Koufakou, M. Georgiopoulos, A fast outlier detection strategy for distributed high-dimensional data sets with mixed attributes. Data Mining Knowl Discov 20(2), 259–289 (2010)

    Article  MathSciNet  Google Scholar 

  34. K. Zhang, H. Jin, An efective pattern based outlier detection approach for mixed attribute data. AI, LNCS. 6464, 122–131 (2010)

    Google Scholar 

  35. Y. Lu, F. Chen, Y. Wang, C. Lu, Discovering anomalies on mixed-type data using a generalized student-t based approach. Exp. Syst. Appl. 28(10), 1–10 (2016)

    Google Scholar 

  36. K. Do, T. Tran, D. Phung, S. Venkatesh, “Outlier detection on mixed-type data: an energy-based approach”, ADMA., SIP.,111–125, (2016)

  37. T. Ehret, A. Davy, J. Morel, M. Delbracio, " Image Anomalies: a Review and Synthesis of Detection Methods", Math. Img. and Vis., (2018)

  38. L. Manevitz, M. Yousef, " One-Class SVMs for Document Classification", Jour. of Mach. Lear. Res., 139–154, (2001)

  39. V. Roth, " Outlier Detection with One-class Kernel Fisher Discriminants", CANIPS., 17, (2004)

  40. V. chandola, A. Banerjee, V. kumar," Anomaly Detection: A Survey", ACM Comp. Sur., 1–72, (2009)

  41. P. Tan, M. Steinbach, V. Kumar, “Introduction to Data Mining”, Add., Wesl., (2005)

  42. A. Jain, R. Dubes, Algorithms for Clustering Data (Hall Inc, Pren., 1988)

    Google Scholar 

  43. S. Basu, M. Bilenko, R. Mooney, A Probabilistic Framework for Semi-Supervised Clustering, tenth ACM SIGKDD. ACM Press, pp. 59–68 (2004)

  44. M. Ester, H. Kriegel, J. Sander, X. Xu, A density-based algorithm for discovering clusters in large spatial databases with noise, ICKDDM, 226–231, (1996)

  45. S. Guha, R. Rastogi, K. Shim, ROCK: A robust clustering algorithm for categorical attributes, Inf. Sys., 25(5), 345–366, (2000)

  46. L. Ert ̈oz, M. Steinbach, V. Kumar, Finding topics in collections of documents: a shared nearest neighbor approach, CIR, pp. 83–104, (2003)

  47. H. Motulsky, Intuitive Biostatistics: Choosing a statistical test, Oxford University Press, Oxford (1995)

  48. P. Oberdiek, M. Rottmann, H. Gottschalk, ‘Classification uncertainty of deep neural networks based on gradient information, CoRR, 1805–08440 (2018)

  49. K. Lee, K. Lee, H. Lee, J. Shin, A simple unified framework for detecting out-of-distribution samples and adversarial attacks, Adv. Neu. Inf. Proc. Syst., 7167–7177 (2018)

  50. Y. Bahat, G. Shakhnarovich, Confidence from invariance to image transformations, arXiv 1804-00657 (2018)

  51. J. Metzen, T. Genewein, V. Fischer, B. Bischoff, “On detecting adversarial perturbations, arXiv 1702-04267 (2017)

  52. J. Ren, P. J. Liu, E. Fertig, J. Snoek, R. Poplin, M. DePristo, J. Dillon, B. Lakshminarayanan, Likelihood ratios for out-of distribution detection, arXiv 1906-02845 (2019)

  53. Q. Yu, K. Aizawa, “ Unsupervised out-of-distribution detection by maximum classifier discrepancy”, IEEE Int. Conf. Comp. Vis., 9518–9526, (2019)

  54. D. Hendrycks, K. Gimpel, A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv 1610-02136, (2016)

  55. S. Liang, Y. Li, R. Srikant, Enhancing the reliability of out-ofdistribution image detection in neural networks, Int. Conf. Learn. Repr. (2018)

  56. P. Schulam, S. Saria, Can you trust this prediction? Auditing pointwise reliability after learning. Mach. Learn. Res. 89, 1022–1031 (2019)

    Google Scholar 

  57. Y. Kantaros, T. Carpenter, S. Park, R. Ivanov, S. Jang, I. Lee, J. Weimer, Vision Guard: runtime detection of adversarial inputs to perception systems. arXiv 09792 (2020)

  58. T. Bailetti, M. Gad, A. Shah, Intrusion learning: an overview of an emergent discipline. Tech. Inn. Man. Rev. 6(2), 15–20 (2016)

    Article  Google Scholar 

  59. U. Bakar, H.Ghayvat, S. Hasanm, S. Mukhopadhyay, Activity and anomaly detection in smart home: a survey, Mukh., Nex. Gen. Sens. and Sys., Spr., pp. 191–220 (2016)

  60. Z. Bakar, R. Mohemad, A. Ahmad, M. Deris, A comparative study for outlier detection techniques in data mining, IEEE ICCIS, pp. 1–6 (2006)

  61. V. Barnett, T. Lewis, Outliers in Statistical Data, 3rd (Wiley, New York, 1994)

    Google Scholar 

  62. S. Bay, M. Schwabacher, Mining distance-based outliers in near linear time with randomization and a simple pruning rule, ACM, SIGKDD, pp. 29–38 (2003)

  63. E. Beh, Simple correspondence analysis of nominal-ordinal contingency tables. App. Math. Dec. Sci. 228, 1–17 (2008)

    MathSciNet  Google Scholar 

  64. A. Beldar, V. Wadne, The detail survey of anomaly/outlier detection methods in data mining. Int. Mult. Cur. Res. 3, 462–472 (2015)

    Google Scholar 

  65. K. Singh, S. Upadhyaya, Outlier detection: applications and techniques. Comput. Sci. Issues. 9(1), 307–323 (2012)

    Google Scholar 

  66. Ghosh, D., Vogt, A.: Outliers: an evaluation of methodologies. Join. Stat. Meet., pp. 3455–3460 (2012)

  67. X. Ding, Y. Li, A. Belatreche, L. Maguire, An experimental evaluation of novelty detection methods. Neur. Comput. 135, 313–327 (2014)

    Google Scholar 

  68. K. Malik, H. Sadawarti, G. Kalra, Comparative analysis of outlier detection techniques. Comput. Appl. 97(8), 12–21 (2014)

    Google Scholar 

  69. K. Divya, N. Kumaran, Survey on outlier detection techniques using categorical data. Int. Eng. Technol. 3, 899–904 (2016)

    Google Scholar 

  70. P. Garcia-Teodoro, J. Diaz-Verdejo, G. Maciá-Fernández, E. Vázquez, Anomaly-based network intrusion detection: Techniques, systems and challenges. Comput. Secur. 28(1), 18–28 (2009)

    Article  Google Scholar 

  71. P. Gogoi, D. Bhattacharyya, B. Borah, J. Kalita, A survey of outlier detection methods in network anomaly identification. Comput. J. 54(4), 570–588 (2011)

    Article  Google Scholar 

  72. G. Golub, v. Loan, Matrix computations, 3rd edit. (2012)

  73. T. Bailetti, M. Gad, A. Shah, Intrusion learning: an overview of an emergent discipline. Tech. Innov. Man. Rev. 6(2), 15–20 (2016)

    Article  Google Scholar 

  74. M. Ahmed, A. Mahmood, J. Hu, A survey of network anomaly detection techniques. Netw. Comput. Appl. 60, 19–31 (2016)

    Article  Google Scholar 

  75. W. Fahy, B.J. Borghetti, A. Sodemann, A survey of distance and similarity measures used within network intrusion anomaly detection. IEEE Commun. Surv. Tutor. 17(1), 70–91 (2015)

    Article  Google Scholar 

  76. Y. Zhang, N.H. Meratnia, Outlier detection techniques for wireless sensor networks: a survey. IEEE Commun. Surv. Tutor. 12(2), 159–170 (2010)

    Article  Google Scholar 

  77. M. Rassam, M.A. Maarof, M. Zainal, A survey of intrusion detection schemes in wireless sensor networks. Appl. Sci. 9(10), 1636–1652 (2012)

    Google Scholar 

  78. J. Daniel, V. Joshna, S. Manjula, A survey of various intrusion detection techniques in wireless sensor networks. Comput. Sci. Mob. Comput. 2(9), 235–246 (2013)

    Google Scholar 

  79. A. Mahapatro, A. Khilar, Fault diagnosis in wireless sensor networks: a survey. IEEE Commun. Surv. Tutor. 15(4), 2000–2026 (2013)

    Article  Google Scholar 

  80. R. Kumar, T. Kaur, Outlier detection in wsn: a survey. Adv. Res. Comput. Sci. Softw. Eng. 3(7), 609–617 (2013)

    Google Scholar 

  81. A. Abduvaliyev, A. Pathan, K. Zhou, J. Roman, R. Wong, On the vital areas of intrusion detection systems in wireless sensor networks. IEEE Commun. Surv. Tutor. 15(3), 1223–1237 (2013)

    Article  Google Scholar 

  82. M. Rassam, A. Zainal, A. Maarof, Advancements of data anomaly detection research in wireless sensor networks: a survey and open issues. Sensors 13(8), 10087–10122 (2013)

    Article  ADS  Google Scholar 

  83. A. Abukhalaf, H. Wang, J. Zhang, Outlier detection techniques for localization in wireless sensor networks: a survey. Fut. Gen. Commun. Netw. 8(6), 99–114 (2015)

    Google Scholar 

  84. C. Shannon, A mathematical theory of communication. Bell Tele. Syst. Tech. Publ. 27(3), 379–423 (1948)

    Article  MathSciNet  Google Scholar 

  85. M. Marinho, J. Granjal, J. Monteiro, A survey on security attacks and countermeasures with primary user detection in cognitive radio networks. EURASIP Inf. Secur. 1, 1–14 (2015)

    Google Scholar 

  86. O. Reilly, C. Gluhak, A. Imran, M. Rajasegarar, Anomaly detection in wireless sensor networks in a non-stationary environment. IEEE Commun. Surv. Tutor. 16(3), 1413–1432 (2014)

    Article  Google Scholar 

  87. S. Shahid, N. Naqvi, I. Qaisar, Characteristics and classification of outlier detection techniques for wireless sensor networks in harsh environments: a survey. AIR. 43(2), 193–228 (2015)

    Google Scholar 

  88. A. Sodemann, A. Ross, M. Borghetti, A review of anomaly detection in automated surveillance. IEEE Trans. Syst 42(6), 1257–1272 (2012)

    Google Scholar 

  89. S. Archana, N. Pawar, Survey on outlier pattern detection techniques for time-series data. IJSR 1(1), 1852–1856 (2014)

    Google Scholar 

  90. J. Faria, R. Gonçalves, A.G. deCarvalho, Novelty detection in data streams. AI. Rev. 45(2), 235–269 (2015)

    Google Scholar 

  91. A. Deshmukh, M. Kapse, A survey on outlier detection technique in streaming data using data clustering approach. Eng. Comput. Sci. 5(1), 15453–15456 (2016)

    Google Scholar 

  92. J. Zhang, Advancements of outlier detection: a survey. ICST Tran. Scal. Info. Sys. 13(1), 1–26 (2013)

    Google Scholar 

  93. P. Purankar, R. Patil, A survey paper on an effective analytical approaches for detecting outlier in continuous time variant data stream. Eng. Comput. Sci. 4(11), 14946–14949 (2015)

    Google Scholar 

  94. M. Joudaki, H. Rashidian, A. Minaei-Bidgoli, B. Mahmoodi, M. Geraili, B. Nasiri, M. Arab, Using data mining to detect health care fraud and abuse: a review of literature. Heal. Sci. 7(1), 194–202 (2015)

    Google Scholar 

  95. S. Cousineau, D. Chartier, Outliers detection and treatment: a review. Psyc. Res. 3(1), 58–67 (2015)

    Google Scholar 

  96. S. Cho, H. Eo, Outlier detection for mass spectrometric data, Spri. Stat. Anal. Prot., 91–102m (2016)

  97. P. Bezerra, G. Costa, B. Guedes, L. Angelov, A comparative study of autonomous learning outlier detection methods applied to fault detection, IEEE Int. Conf. on Fuz. Sys., FUZZ-IEEE, pp. 1–7 (2015)

  98. R. Phua, C. Lee, S.Smith-Miles, K. Gayler, A comprehensive survey of data mining-based fraud detection research, pp. 1–14 (2010)

  99. S. Pawar, S. Amruta, D. Tambe, A survey on outlier detection techniques for credit card fraud detection. IOSR Comput. Eng. 16(2), 44–48 (2014)

    Article  Google Scholar 

  100. A. Kathiresan, V. Vasanthi, A survey on outlier detection techniques useful for financial card fraud detection. IJIET 6(1), 226–235 (2015)

    Google Scholar 

  101. R. Ahmed, M. Mahmood, N. Islam, A survey of anomaly detection techniques in financial domain. Fut. Gen. Comput. Syst. 55, 278–288 (2016)

    Article  Google Scholar 

  102. M. West, J. Bhattacharya, Intelligent financial fraud detection: a comprehensive review. Comput. Sec. 57, 47–66 (2016)

    Article  Google Scholar 

  103. R. Dhimmar, J. Chauhan, A survey on profile-injection attacks in recommender systems using outlier analysis. Comput. Sci. Man. Stud. 2(12), 356–359 (2014)

    Google Scholar 

  104. P. Dokas, P. Ertoz, L. Kumar, V. Lazarevic, A. Srivastava, J. Tan, Data mining for network intrusion detection, NSF Work. Data Min., pp. 21–30 (2002)

  105. S. Kaur, R. Singh, A survey of data mining and social network analysis based anomaly detection techniques. Egypt. Inf. J. 39, 1–18 (2015)

    Google Scholar 

  106. E. Ajitha, P. Chandra, A survey on outliers detection in distributed data mining for big data. Basic Appl. Sci. Res. 5(2), 31–38 (2015)

    Google Scholar 

  107. S. Markou, M. Singh, Novelty detection: a review-part 2: neural network based approaches. Sig. Proc. 83, 2499–2521 (2003)

    Article  Google Scholar 

  108. S. Ankur, Y. Singh, Oulier analysis using frequent pattern mining: a review. Comput. Sci. Inf. Technol. 5(1), 47–50 (2014)

    Google Scholar 

  109. B. Said, A. Dominic, D. Samir, Outlier detection scoring measurements based on frequent pattern technique. Appl. Sci. Eng. Technol. 6(8), 1340–2134 (2013)

    Google Scholar 

  110. L. Pimentel, M. Clifton, D. Clifton, L. Tarassenko, A review of novelty detection. Sig. Proc. 99, 215–249 (2014)

    Article  Google Scholar 

  111. S. Markou, M. Singh, Novelty detection: a review-part 1: statistical approaches. Sig. Proc. 83, 2481–2497 (2003)

    Article  Google Scholar 

  112. M. Hadi, S. Imon, A. Werner, Etection of outliers. Wiley Inter. Rev Comput. Stat. 1, 57–70 (2009)

    Article  Google Scholar 

  113. K. Ge, Y. Xiong, H. Zhou, Z.-H. Ozdemir, H. Yu, J. Lee, Top-eye: top-k evolving trajectory outlier detection, ACM, CIKM, pp. 1–4 (2010)

  114. T. Dave, D. Varma, A review of various statistical methods for outlier detection. IJCSET 5(2), 137–140 (2014)

    Google Scholar 

  115. H. Cho, S. Eo, Outlier detection for mass spectrometric data. Methods Mol. Biol. 1362, 91–102 (2016)

    Article  Google Scholar 

  116. T. Rezaei, A. Kasirun, M. Rohani, V. Khodadadi, Anomaly detection in online social networks using structure-based technique, ICITST, pp. 619–622 (2013)

  117. Q. Savage, D. Zhang, X. Yu, X. Chou, P. Wang, Anomaly detection in online social networks. Soc. Netw. 39, 62–70 (2014)

    Article  Google Scholar 

  118. Y. Yu, R. Qiu, H. Wen, Z. Lin, C. Liu, A survey on social media anomaly detection, pp. 1–24, (2016)

  119. S. Sarma, S. Sarma, A survey on different graph based anomaly detection techniques. India J. Sci. Technol. 8(31), 1–7 (2015)

    Article  Google Scholar 

  120. F. Ranshous, S. Shen, S. Koutra, D. Harenberg, S. Faloutsos, C. Samatova, Anomaly detection in dynamic networks: a survey. Wiley. Inter. Rev. Comput. Stat. 7(3), 223–247 (2015)

    Article  MathSciNet  Google Scholar 

  121. D. Akoglu, L. Tong, H. Koutra, Graph based anomaly detection and description: a survey. Data Min. Knowl. Discov. 29(3), 626–688 (2015)

    Article  MathSciNet  Google Scholar 

  122. J. Gupta, M. Gao, J. Aggarwal, C. Han, Outlier detection for temporal data: a survey. IEEE Trans. Knowl. Data Eng. 26(9), 2250–2267 (2014)

    Article  Google Scholar 

  123. J. Gupta, M. Gao, J. Aggarwal, C. Han, Outlier detection for temporal data. SLDMKD 5(1), 1–129 (2014)

    Google Scholar 

  124. R. Devi, R. Amalraj, Hubness in unsupervised outlier detection techniques for high dimensional data–a survey. Comput. Appl. Tech. Res. 4(11), 797–801 (2015)

    Google Scholar 

  125. J. Fanaee-T, H. Gama, Tensor-based anomaly detection: an interdisciplinary survey. Knowl. Based. Syst. 98, 130–147 (2016)

    Article  Google Scholar 

  126. S. Aggarwal, C. Yu, Outlier detection for high dimensional data, ACM, SIGMOD, pp. 37–46, (2001)

  127. H. Zimek, A. Schubert, E. Kriegel, A survey on unsupervised outlier detection in high-dimensional numerical data. Stat. Anal. Data. Min. 5(5), 363–387 (2012)

    Article  MathSciNet  Google Scholar 

  128. M. Supriya, G. Shinde, Outliers detection using subspace method: a survey. Comput. Appl. 112(16), 20–22 (2015)

    Google Scholar 

  129. A. Otey, E. Parthasarathy, S. Ghoting, An empirical comparison of outlier detection algorithms, ACM, SIGKDD, pp. 1–8 (2005)

  130. E. Campos, O. Zimek, A. Sander, J. Campello, J. Micenková, B. Schubert, E. Assent, I. Houle, On the evaluation of unsupervised outlier detection: measures, datasets, and an empirical study. Data Min. Knowl. Disc. 30(4), 891–927 (2016)

    Article  MathSciNet  Google Scholar 

  131. Y. Wang, Statistical techniques for network security: modern statistically-based intrusion detection and protection, IGI Glo (2008)

  132. A. Sari, A review of anomaly detection systems in cloud networks and survey of cloud security measures in cloud storage applications. Info. Sec. 6(02), 142–154 (2015)

    Google Scholar 

  133. X. Liu, W. Zheng, Y. Chawla, S. Yuan, J. Xie, Discovering spatio-temporal causal interactions in traffic data streams, ACM, SIGKDD, pp. 1010–1018 (2011)

  134. M. Ieva, F. Paganoni, Detecting and visualizing outliers in provider profiling via funnel plots and mixed effect models. Heal. Care Man. Sci. 18(2), 166–172 (2015)

    Article  Google Scholar 

  135. S. Aggarwal, C. Zhao, Y. Yu, Outlier detection in graph streams, ACM IEEE, ICDE, pp. 399–409, (2011)

  136. J. Gao, J. Liang, F. Fan, W. Wang, C. Sun, Y. Han, On community outliers and their efficient detection in information networks, ACM, SIGKDD, pp. 813–822 (2010)

  137. W. Wang, Y. Xu, Leveraging deep learning with lda-based text analytics to detect automobile insurance fraud. Dec. Sup. Sys. 105, 87–95 (2018)

    Article  Google Scholar 

  138. K. Zijlstra, W. van der Ark, L. Sijtsma, Outliers in questionnaire data: Can they be detected and should they be removed. Edu. and Beh. Stat. 36, 186–212 (2011)

    Article  Google Scholar 

  139. C. Liu, X. Chen, F. Lu, On detecting spatial categorical outliers. Geo. Inf. 18(3), 501–536 (2014)

    Google Scholar 

  140. C. Aggarwal, Outlier Analysis, 2nd edn. (Springer, Berlin, 2017)

    Book  Google Scholar 

  141. P. Billor, N. Hadi, A. Velleman, Blocked adaptive computationally-efficient outlier nominators. Comput. Stat. Data Anal. 34, 279–298 (2000)

    Article  Google Scholar 

  142. V. Boriah, S. Chandola, V. Kumar, Similarity measures for categorical data: a comparative evaluation, SIAM, SDM, pp. 243–254 (2008)

  143. V. Chandola, V. Boriah, S. Kumar, A framework for exploring categorical data, SIAM, SDM, 187–198 (2009)

  144. S. Wu, S. Wang, Parameter-free anomaly detection for categorical data. Machine learning and data mining in pattern recognition. Lecture Notes Comput. Sci. 6871, 112–126 (2011)

    Article  Google Scholar 

  145. S. Taha, A. Hadi, A general approach for automating outliers identification in categorical data, ACS/IEEE (AICCSA), pp 1–8 (2013)

  146. T. Shyu, M. Sarinnapakorn, K. Kuruppu-Appuhamilage, I. Chen, S. Chang, W. Goldring, Handling nominal features in anomaly intrusion detection problems, Work. Res. Iss. Dat. Eng. STDMA., 55–62 (2005)

  147. K. Koufakou, A. Ortiz, E. Georgiopoulos, M. Anagnostopoulos, G. Reynolds, A scalable and efficient outlier detection strategy for categorical data, IEEE, ICTAI, pp. 210–217 (2007)

  148. G. Koufakou, A. Georgiopoulos, M. Anagnostopoulos, Detecting outliers in high-dimensional datasets with mixed attributes, DMIN (2008)

  149. O. Taha, A. Hegazy, A proposed outliers identification algorithm for categorical data sets, INFOS, 1–5, (2010)

  150. E. Rokhman, N. Subanar, Winarko, Improving the performance of outlier detection methods for categorical data by using weighting function. Theor. App. Inf. Technol. 83, 327–336 (2016)

    Google Scholar 

  151. F. Zhao, X. Liang, J. Cao, A simple and effective outlier detection algorithm for categorical data. Mach. Learn. Cybern. 5, 469–477 (2014)

    Article  Google Scholar 

  152. L. Lei, D. Zhang, L. Zhang, Cloud model-based outlier detect algorithm for categorical data. Int. J. Database Theory Appl. 6(14), 199–213 (2013)

    Google Scholar 

  153. M. Bouguessa, A practical outlier detection approach for mixed-attribute data. Exp. Sys. Appl. 42, 8637–8649 (2015)

    Article  Google Scholar 

  154. A. Reddy, S. Babu, B. Govardhan, Outlier analysis of categorical data using NAVF. Inf. Econ. 17(1), 1–5 (2013)

    Google Scholar 

  155. Y. Liang, J. Chin, S. Dang, A new method for measuring uncertainty and fuzziness in rough set theory. Int J Gen Syst 31, 331–342 (2002)

    Article  MathSciNet  Google Scholar 

  156. S. He, Z. Xu, X. Huang, Z. Deng, Fp-outlier: Frequent pattern based outlier detection. ComSIS 2, 726–732 (2005)

    Article  Google Scholar 

  157. S. Ghoting, A. Otey, M. Parthasarathy, Loaded: link-based outlier and anomaly detection in evolving data sets, IEEE, ICDM, pp. 387–390 (2004)

  158. S. Otey, E. Ghoting, A. Parthasarathy, Fast distributed outlier detection in mixed-attribute data sets. Data Min. Knowl. Discov. 12(2–3), 203–228 (2006)

    Article  MathSciNet  Google Scholar 

  159. L. Pang, G. Cao, L. Chen, Outlier detection in complex categorical data by modeling the feature value couplings, in 25th International Conference on Artificial Intelligence and Statistics,,pp. 1902–1908 (2016)

  160. J. Das, K. Schneider, Detecting anomalous records in categorical datasets, ACM, SIGKDD, 220–229 (2007)

  161. H. Narita, K. Kitagawa, Detecting outliers in categorical record databases based on attribute associations, Progress in WWW Research and Development, pp. 111–123 (2008)

  162. A. Rashidi, L. Hashemi, S. Hamzeh, Anomaly detection in categorical datasets using Bayesian networks, Part II, AICI, pp. 610–619 (2011)

  163. B. Das, K. Schneider, J. Neill, Anomaly pattern detection in categorical datasets, ACM, SIGKDD, pp. 169–176 (2008)

  164. E. Castillo, J.M. Gutiérrez, A.S. Hadi, Expert Systems and Probabilistic Network Models (Springer, Berlin, 1997)

    Book  Google Scholar 

  165. K. Moore, A. Wong, Optimal reinsertion: a new search operator for accelerated and more accurate bayesian network structure learning, in 20th IEEE International Conference on Machine Learning and Applications - ICMLA 2021, pp. 552–559 (2003)

  166. J. Breunig, M. Kriegel, H. Ng, R. Sander, LOF: identifying density–based local outliers, ACM, SIGMOD, pp. 93–104 (2000)

  167. A. Yu, X. Qian, W. Lu, H. Zhou, Finding centric local outliers in categorical/numerical spaces. Knowl. Inf. Syst. 9, 309–338 (2006)

    Article  Google Scholar 

  168. P. Chawla, S. Sun, Slom: a new measure for local spatial outliers. Knowl. Inf. Syst. 9, 412–429 (2006)

    Article  Google Scholar 

  169. R. Joshi, V. Bhatnagar, Cbof: Cohesiveness-based outlier factor a novel definition of outlier-ness, Mach. Learn. Data Min. Pattern Recognit., 175–189 (2014)

  170. G. Suri, R. Murty, M. Athithan, A rough clustering algorithm for mining outliers in categorical data, 4th Int. Con. PReMI, pp. 170–175 (2013)

  171. G. Suri, R. Murty, M. Athithan, Detecting outliers in categorical data through rough clustering. Nat. Comput. 15, 385–394 (2016)

    Article  MathSciNet  Google Scholar 

  172. G. Suri, R. Murty, M. Athithan, An algorithm for mining outliers in categorical data through ranking, in 12th Int. Conf. (HIS), IEEE, pp. 247–252 (2012)

  173. G. Suri, R. Murty, N. Athithan, A ranking-based algorithm for detection of outliers in categorical data. Int. J. Hybrid Intell. Syst. 11, 1–11 (2014)

    Google Scholar 

  174. Z. Huang, A fast clustering algorithm to cluster very large categorical data sets in data mining. DMKM, ACM, SIGKDD, pp. 1–8 (1997)

  175. T. Knorr, E. Ng, A unified approach for mining outliers. CASCON, pp. 236–248 (1997)

  176. V. Knorr, E. Ng, R. Tucakov, Distance-based outliers: algorithms and applications. VLDB J 8, 237–253 (2000)

    Article  Google Scholar 

  177. K. Ramaswamy, S. Rastogi, R. Shim, Efficient algorithms for mining outliers from large data sets, ACM, SIGMOD, pp. 427–438 (2000)

  178. C. Angiulli, F. Basta, S. Pizzuti, Distance-based detection and prediction of outliers. IEEE Trans. Knowl. Data Eng. 18(2), 145–160 (2006)

    Article  Google Scholar 

  179. F. Angiulli, F. Fassetti, Fast outlier detection in high dimensional spaces, in Euro. Conf. on the Prin. of Data Min and Kno Disc., pp. 19–26 (2002)

  180. D. Ebdon, Statistics in Geography: A Practical Approach-Revised with 17 Programs (Wiley-Blackwell, Hoboken, 1991)

    Google Scholar 

  181. R. Bhaduri, K. Matthews, B. Giannella, Algorithms for speeding up distance-based outlier detection, ACM, SIGKDD, pp. 895–867 (2011)

  182. S. Li, S. Lee, R. Lang, Mining distance-based outliers from categorical data, IEEE, ICDM, pp. 225–230 (2007)

  183. E. Ghoting, A. Parthasarathy, S. Otey, Fast mining of distance-based outliers in high dimensional datasets. DMKD 16(3), 349–364 (2008)

    MathSciNet  Google Scholar 

  184. C. Böhm, C. Haegler, K. Müller, N. Plant, Coco: coding cost for parameter-free outlier detection, ACM, SIGKDD, pp. 149–158 (2009)

  185. J. Smets, K. Vreeken, The odd one out: identifying and characterising anomalies, SIAM, SDM, pp. 804–815 (2011)

  186. C. Akoglu, L. Tong, H. Vreeken, J. Faloutsos, Fast and reliable anomaly detection in categorical data, ACM, CIKM, pp. 415–424 (2012)

  187. A. Taha, A. S. Hadi, Anomaly detection methods for categorical data: a review. ACM Comput. Surv. 52(2), 1–35 (2019). https://doi.org/10.1145/3312739

    Article  Google Scholar 

  188. G. Desrosiers, C. Karypis, A comprehensive survey of neighborhood-based recommendation methods, in Recommender Systems Handbook Recommender Systems Handbook, pp. 107–144 (2011)

  189. V. Chandola, V. Banerjee, A. Kumar, Anomaly detection for discrete sequences: a survey. IEEE Trans. Knowl. Data Eng. 24(5), 823–839 (2012)

    Article  Google Scholar 

  190. K. Ge, Y. Xiong, H. Zhou, Z. Ozdemir, H. Yu, J. Lee, Top-eye: top-k evolving trajectory outlier detection, ACM, CIKM, pp. 1–4 (2010)

Download references

Author information

Authors and Affiliations

  1. Department of Electronics and Electrical Communications Engineering, Faculty of Electronic Engineering, Menoufia University, Menouf, 32952, Egypt

    Fatma M. Ghamry, Adel S. El-Fishawy, Fathi E. Abd El-Samie & Moawad I. Dessouky

  2. Communication and Electronics Department, Bilbeis Higher Institute for Engineering, BHIE, Bilbeis, Egypt

    Fatma M. Ghamry

  3. Department of Industrial Electronics and Control Engineering, Faculty of Electronic Engineering, Menoufia University, Menouf, 32952, Egypt

    Ghada M. El-Banby

  4. Department of Information Technology, College of Computer and Information Sciences, Princess Nourah Bint Abdulrahman University, Riyadh, Saudi Arabia

    Fathi E. Abd El-Samie

Authors
  1. Fatma M. Ghamry
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ghada M. El-Banby
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Adel S. El-Fishawy
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fathi E. Abd El-Samie
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Moawad I. Dessouky
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fatma M. Ghamry.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ghamry, F.M., El-Banby, G.M., El-Fishawy, A.S. et al. A survey of anomaly detection techniques. J Opt 53, 756–774 (2024). https://doi.org/10.1007/s12596-023-01147-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12596-023-01147-4

Keywords

Search

Navigation