Investigating Deep Learning for Collective Anomaly Detection - An Experimental Study

  • Mohiuddin Ahmed
  • Al-Sakib Khan PathanEmail author
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 969)


This paper explores the effectiveness of deep learning and other supervised learning algorithms for collective anomaly detection. Almost all the approaches so far proposed for DoS (Denial of Service) attack detection with the aid of collective anomaly detection are unsupervised in nature. Due to this reason, often those approaches show high false alarm rates. To reduce the high false alarm rate, we have done some experiments to investigate the suitability of deep learning for this field. Interestingly, the obtained experimental results on UNSW-NB15 and KDD Cup 1999 datasets show that the deep learning implemented using H2O achieves approximately 97% recall for collective anomaly detection. Hence, deep learning outperforms a wide range of unsupervised techniques for collective anomaly detection. This is the first reported work that investigates collective anomaly detection problem using deep learning.


Deep learning Collective anomaly DoS attack Network traffic analysis 


  1. 1.
    Yu, R., He, X., Liu, Y.: Glad: group anomaly detection in social media analysis. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2014, pp. 372–381. ACM, New York (2014)Google Scholar
  2. 2.
    Ahmed, M., Mahmood, A., Hu, J.: A survey of network anomaly detection techniques. J. Netw. Comput. Appl. 60, 19–31 (2015)CrossRefGoogle Scholar
  3. 3.
    Ahmed, M., Mahmood, A.N., Hu, J.: Outlier detection. In: The State of the Art in Intrusion Prevention and Detection, pp. 3–21. CRC Press, New York (2014)Google Scholar
  4. 4.
    Hawkins, D.: Identification of Outliers (Monographs on Statistics and Applied Probability), 1st edn. Springer, Dordrecht (1980). Scholar
  5. 5.
    Cheng, T., Li, Z.: A multiscale approach for spatio-temporal outlier detection. Trans. GIS 10(2), 253–263 (2006)CrossRefGoogle Scholar
  6. 6.
    Breunig, M.M., Kriegel, H.-P., Ng, R.T., Sander, J.: Lof: identifying density-based local outliers. SIGMOD Rec. 29(2), 93–104 (2000)CrossRefGoogle Scholar
  7. 7.
    Ramaswamy, S., Rastogi, R., Shim, K.: Efficient algorithms for mining outliers from large data sets. SIGMOD Rec. 29(2), 427–438 (2000)CrossRefGoogle Scholar
  8. 8.
    Muandet, K., Schölkopf, B.: One-class support measure machines for group anomaly detection. CoRR, abs/1303.0309(2013)Google Scholar
  9. 9.
    Struyf, A., Hubert, M., Rousseeuw, P.: Clustering in an object-oriented environment. J. Stat. Softw. 1(4), 1–30 (1997)Google Scholar
  10. 10.
    Ahmed, M.: Collective anomaly detection techniques for network traffic analysis. Ann. Data Sci. 5, 497–512 (2018)CrossRefGoogle Scholar
  11. 11.
    Ahmed, M., Mahmood, A.: Network traffic analysis based on collective anomaly detection. In: 9th IEEE International Conference on Industrial Electronics and Applications, pp. 1141–1146. IEEE (2014)Google Scholar
  12. 12.
    Ahmed, M., Mahmood, A.N.: Novel approach for network traffic pattern analysis using clustering-based collective anomaly detection. Ann. Data Sci. 2(1), 111–130 (2015)CrossRefGoogle Scholar
  13. 13.
    Ahmed, M., Mahmood, A.N.: Network traffic pattern analysis using improved information theoretic co-clustering based collective anomaly detection. In: Tian, J., Jing, J., Srivatsa, M. (eds.) SecureComm 2014. LNICSSITE, vol. 153, pp. 204–219. Springer, Cham (2015). Scholar
  14. 14.
    Ahmed, M.: Thwarting dos attacks: a framework for detection based on collective anomalies and clustering. Computer 50(9), 76–82 (2017)CrossRefGoogle Scholar
  15. 15.
    Ahmed, M., Anwar, A., Mahmood, A.N., Shah, Z., Maher, M.J.: An investigation of performance analysis of anomaly detection techniques for big data in scada systems. EAI Endorsed Trans. Ind. Netw. Intell. Syst. 15(3), 1–16 (2015)Google Scholar
  16. 16.
    Leung, K., Leckie, C.: Unsupervised anomaly detection in network intrusion detection using clusters. In: Proceedings of the Twenty-eighth Australasian Conference on Computer Science - Volume 38, ACSC 2005, Darlinghurst, Australia, pp. 333–342. Australian Computer Society, Inc. (2005)Google Scholar
  17. 17.
    Deng, L., Yu, D.: Deep learning: methods and applications. Found. Trends Signal Process. 7(4), 197–387 (2014)MathSciNetCrossRefGoogle Scholar
  18. 18.
    Ahmed, M., Choudhury, N., Uddin, S.: Anomaly detection on big data in financial markets. In: Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017, ASONAM 2017, pp. 998–1001. ACM, New York (2017)Google Scholar
  19. 19.
  20. 20.
    Candel, A., Parmar, V., LeDell, E., Arora, A.: Deep Learning with H2O (2016). Accessed 24 Aug 2017
  21. 21.
    Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016). http://www.deeplearningbook.orgzbMATHGoogle Scholar
  22. 22.
    Tavallaee, M., Bagheri, E., Lu, W., Ghorbani, A.A.: A detailed analysis of the KDD cup 99 dataset. In: Proceedings of the 2nd IEEE International Conference on Computational Intelligence for Security and Defense Applications, CISDA 2009, pp. 53–58. IEEE Press, Piscataway (2009)Google Scholar
  23. 23.
    Moustafa, N., Slay, J.: The evaluation of network anomaly detection systems: statistical analysis of the UNSW-NB15 data set and the comparison with the KDD99 data set. Inf. Secur. J. A Global Perspect. 25(1–3), 18–31 (2016)CrossRefGoogle Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  1. 1.Centre for Cyber Security and GamesCanberra Institute of TechnologyCanberraAustralia
  2. 2.Department of Computer Science and EngineeringSoutheast UniversityDhakaBangladesh

Personalised recommendations