A Survey and Comparison of Performance Evaluation in Intrusion Detection Systems

  • Jason ErnstEmail author
  • Tarfa Hamed
  • Stefan Kremer


Performance evaluation is an important aspect when designing a system. However, with intrusion detection systems (IDS), there are many other factors to consider. What are the metrics which are being used to compare the systems? Which attacks do particular approaches detect? Is the solution able to adapt and recognize new attacks, or is it limited to a set of attacks which were known at the time the system was designed? This chapter provides an overview of some of these concerns and tries to highlight in each surveyed IDS which metrics are used for performance evaluation, whether or not the solution is flexible, and which attacks the IDS is able to detect. This will provide the reader with a good basis for choosing the type of approach to use to guard against attacks, or as a basis to dig deeper into a particular aspect of intrusion detection.


  1. 1.
    Cup, K. (1999). Dataset. Available at the following website
  2. 2.
    Sharma, V., & Nema, A. (2013). Innovative genetic approach for intrusion detection by using decision tree. In 2013 international conference on communication systems and network technologies (pp. 418–422).Google Scholar
  3. 3.
    Lippmann, R., Haines, J. W., Fried, D. J., Korba, J., & Das, K. (2000). The 1999 DARPA off-line intrusion detection evaluation. Computer Networks, 34(4), 579–595.CrossRefGoogle Scholar
  4. 4.
    J. G. Elevate Communications (2016). Terabit-scale multi-vector DDoS attacks to become the new normal in 2017, Predict DDoS Experts, Business Wire.Google Scholar
  5. 5.
    García, S., Grill, M., Stiborek, J., & Zunino, A. (2014). An empirical comparison of botnet detection methods. Computers & Security, 45, 100–123.CrossRefGoogle Scholar
  6. 6.
    Małowidzki, M., Berezinski, P., & Mazur, M. (2015). Network intrusion detection: half a kingdom for a good dataset. In Proceedings of NATO STO SAS-139 Workshop. Portugal.Google Scholar
  7. 7.
    Scully, P. (2016). Where can I get the latest dataset for a network intrusion detection system?. Quora [Online]. Available: . Accessed January 12, 2017.
  8. 8.
    ubershmekel (2012). Precision, recall, sensitivity and specificity. Ubershmekel’s Uberpython Pythonlog [Online]. Available: . Accessed February 09, 2017.
  9. 9.
    Natesan, P., Balasubramanie, P., & Gowrison, G. (2012). Improving the attack detection rate in network intrusion detection using adaboost algorithm. Journal of Computer Science, 8(7), 1041–1048.CrossRefGoogle Scholar
  10. 10.
    Mo, Y., Ma, Y., & Xu, L. (2008). Design and implementation of intrusion detection based on mobile agents. In 2008 IEEE international symposium on IT in medicine and education (pp. 278–281).Google Scholar
  11. 11.
    Uppuluri, P., & Sekar, R. (2001). Experiences with specification-based intrusion detection. In Recent advances in intrusion detection (pp. 172–189).Google Scholar
  12. 12.
    Sekar, R. et al. (2002). Specification-based anomaly detection: A new approach for detecting network intrusions. In Proceedings of the 9th ACM conference on computer and communications security (pp. 265–274). Washington, DC, USA.Google Scholar
  13. 13.
    Shon, T., & Moon, J. (2007). A hybrid machine learning approach to network anomaly detection. Information Science, 177(18), 3799–3821.CrossRefGoogle Scholar
  14. 14.
    MeeraGandhi, G., Appavoo, K., & Srivasta, S. (2010). Effective network intrusion detection using classifiers decision trees and decision rules. International Journal Advanced network and Application, 2(3), 686–692.Google Scholar
  15. 15.
    Trinius, P., Willems, C., Holz, T., & Rieck, K. (2009). A malware instruction set for behavior-based analysis. Tech. Rep. TR-2009-07, University of Mannheim.Google Scholar
  16. 16.
    Xu, J., & Wu, S. (2010). Intrusion detection model of mobile agent based on Aglets. In 2010 international conference on computer application and system modeling (ICCASM 2010) (Vol. 4, pp. V4-347–V4-350).Google Scholar
  17. 17.
    Gong, Y., Mabu, S., Chen, C., Wang, Y., & Hirasawa, K. (2009). Intrusion detection system combining misuse detection and anomaly detection using Genetic Network Programming. ICCAS-SICE, 2009.Google Scholar
  18. 18.
    Yang, W., Wan, W., Guo, L., & Zhang L. J. (2007). An efficient intrusion detection model based on fast inductive learning. In 2007 international conference on machine learning and cybernetics (Vol. 6, pp. 3249–3254).Google Scholar
  19. 19.
    Lan, F., Chunlei, W., & Guoqing, M. (2010). A framework for network security situation awareness based on knowledge discovery. In 2nd international conference on computer engineering and technology (Vol. 1, pp. V1-226–V1-231).Google Scholar
  20. 20.
    Jaiganesh, V., Sumathi, P., & Mangayarkarasi, S. (2013). An analysis of intrusion detection system using back propagation neural network. In 2013 international conference on information communication and embedded systems (ICICES) (pp. 232–236).Google Scholar
  21. 21.
    Shanmugavadivu, R., & Nagarajan, N. (2011). Network intrusion detection system using fuzzy logic. Indian Journal of Computer Science and Engineering (IJCSE), 2(1), 101–111.Google Scholar
  22. 22.
    Sen, J. (2010). Efficient routing anomaly detection in wireless mesh networks. In 2010 first international conference on integrated intelligent computing (pp. 302–307).Google Scholar
  23. 23.
    Aggarwal, P., & Sharma, S. K. (2015). An empirical comparison of classifiers to analyze intrusion detection. In 2015 fifth international conference on advanced computing communication technologies (pp. 446–450).Google Scholar
  24. 24.
    Vyas, T., Prajapati, P., & Gadhwal, S. (2015). A survey and evaluation of supervised machine learning techniques for spam e-mail filtering. In 2015 IEEE international conference on electrical, computer and communication technologies (ICECCT) (pp. 1–7).Google Scholar
  25. 25.
    Rieck, K., Schwenk, G., Limmer, T., Holz, T., & Laskov, P. (2010). Botzilla: Detecting the phoning home of malicious software. In Proceedings of the 2010 ACM symposium on applied computing (pp. 1978–1984).Google Scholar
  26. 26.
    Lane, T. (2006). A decision-theoretic, semi-supervised model for intrusion detection. In M. A. Maloof (Ed.), Machine learning and data mining for computer security (pp. 157–177). London: Springer.CrossRefGoogle Scholar
  27. 27.
    Warrender, C., Forrest, S., & Pearlmutter, B. (1999). Detecting intrusions using system calls: alternative data models. In Proceedings of the 1999 IEEE symposium on security and privacy (Cat. No.99CB36344) (pp. 133–145).Google Scholar
  28. 28.
    Joo, D., Hong, T., & Han, I. (2003). The neural network models for IDS based on the asymmetric costs of false negative errors and false positive errors. Expert System with Applications, 25(1), 69–75.CrossRefGoogle Scholar
  29. 29.
    Kolias, C., Kambourakis, G., Stavrou, A., & Gritzalis, S. (2016). Intrusion detection in 802.11 networks: Empirical evaluation of threats and a public dataset. IEEE Communications Surveys Tutorials, 18(1), 184–208.CrossRefGoogle Scholar
  30. 30.
    Subramanian, U., & Ong, H. S. (2014). Analysis of the effect of clustering the training data in Naive Bayes classifier for anomaly network intrusion detection. Journal of Advances in Computer Networks, 2(1), 91–94.CrossRefGoogle Scholar
  31. 31.
    Casas, P., Mazel, J., & Owezarski, P. (2012). Unsupervised network intrusion detection systems: Detecting the unknown without knowledge. Computer Communications, 35(7), 772–783.CrossRefGoogle Scholar
  32. 32.
    Muzammil, M. J., Qazi, S., & Ali, T. (2013). Comparative analysis of classification algorithms performance for statistical based intrusion detection system. In 3rd IEEE international conference on computer, control and communication (IC4) (pp. 1–6).Google Scholar
  33. 33.
    Tan, Z., Jamdagni, A., He, X., Nanda, P., Liu, R. P., & Hu, J. (2015). Detection of denial-of-service attacks based on computer vision techniques. IEEE Transactions Computers, 64(9), 2519–2533.MathSciNetCrossRefzbMATHGoogle Scholar
  34. 34.
    Bhuse, V., & Gupta, A. (2006). Anomaly intrusion detection in wireless sensor networks. Journal of High Speed Networks, 15(1), 33–51.Google Scholar
  35. 35.
    Zhao, Y. J., Wei, M. J., & Wang, J. (2013). Realization of intrusion detection system based on the improved data mining technology. In 8th international conference on Computer Science and Education. Colombo, Sri Lanka.Google Scholar
  36. 36.
    Mahoney, M. V., & Chan, P. K. (2001). PHAD: Packet header anomaly detection for identifying hostile network traffic (Tech. Rep. CS-2001-4). Melbourne, FL: Florida Institute of Technology.Google Scholar
  37. 37.
    Sedjelmaci, H., & Senouci, S. M. (2015). An accurate and efficient collaborative intrusion detection framework to secure vehicular networks. Computers and Electrical Engineering, 43, 33–47.CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  1. 1.University of GuelphGuelphCanada

Personalised recommendations