Benchmarking Intrusion Detection Systems with Adaptive Provisioning of Virtualized Resources

  • Aleksandar MilenkoskiEmail author
  • K. R. Jayaram
  • Samuel Kounev


With the increasing popularity of virtualization, deploying intrusion detection systems (IDSes) in virtualized environments, for example, in virtual machines as virtualized network functions, has become an emerging practice. Modern virtualized environments feature on demand provisioning of virtualized processing and memory resources to virtual machines, dynamically adapting its intensity in order to meet resource demands. Such a provisioning may have a significant impact on many properties of an IDS deployed in a virtual machine, for example, on its attack detection accuracy. However, conventional metrics for quantifying IDS attack detection accuracy do not capture this impact, which may lead to inaccurate assessments of the IDS’s accuracy at detecting attacks. In this chapter, we discuss in detail on the impact of on demand provisioning of virtualized resources on IDS attack detection accuracy. Further, we discuss on relevant issues related to the use of conventional metrics for quantifying IDS attack detection accuracy. Finally, we present a preliminary metric and measurement methodologies, which allow for the accurate assessment of IDS attack detection accuracy taking on-demand resource provisioning into account.


Virtual Machine False Positive Rate Intrusion Detection True Positive Rate Intrusion Detection System 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.



This work was funded by the German Research Foundation (DFG) under grant No. KO 3445/16-1. This research has been supported by the Research Group of the Standard Performance Evaluation Corporation (SPEC,, The authors would like to thank Alexander Leonhardt for providing experimental data.


  1. 1.
    Wesam Dawoud, Ibrahim Takouna, and Christoph Meinel. Elastic Virtual Machine for Fine-Grained Cloud Resource Provisioning. In P.Venkata Krishna, M.Rajasekhara Babu, and Ezendu Ariwa, editors, Global Trends in Computing and Communication Systems, volume 269 of Communications in Computer and Information Science, pages 11–25. Springer, 2012.Google Scholar
  2. 2.
    Jr. Gaffney, J.E. and J.W. Ulvila. Evaluation of intrusion detectors: a decision theory approach. In Proceedings of the 2001 IEEE Symposium on Security and Privacy, pages 50–61, 2001.Google Scholar
  3. 3.
    Frank Gens, Robert Mahowald, Richard L. Willards, David Bradshaw, and Chris Morris. Cloud computing 2010: An idc update, 2010.Google Scholar
  4. 4.
    Guofei Gu, Prahlad Fogla, David Dagon, Wenke Lee, and Boris Skorić. Measuring intrusion detection capability: an information-theoretic approach. In Proceedings of the 2006 ACM Symposium on Information, computer and communications security (ASIACCS), pages 90–101, New York, NY, USA, 2006. ACM.Google Scholar
  5. 5.
    Mike Hall and Kevin Wiley. Capacity verification for high speed network intrusion detection systems. In Proceedings of the 5th International Conference on Recent Advances in Intrusion Detection (RAID), pages 239–251, Berlin, Heidelberg, 2002. Springer-Verlag.Google Scholar
  6. 6.
    J. Hancock and P. Wintz. Signal Detection Theory. McGraw–Hill, New York, 1966.Google Scholar
  7. 7.
    Evangelos Kotsovinos. Virtualization: Blessing or curse? Queue, 8(11):40:40–40:46, November 2010.Google Scholar
  8. 8.
    Sajib Kundu, Raju Rangaswami, Ajay Gulati, Ming Zhao, and Kaushik Dutta. Modeling virtualized applications using machine learning techniques. In Proceedings of the 8th ACM SIGPLAN/SIGOPS conference on Virtual Execution Environments, VEE ’12, pages 3–14, New York, NY, USA, 2012. ACM.Google Scholar
  9. 9.
    Flavio Lombardi and Roberto Di Pietro. Secure virtualization for cloud computing. Journal of Network and Computer Applications, 34(4):1113–1122, July 2011.Google Scholar
  10. 10.
    Neil MacDonald. Yes, Hypervisors are vulnerable., 2011.
  11. 11.
    R. A. Maxion and R. R. Roberts. Proper Use of ROC Curves in Intrusion/Anomaly detection. Technical Report CS-TR-871, School of Computing Science, University of Newcastle upon Tyne, November 2004.Google Scholar
  12. 12.
    Peter Mell, Vincent Hu, Richard Lippmann, Josh Haines, and Marc Zissman. An Overview of Issues in Testing Intrusion Detection Systems, 2003.Google Scholar
  13. 13.
    Yuxin Meng and Wenjuan Li. Adaptive Character Frequency-Based Exclusive Signature Matching Scheme in Distributed Intrusion Detection Environment. In IEEE 11th International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom), pages 223–230, June 2012.Google Scholar
  14. 14.
    Aleksandar Milenkoski, Marco Vieira, Samuel Kounev, Alberto Avrtizer, and Bryan D. Payne. Evaluating Computer Intrusion Detection Systems: A Survey of Common Practices. ACM Computing Surveys, 2015. To appear.Google Scholar
  15. 15.
    N. Mohammed, H. Otrok, Lingyu Wang, M. Debbabi, and P. Bhattacharya. Mechanism Design-Based Secure Leader Election Model for Intrusion Detection in MANET. IEEE Transactions on Dependable and Secure Computing, 8(1):89–103, January-February 2011.Google Scholar
  16. 16.
    Diego Perez-Botero, Jakub Szefer, and Ruby B. Lee. Characterizing hypervisor vulnerabilities in cloud computing servers. In Proceedings of the 2013 International Workshop on Security in Cloud Computing, Cloud Computing ’13, pages 3–10. ACM, 2013.Google Scholar
  17. 17.
    Martin Roesch. Snort - Lightweight Intrusion Detection for Networks. In Proceedings of the 13th USENIX conference on System Administration (LISA), pages 229–238. USENIX Association, 1999.Google Scholar
  18. 18.
    Karen Scarfone and Peter Mell. Guide to Intrusion Detection and Prevention Systems (IDPS), 2007. NIST Special Publication 900-94.Google Scholar
  19. 19.
    Sushant Sinha, Farnam Jahanian, and Jignesh M. Patel. WIND: Workload-aware INtrusion Detection. In Proceedings of the 9th International Conference on Recent Advances in Intrusion Detection (RAID), pages 290–310, Berlin, Heidelberg, 2006. Springer Verlag.Google Scholar
  20. 20.
    S. Spinner, S. Kounev, Xiaoyun Zhu, Lei Lu, M. Uysal, A. Holler, and R. Griffith. Runtime Vertical Scaling of Virtualized Applications via Online Model Estimation. In IEEE Eighth International Conference on Self-Adaptive and Self-Organizing Systems (SASO), pages 157–166, 2014.Google Scholar
  21. 21.
    Tcpreplay.Google Scholar
  22. 22.
    Jing Xu, Ming Zhao, José Fortes, Robert Carpenter, and Mazin Yousif. Autonomic Resource Management in Virtualized Data Centers Using Fuzzy Logic-based Approaches. Cluster Computing, 11(3):213–227, 2008.CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Aleksandar Milenkoski
    • 1
    Email author
  • K. R. Jayaram
    • 2
  • Samuel Kounev
    • 1
  1. 1.University of WürzburgWürzburgGermany
  2. 2.Thomas J. Watson Research CenterNew York CityUSA

Personalised recommendations