Towards a Scalability and Energy Efficiency Benchmark for VNF

  • Norbert SchmittEmail author
  • Jóakim von Kistowski
  • Samuel Kounev
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10661)


Network Function Virtualization (NFV) is the transfer of network functions from dedicated devices to high-volume commodity servers. It opens opportunities for flexibility and energy savings. Concrete insights on the flexibility of specific NFV environments require measurement methodologies and benchmarks. However, current benchmarks are not measuring the ability of a virtual network function (VNF) to scale either horizontally or vertically. We therefore envision a new benchmark that measures a VNF’s ability to scale while evaluating its energy efficiency at the same time. Such a benchmark would enable the selection of a suitable VNF for changing demands, deployed at an existing or new resource landscape, while minimizing energy costs.


  1. 1.
    Brown, R., Masanet, E., Nordman, B., Tschudi, B., Shehabi, A., Stanley, J., Koomey, J., Sartor, D., Chan, P., Loper, J., Capana, S., Hedman, B., Duff, R., Haines, E., Sass, D., Fanara, A.: Report to Congress on Server and Data Center Energy Efficiency: Public Law 109–431 (2007)Google Scholar
  2. 2.
    Whitney, J., Delforge, P.: Scaling Up Energy Efficiency Across the Data Center Industry: Evaluating Key Drivers and Barriers. Data Center Efficiency Assessment (2014)Google Scholar
  3. 3.
    von Kistowski, J., Herbst, N., Kounev, S.: Modeling variations in load intensity over time. In: Proceedings of the third International Workshop on Large Scale Testing, LT 2014, pp. 1–4 (2014)Google Scholar
  4. 4.
    Chiosi, M., Clarke, D., Willis, P., Reid, A., Feger, J., Bugenhagen, M., Khan, W., Fargano, M., Cui, C., Deng, H., Benitez, J., Michel, U., Damker, H., Ogaki, K., Matsuzaki, T., Fukui, M., Shimano, K., Delisle, D., Loudier, Q., Kolias, C., Guardini, I., Demaria, E., Miverva, R., Manzalini, A., López, D., Salguero, F., Ruhl, F., Sen, P.: Network functions virtualisation - introductory white paper. In: SDN and OpenFlow World Congress (2012)Google Scholar
  5. 5.
    Lorido-Botran, T., Miguel-Alonso, J., Lozano, J.A.: A review of auto-scaling techniques for elastic applications in cloud environments. J. Grid Comput. 12(4), 559–592 (2014)CrossRefGoogle Scholar
  6. 6.
    Herbst, N., Kounev, S., Reussner, R.: Elasticity in cloud computing: what it is, and what it is not. In: Proceedings of the 10th International Conference on Autonomic Computing, ICAC, San Jose, California, USA (2013)Google Scholar
  7. 7.
    von Kistowski, J., A. Arnold, J., Huppler, K., Lange, K.-D., L. Henning, J., Cao, P.: How to build a benchmark. In: Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering, ICPE, Austin, Texas, USA (2015)Google Scholar
  8. 8.
    Power and Performance Benchmark Methodology V2.1. Standard Performance Evaluation Corporation (SPEC), Gainesville, USA (2012)Google Scholar
  9. 9.
    Huppler, K.R.: Performance per watt - benchmarking ways to get more for less. In: Nambiar, R., Poess, M. (eds.) TPCTC 2012. LNCS, vol. 7755, pp. 60–74. Springer, Heidelberg (2013). CrossRefGoogle Scholar
  10. 10.
    Schall, D., Hoefner, V., Kern, M.: Towards an enhanced benchmark advocating energy-efficient systems. In: Nambiar, R., Poess, M. (eds.) TPCTC 2011. LNCS, vol. 7144, pp. 31–45. Springer, Heidelberg (2012). CrossRefGoogle Scholar
  11. 11.
    von Kistowski, J., Becket, J., Lange, K.-D., Block, H., A. Arnold, J., Kounev, S.: Energy efficiency of hierarchical server load distributions strategies. In: 2015 IEEE 23rd International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunications Systems (MASCOTS), Atlanta, USA (2015)Google Scholar
  12. 12.
    von Kistowski, J., Block, H., Beckett, J., Spradling, C., Lange, K.-D., Kounev, S.: Variations in CPU power consumption. In: Proceedings of the 7th ACM/SPEC International Conference on Performance Engineering, ICPE 2016, Delft, The Netherlands, pp. 147–158 (2016)Google Scholar
  13. 13.
    von Kistowski, J., Block, H., Beckett, J., Lange, K.-D., A. Arnold, J., Kounev, S.: Analysis of the influences on server power consumption and energy efficiency for CPU-intensive workloads. In: Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering, ICPE 2015, Austin, Texas, pp. 223–234 (2015)Google Scholar
  14. 14.
    Lange, S., Nguyen-Ngoc, A., Gebert, S., Zinner, T., Jarschel, M., Köpsel, A., Sune, M., Raumer, D., Gallenmüller, S., Calre, G., Tran-Gia, P.: Performance benchmarking of a software-based LTE SGW. In: 2015 11th International Conference on Network and Service Management (CNSM), Barcelona, Spain (2015)Google Scholar
  15. 15.
    Falkner, M., Leivadeas, A., Lambadaris, I., Kesidis, G.: Performance analysis of virtualized network functions on virtualized systems architectures. In: 2016 IEEE 21st International Workshop on Computer Aided Modelling and Design of Communication Links and Networks (CAMAD), Toronto, Canada (2016)Google Scholar
  16. 16.
    Bradner, S., McQuaid, J.: RFC 2544: Benchmarking Methodology for Network Interconnected Devices. IETF (1999)Google Scholar
  17. 17.
    Bradner, S., McQuaid, J.: RFC 6201: Device Reset Characterization. IETF (2011)Google Scholar
  18. 18.
    Bradner, S., Dubray, K., McQuaid, J., Morton, A.: RFC 6815: Applicability Statement for RFC 2544: Use on Production Networks Considered Harmful. IETF (2012)Google Scholar
  19. 19.
    Rosa Ed., E., Szabo, R.: VNF Benchmarking Methodology (draft expired in September 2016). IETF (2016)Google Scholar
  20. 20.
    Morton, A.: Considerations for Benchmarking Virtual Network Functions and Their Infrastructure (draft expired in August 2015). IETF (2015)Google Scholar
  21. 21.
    Moens, H., De Turck, F.: VNF-P: a model for efficient placement for virtualized network functions. In: 2014 10th International Conference on Network and Service Management (CNSM), Rio de Janeiro, Brazil (2014)Google Scholar
  22. 22.
    Addis, B., Belabed, D., Bouet, M., Secci, S.: Virtual network functions placement and routing optimization. In: 2015 IEEE 4th International Conference on Cloud Networking (CloudNet), Niagara Falls, Canada (2015)Google Scholar
  23. 23.
    SPEC VIRT_SC 2013. The Standard Performance Evaluation Corporation (SPEC) (2013).
  24. 24.
    Smith, W.D., Sebastian, S.: Virtualization Performance Insights from TPC-VMS. Transaction Processing Performance Council (2013)Google Scholar
  25. 25.
    Bond, A., Johnson, D., Kopczynski, G., Taheri, H.R.: Profiling the performance of virtualized databases with the TPCx-V benchmark. In: Nambiar, R., Poess, M. (eds.) TPCTC 2015. LNCS, vol. 9508, pp. 156–172. Springer, Cham (2016). CrossRefGoogle Scholar
  26. 26.
    TPC-Energy Specification. Transaction Processing Performance Council, Version 1.5.0 (2012).
  27. 27.
    SPECpower_ssj2008. The Standard Performance Evaluation Corporation (SPEC) (2007).
  28. 28.
    Server Efficiency Rating Tool 2. The Standard Performance Evaluation Corporation (SPEC) (2017).
  29. 29.
    Chauffeur™WDK 2. The Standard Performance Evaluation Corporation (SPEC) (2017).

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  • Norbert Schmitt
    • 1
    Email author
  • Jóakim von Kistowski
    • 1
  • Samuel Kounev
    • 1
  1. 1.University of WürzburgWürzburgGermany

Personalised recommendations