Multi-tenancy Performance Benchmark for Web Application Platforms

  • Rouven Krebs
  • Alexander Wert
  • Samuel Kounev
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7977)


Cloud environments reduce data center operating costs through resource sharing and economies of scale. Infrastructure-as-a-Service is one example that leverages virtualization to share infrastructure resources. However, virtualization is often insufficient to provide Software-as-a-Service applications due to the need to replicate the operating system, middleware and application components for each customer. To overcome this problem, multi-tenancy has emerged as an architectural style that allows to share a single Web application instance among multiple independent customers, thereby significantly improving the efficiency of Software-as-a-Service offerings. A number of platforms are available today that support the development and hosting of multi-tenant applications by encapsulating multi-tenancy specific functionality. Although a lack of performance guarantees is one of the major obstacles to the adoption of cloud computing, in general, and multi-tenant applications, in particular, these kinds of applications and platforms have so far not been in the focus of the performance and benchmarking community. In this paper, we present an extended version of an existing and widely accepted application benchmark adding support for multi-tenant platform features. The benchmark is focused on evaluating the maximum throughput and the amount of tenants that can be served by a platform. We present a case study comparing virtualization and multi-tenancy. The results demonstrate the practical usability of the proposed benchmark in evaluating multi-tenant platforms and gives insights that help to decide for one sharing approach.


Platform SaaS Multi-tenancy Benchmark 


  1. 1.
    TPC Benchmark W, Transaction Processing Performance Council (2002)Google Scholar
  2. 2.
    Armbrust, M., Fox, A., Griffith, R., Joseph, A.D., Katz, R.H., Konwinski, A., Lee, G., Patterson, D.A., Rabkin, A., Stoica, I., Zaharia, M.: Above the clouds: A berkeley view of cloud computing. Tech. Rep. UCB/EECS-2009-28, EECS Department, University of California, Berkeley (February 2009)Google Scholar
  3. 3.
    Binnig, C., Kossmann, D., Kraska, T., Loesing, S.: How is the weather tomorrow?: towards a benchmark for the cloud. In: Proceedings of the Second International Workshop on Testing Database Systems (2009)Google Scholar
  4. 4.
    Bitcurrent. Bitcurrent cloud computing survey 2011. Tech. rep., bitcurrent (2011)Google Scholar
  5. 5.
    Cain, H.W., Rajwar, R., Marden, M., Lipasti, M.H.: An architectural evaluation of Java TPC-W. In: Proceedings of the Seventh International Symposium on High-Performance Computer Architecture (2001)Google Scholar
  6. 6.
    Chong, F., Carraro, G., Wolter, R.: Multi-tenant data architecture (June 2006)Google Scholar
  7. 7.
    Cooper, B.F., Silberstein, A., Tam, E., Ramakrishnan, R., Sears, R.: Benchmarking cloud serving systems with ycsb. In: Proceedings of the 1st ACM Symposium on Cloud Computing, SoCC 2010, New York, NY, USA (2010)Google Scholar
  8. 8.
    Eclipse Foundation. Eclipselink/development/indigo/multi-tenancy (October 2012)Google Scholar
  9. 9.
    Fehling, C., Leymann, F., Mietzner, R.: A framework for optimized distribution of tenants in cloud applications. In: 2010 IEEE 3rd International Conference on Cloud Computing, CLOUD (2010)Google Scholar
  10. 10.
    Google. Google Cloud Platform (November 2012),
  11. 11.
    Guo, C.J., Sun, W., Huang, Y., Wang, Z.H., Gao, B.: A framework for native multi-tenancy application development and management. In: E-Commerce Technology and the 4th IEEE International Conference on Enterprise Computing, E-Commerce, and E-Services (2007)Google Scholar
  12. 12.
    Herndon, B., Smith, P., Roderick, L., Zamost, E., Anderson, J., Makhija, V., Herndon, B., Smith, P., Zamost, E., Anderson, J.: Vmmark: A scalable benchmark for virtualized systems. Tech. rep., VMware (2006)Google Scholar
  13. 13.
    Huang, S., Huang, J., Dai, J., Xie, T., Huang, B.: The hibench benchmark suite: Characterization of the mapreduce-based data analysis. In: ICDE Workshops (2010)Google Scholar
  14. 14.
    Islam, S., Lee, K., Fekete, A., Liu, A.: How a consumer can measure elasticity for cloud platforms. In: Proceedings of the Third Joint WOSP/SIPEW International Conference on Performance Engineering, New York, NY, USA (2012)Google Scholar
  15. 15.
    Kiefer, T., Schlegel, B., Lehner, W.: MulTe: A multi-tenancy database benchmark framework. In: Nambiar, R., Poess, M. (eds.) TPCTC 2012. LNCS, vol. 7755, pp. 92–107. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  16. 16.
    Koziolek, H.: Towards an architectural style for multi-tenant software applications. In: Proc. Software Engineering (SE 2010). LNI, vol. 159 (February 2010)Google Scholar
  17. 17.
    Koziolek, H.: The sposad architectural style for multi-tenant software applications. In: Proc. 9th Working IEEE/IFIP Conf. on Software Architecture (WICSA 2011), Workshop on Architecting Cloud Computing Applications and Systems (July 2011)Google Scholar
  18. 18.
    Krebs, R., Momm, C., Kounev, S.: Architectural Concerns in Multi-Tenant SaaS Applications. In: Proceedings of the 2nd International Conference on Cloud Computing and Services Science, CLOSER 2012 (2012)Google Scholar
  19. 19.
    Mell, P., Grance, T.: The NIST definition of cloud computing. digital (2011)Google Scholar
  20. 20.
    Momm, C., Krebs, R.: A Qualitative Discussion of Different Approaches for Implementing Multi-Tenant SaaS Offerings. In: Proceedings of Software Engineering 2011 (SE 2011), Workshop (ESoSyM 2011) (2011)Google Scholar
  21. 21.
    Natis, Y.: Gartner reference model for elasticity and multitenancy. Gartner report, Gartner (June 2012)Google Scholar
  22. 22.
    Packman, E., Taylor, P., Rachitsky, L., Rejali, S., Power, S., Rae, I., Koffler, D.: Bitcurrent: Cloud comuting performance. Tech. rep., bitcurrent (June 2010)Google Scholar
  23. 23.
    SAP AG. SAP NetWeaver Cloud (November 2012),
  24. 24.
    Smith, D.: Hype cycle for cloud computing. Tech. rep., Gartner, ID Number: G00214915 (July 2011)Google Scholar
  25. 25.
    Spec. Specjenterprise2010, (November 2012),
  26. 26.
    Wang, W., Huang, X., Qin, X., Zhang, W., Wei, J., Zhong, H.: Application-level cpu consumption estimation: Towards performance isolation of multi-tenancy web applications. In: IEEE CLOUD (2012)Google Scholar
  27. 27.
    Wang, Z.H., Guo, C.J., Gao, B., Sun, W., Zhang, Z., An, W.H.: A study and performance evaluation of the multi-tenant data tier design patterns for service oriented computing. In: IEEE International Conference on e-Business Engineering, ICEBE 2008 (2008)Google Scholar
  28. 28.
    Weissman, C.D., Bobrowski, S.: The design of the multitenant Internet application development platform. In: Proceedings of the 35th SIGMOD International Conference on Management of Data, SIGMOD 2009. ACM (2009)Google Scholar
  29. 29.
    Zhang, Y., Wang, Z., Gao, B., Guo, C., Sun, W., Li, X.: An effective heuristic for on-line tenant placement problem in saas. In: IEEE International Conference on Web Services (2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Rouven Krebs
    • 1
  • Alexander Wert
    • 2
  • Samuel Kounev
    • 2
  1. 1.Applied ResearchSAP AGWalldorfGermany
  2. 2.IPDKarlsruhe Institute of TechnologyKarlsruheGermany

Personalised recommendations