Skip to main content
View expanded cover

Technology Conference on Performance Evaluation and Benchmarking

TPCTC 2021: Performance Evaluation and Benchmarking pp 81–97Cite as

Orchestrating DBMS Benchmarking in the Cloud with Kubernetes

Part of the Lecture Notes in Computer Science book series (LNPSE,volume 13169)

Abstract

Containerization has become a common practise in software provisioning. Kubernetes (K8s) is useful in deploying containers in clusters, in managing their lifecycle, in scheduling and resource allocation. The benchmarking process requires the interaction of various components. We propose a way to organize benchmarking in the Cloud by looking at typical components in the process and ask if they could be managed by K8s as containerized Microservices. We aim at scalability for the process, parallelized execution and minimized traffic I/O from and into the Cloud. This supports planning a series of experiments to investigate a high-dimensional parameter space and avoiding complex installations. This also provides a way for Cross-Cloud comparison.

In this article we discuss 1. how objects of K8s can match components of a benchmarking process, 2. how to orchestrate the benchmarking workflow in K8s. We also present an implementation. We show this approach is feasible, relevant, portable and scalable by designing and inspecting a basic profiling benchmark on TPC-DS data handled by 13 DBMS at two private Clouds and five commercial Cloud providers.

Keywords

  • Database Management Systems
  • Performance Evaluation
  • Benchmarking
  • Virtualization
  • Docker
  • Cloud-based systems
  • Kubernetes
  • Microservices
  • Tools
  • Amazon Web Services
  • Google Cloud Platform
  • IBM Cloud
  • Microsoft Azure
  • Oracle Cloud Infrastructure

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-94437-7_6
  • Chapter length: 17 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   44.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-94437-7
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   59.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.
Fig. 7.

Notes

  1. 1.

    https://github.com/kubernetes/kubernetes.

  2. 2.

    https://aws.amazon.com/de/kubernetes/.

  3. 3.

    https://cloud.google.com/kubernetes-engine.

  4. 4.

    https://www.ibm.com/cloud/kubernetes-service.

  5. 5.

    https://azure.microsoft.com/de-de/services/kubernetes-service/.

  6. 6.

    https://www.oracle.com/cloud-native/container-engine-kubernetes/.

  7. 7.

    https://www.docker.com/products/kubernetes.

  8. 8.

    https://minikube.sigs.k8s.io/docs/.

  9. 9.

    https://kubernetes.io/docs/home/.

  10. 10.

    https://github.com/Beuth-Erdelt/DBMS-Benchmarker.

  11. 11.

    https://github.com/Beuth-Erdelt/Benchmark-Experiment-Host-Manager.

  12. 12.

    https://prometheus.io/.

  13. 13.

    https://github.com/google/cadvisor.

References

  1. IEEE Guide for Terms and Concepts in Intelligent Process Automation. In: IEEE Std. 2755–2017, pp. 1–16 (2017)

    Google Scholar 

  2. Abedjan, Z., Golab, L., Naumann, F.: Profiling relational data: a survey. VLDB J. 24(4), 557–581 (2015). https://doi.org/10.1007/s00778-015-0389-y

    CrossRef  Google Scholar 

  3. Avula, R.N., Zou, C.: Performance evaluation of TPC-C benchmark on various cloud providers. In: 2020 11th IEEE Annual Ubiquitous Computing, Electronics Mobile Communication Conference (UEMCON), pp. 0226–0233 (2020)

    Google Scholar 

  4. Bachiega, N.G., Souza, P.S.L., Bruschi, S.M., de Souza, S.D.R.: Container-based performance evaluation: a survey and challenges. In: 2018 IEEE International Conference on Cloud Engineering (IC2E), pp. 398–403 (2018)

    Google Scholar 

  5. Baur, D., Seybold, D., Griesinger, F., Tsitsipas, A., Hauser, C.B., Domaschka, J.: Cloud orchestration features: are tools fit for purpose? In: 2015 IEEE/ACM 8th International Conference on Utility and Cloud Computing (UCC), pp. 95–101 (2015)

    Google Scholar 

  6. Bermbach, D., Kuhlenkamp, J., Dey, A., Sakr, S., Nambiar, R.: Towards an extensible middleware for database benchmarking. In: Nambiar, R., Poess, M. (eds.) TPCTC 2014. LNCS, vol. 8904, pp. 82–96. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-15350-6_6

    CrossRef  Google Scholar 

  7. Brent, L., Fekete, A.: A versatile framework for painless benchmarking of database management systems. In: Chang, L., Gan, J., Cao, X. (eds.) ADC 2019. LNCS, vol. 11393, pp. 45–56. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-12079-5_4

    CrossRef  Google Scholar 

  8. Dewi, L.P., Noertjahyana, A., Palit, H.N., Yedutun, K.: Server scalability using kubernetes. In: 2019 4th Technology Innovation Management and Engineering Science International Conference (TIMES-iCON), pp. 1–4 (2019)

    Google Scholar 

  9. Difallah, D.E., Pavlo, A., Curino, C., Cudre-Mauroux, P.: OLTP-Bench: an extensible testbed for benchmarking relational databases. In: Proceedings of the VLDB Endow. 7, Dezember, Nr. 4, pp. 277–288 (2013). ISSN 2150–8097

    Google Scholar 

  10. Erdelt, P.K.: A framework for supporting repetition and evaluation in the process of cloud-based DBMS performance benchmarking. In: Nambiar, R., Poess, M. (eds.) TPCTC 2020. LNCS, vol. 12752, pp. 75–92. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-84924-5_6

    CrossRef  Google Scholar 

  11. Gray, J.: Benchmark Handbook: For Database and Transaction Processing Systems. Morgan Kaufmann Publishers Inc., San Francisco (1992). ISBN 1558601597

    Google Scholar 

  12. Jamshidi, P., Pahl, C., Mendonca, N.C., Lewis, J., Tilkov, S.: Microservices: the journey so far and challenges ahead. IEEE Softw. 35, 24–35 (2018). ISSN 1937–4194

    Google Scholar 

  13. Papadopoulos, A.V.: Methodological principles for reproducible performance evaluation in cloud computing. IEEE Trans. Softw. Eng. 1 (2019)

    Google Scholar 

  14. Pereira Ferreira, A., Sinnott, R.: A performance evaluation of containers running on managed Kubernetes services. In: 2019 IEEE International Conference on Cloud Computing Technology and Science (CloudCom), 199–208 (2019)

    Google Scholar 

  15. Seybold, D., Domaschka, J.: Is distributed database evaluation cloud-ready? In: Kirikova, M., et al. (eds.) ADBIS 2017. CCIS, vol. 767, pp. 100–108. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-67162-8_12

    CrossRef  Google Scholar 

  16. Seybold, D., Wesner, S., Domaschka, J.: King louie: reproducible availability benchmarking of cloud-hosted DBMS. In: Proceedings of the 35th Annual ACM Symposium on Applied Computing 2020 (SAC 2020), pp. 144–153. Association for Computing Machinery, New York (2020). ISBN 9781450368667

    Google Scholar 

  17. Tan, J., et al.: Choosing a cloud DBMS: architectures and tradeoffs. Proc. VLDB Endow. 12, 2170—2182 (2019). ISSN 2150–8097

    Google Scholar 

  18. The Kubernetes Authors: What is Kubernetes? https://kubernetes.io/docs/concepts/overview/what-is-kubernetes. Accessed 8 Apr 2021

  19. Thurgood, B., Lennon, R.G.: Cloud computing with kubernetes cluster elastic scaling. In: Proceedings of the 3rd International Conference on Future Networks and Distributed Systems, (ICFNDS 2019). Association for Computing Machinery, New York (2019). ISBN 9781450371636

    Google Scholar 

  20. Tomarchio, O., Calcaterra, D., Modica, G.Di.: Cloud resource orchestration in the multi-cloud landscape: a systematic review of existing frameworks. J. Cloud Comput. 9(1), 1–24 (2020). https://doi.org/10.1186/s13677-020-00194-7

  21. Transaction Processing Performance Council: TPCx-AI Homepage. http://tpc.org/tpcx-ai/default5.asp. Accessed 5 Aug 2021

  22. Weerasiri, D., Barukh, M.C., Benatallah, B., Sheng, Q.Z., Ranjan, R.: A taxonomy and survey of cloud resource orchestration techniques. ACM Comput. Surv. (CSUR) 50, 1–41 (2017)

    CrossRef  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Beuth Hochschule fuer Technik Berlin, Luxemburger Strasse 10, 13353, Berlin, Germany

    Patrick K. Erdelt

Authors
  1. Patrick K. Erdelt
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Patrick K. Erdelt .

Editor information

Editors and Affiliations

  1. Advanced Micro Devices Inc., Santa Clara, CA, USA

    Raghunath Nambiar

  2. Oracle Corporation, Redwood Shores, CA, USA

    Meikel Poess

Rights and permissions

Reprints and Permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Erdelt, P.K. (2022). Orchestrating DBMS Benchmarking in the Cloud with Kubernetes. In: Nambiar, R., Poess, M. (eds) Performance Evaluation and Benchmarking. TPCTC 2021. Lecture Notes in Computer Science(), vol 13169. Springer, Cham. https://doi.org/10.1007/978-3-030-94437-7_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-94437-7_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-94436-0

  • Online ISBN: 978-3-030-94437-7

  • eBook Packages: Computer ScienceComputer Science (R0)