Cloud migration allows organizations to benefit from reduced operational costs, improved flexibility, and greater scalability, and enables them to focus on core business goals. However, it also has the flip side of reduced visibility. Enterprises considering migration of their IT systems to the cloud only have a black box view of the offered infrastructure. While information about server pricing and specification is publicly available, there is limited information about cloud infrastructure performance. Comparison of alternative cloud infrastructure offerings based only on price and server specification is difficult because cloud vendors use heterogeneous hardware resources, offer different server configurations, apply different pricing models and use different virtualization techniques to provision them. Benchmarking the performance of software systems deployed on the top of the black box cloud infrastructure offers one way to evaluate the performance of available cloud server alternatives. However, this process can be complex, time-consuming and expensive, and cloud consumers can greatly benefit from tools that can automate it. Smart CloudBench is a generic framework and system that offers automated, on-demand, real-time and customized benchmarking of software systems deployed on cloud infrastructure. It provides greater visibility and insight into the run-time behavior of cloud infrastructure, helping consumers to compare and contrast available offerings during the initial cloud selection phase, and monitor performance for service quality assurance during the subsequent cloud consumption phase. In this paper, we first discuss the rationale behind our approach for benchmarking the black box cloud infrastructure. Then, we propose a generic architecture for benchmarking representative applications on the heterogeneous cloud infrastructure and describe the Smart CloudBench benchmarking workflow. We also present simple use case scenarios that highlight the need for tools such as Smart CloudBench.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Some example representative applications include TPC-W for a transactional e-commerce web application (TPCW 2003) and Media Streaming benchmark application for media streaming applications such as Netflix or Yuku (Ferdman et.al. 2012)
Baruwal Chhetri, M, Chichin, S., Bao Vo, Q., & Kowalczyk, R. (2013a). Smart Cloud Broker: Finding your home in the clouds. In Automated Software Engineering (ASE), 2013 IEEE/ACM 28th International Conference on (pp. 698–701). IEEE.
Baruwal Chhetri, M., Chichin, S., Vo, Q. B., & Kowalczyk, R. (2013b). Smart CloudBench-- Automated Performance Benchmarking of the Cloud. In Cloud Computing (CLOUD), 2013 I.E. Sixth International Conference on (pp. 414–421). IEEE.
Baruwal Chhetri, M., Chichin, S., Vo, Q.B., & Kowalczyk, R. (2014). Smart CloudMonitor-Providing Visibility into Performance of Black-Box Clouds. In Cloud Computing (CLOUD), 2014 I.E. 7th International Conference on (pp. 777–784). IEEE.
Ferdman, M., Adileh, A., Kocberber, O., Volos, S., Alisafaee, M., Jevdjic, D., & Falsafi, B. (2012). Clearing the clouds: a study of emerging scale-out workloads on modern hardware. In ACM SIGARCH Computer Architecture News (Vol. 40, No. 1, pp. 37–48). ACM.
Folkerts, E., Alexandrov, A., Sachs, K., Iosup, A., Markl, V., & Tosun, C. (2013). Benchmarking in the cloud: What it should, can, and cannot be. In Selected Topics in Performance Evaluation and Benchmarking (pp. 173–188). Springer Berlin Heidelberg.
Gmach, D., Rolia, J., & Cherkasova, L. (2012, April). Comparing efficiency and costs of cloud computing models. In Network Operations and Management Symposium (NOMS), 2012 I.E. (pp. 647–650). IEEE.
Iosup, A., Prodan, R., & Epema, D. (2013, May). IaaS cloud benchmarking: approaches, challenges, and experience. In HotTopiCS (pp. 1–2).
Lenk, A., Menzel, M., Lipsky, J., Tai, S., & Offermann, P. (2011). What are you paying for? performance benchmarking for infrastructure-as-a-service offerings. In Cloud Computing (CLOUD), 2011 I.E. International Conference on (pp. 484–491). IEEE.
Li, A, Yang, X., Kandula, S. and Zhang, M. (2010) CloudCmp: comparing public cloud providers. In Proceedings of the 10th Annual Conference on Internet Measurement.
Li, A., Yang, X., Kandula, S., Yang, X. and Zhang, M. (2011) CloudProphet: towards application performance prediction in cloud. In ACM SIGCOMM Computer Communication Review (Vol. 41, No. 4, pp. 426–427). ACM.
Luo, C., Zhan, J., Jia, Z., Wang, L., Lu, G., Zhang, L., & Sun, N. (2012). Cloudrank-d: benchmarking and ranking cloud computing systems for data processing applications. Frontiers of Computer Science, 6(4), 347–362.
Nielsen J. (1994). Usability Engineering. Elsevier.
Oliveira, F., Eilam, T., Kalantar, M., & Rosenberg, F. (2012). Semantically-Rich Composition of Virtual Images. In Cloud Computing (CLOUD), 2012 I.E. 5th International Conference on (pp. 277–284). IEEE.
RightScale (n.d) 2014 State of the Cloud Report. Retrieved from http://assets.rightscale.com/uploads/pdfs/RightScale-2014-State-of-the-Cloud-Report.pdf
Zhao, L., Liu, A., & Keung, J. (2010). Evaluating cloud platform architecture with the care framework. In Software Engineering Conference (APSEC), 2010 17th Asia Pacific (pp. 60–69). IEEE.
About this article
Cite this article
Baruwal Chhetri, M., Chichin, S., Vo, Q.B. et al. Smart CloudBench—A framework for evaluating cloud infrastructure performance. Inf Syst Front 18, 413–428 (2016). https://doi.org/10.1007/s10796-015-9557-2
- Cloud performance benchmarking
- Automated benchmarking
- Cloud bench