Smart CloudBench - Test Drive the Cloud Before You Buy

  • Mohan Baruwal Chhetri
  • Sergei Chichin
  • Quoc Bao Vo
  • Ryszard Kowalczyk
Conference paper
Part of the Lecture Notes in Business Information Processing book series (LNBIP, volume 177)


In recent years there has been an exponential growth in the number of vendors offering Infrastructure-as-a-Service (IaaS), with a corresponding increase in the number of enterprises looking to migrate some, or all of their IT systems to the cloud. Prospective cloud consumers need to identify providers that offer resources with the most appropriate pricing and performance levels to match their specific business needs before making any migration decisions. However, no two vendors offer the same resource configurations, pricing models or provisioning models. Moreover, cloud vendors tend to use different virtualization techniques which impact the performance of the software systems running on top of their infrastructure. Since consumers only have a black-box view of the cloud, it makes the task of comparing and selecting appropriate computing resources a very complex exercise. In this paper, we present Smart CloudBench, which is a suite of software tools that allows prospective cloud consumers to test drive the cloud and make purchasing decisions based on price, specification AND performance. Cloud consumers can use Smart CloudBench for the automated, on-demand, real-time and customized benchmarking of cloud infrastructure and use the benchmarking results along with the pricing and specification information to make more informed purchasing decisions. Tests using Smart CloudBench show that the performance of higher priced servers is not necessarily better than that of lower priced ones, and it has to be tested extensively in order to substitute assumptions with facts.


Cloud infrastructure selection Performance benchmarking Automated benchmarking 



This work was partially funded by the Service Delivery and Aggregation Project within the Smart Services CRC.


  1. 1.
    Baruwal Chhetri, M., Vo, Q.B., Kowalczyk, R., Lan Do, C.: Cloud broker: helping you buy better. In: Bouguettaya, A., Hauswirth, M., Liu, L. (eds.) WISE 2011. LNCS, vol. 6997, pp. 341–342. Springer, Heidelberg (2011) CrossRefGoogle Scholar
  2. 2.
    Baruwal Chhetri, M., Chichin, S., Vo, Q.B., Kowalczyk, R.: Smart CloudBench: automated performance benchmarking of the cloud. In: Proceedings of CLOUD 2013, 27 June–2 July 2013, California (2013)Google Scholar
  3. 3.
    Baruwal Chhetri, M., Vo, Q.B., Kowalczyk, R.: Policy-based automation of SLA establishment for cloud computing services. In: Proceedings of the 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID-12), Ottawa (Canada), 13–16 May 2012Google Scholar
  4. 4.
    Baruwal Chhetri, M., Vo, Q.B., Kowalczyk, R.: AutoSLAM - a policy-driven middleware for automated SLA establishment in SOA environments. In: Proceedings of the 9th International Conference on Service Computing (SCC 2012), Honolulu, Hawaii, USA, pp. 9–16. IEEE (2012)Google Scholar
  5. 5.
    Transaction Processing Performance Council. TPC Benchmark W (Web Commerce) Specification, version 2.0r. Technical Specification (2003).
  6. 6.
    Ferdman, M., et al.: Clearing the clouds: a study of emerging scale-out workloads on modern hardware. In: The 17th International Conference on Architectural Support for Programming Languages and Operating Systems (2012)Google Scholar
  7. 7.
    Iosup, A., Prodan, R., Epema, D.: IaaS cloud benchmarking: approaches, challenges, and experience. In: Proceedings of 5th Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS) (2012)Google Scholar
  8. 8.
    Li, A., Yang, X., Kandula, S., Zhang, M.: CloudCmp: comparing public cloud providers. In: Proceedings of the 10th Annual Conference on Internet Measurement, November 2010Google Scholar
  9. 9.
    Li, A., Yang, X., Kandula, S., Yang, X., Zhang, M.: CloudProphet: towards application performance prediction in cloud. In: Proceedings of ACM SIGCOMM 2011, Toronto, pp 426–427 (2011)Google Scholar
  10. 10.
    Zhao, L., Liu, A., Keung, J.: Evaluating cloud platform architecture with the care framework. In: Proceedings of 17th APSEC, pp. 60–69 (2010)Google Scholar
  11. 11.
    Luo, C., et al.: CloudRank-D: benchmarking and ranking cloud computing systems for data processing applications. Front. Comput. Sci. 6(4), 347–362 (2012)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Mohan Baruwal Chhetri
    • 1
  • Sergei Chichin
    • 1
  • Quoc Bao Vo
    • 1
  • Ryszard Kowalczyk
    • 1
  1. 1.Faculty of Information and Communication TechnologiesSwinburne University of TechnologyMelbourneAustralia

Personalised recommendations