A Scalable Benchmark as a Service Platform

  • Alain Tchana
  • Noel De Palma
  • Ahmed El-Rheddane
  • Bruno Dillenseger
  • Xavier Etchevers
  • Ibrahim Safieddine
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7891)


Load testing has always been a crucial and expensive activity for software companies. Classical solutions are a real burden to setup statically and their cost are prohibitive in terms of human and hardware resources. Cloud computing brings new opportunities to stress application scalability as load testing solutions can be provided on demand by the cloud. This paper describes a Benchmark-as-a-Service solution that scales automatically the load injection platform and eases its setup according to load profiles. Our approach is based on: (i) the virtualization of the Benchmarking platform to enable the injector’s self-scalability, (ii) an online calibration mechanism to characterize injector capacity and impact on the benched application, (iii) a provisioning solution to scale the load injection platform sufficiently ahead of time. We also report experiments on a benchmark that shows the benefits in terms of cost and resources savings.


Benchmarking as a service Cloud 


  1. 1.
    Dillenseger, B.: CLIF, a framework based on fractal for flexible, distributed load testing. Annals of Telecommunications 64(1-2), 101–120 (2009)CrossRefGoogle Scholar
  2. 2.
    The CLIF Project, (visited on February 2013)
  3. 3.
    Bruneton, E., Coupaye, T., Leclercq, M., Quema, V., Stefani, J.-B.: An Open Component Model and Its Support in Java. In: International Symposium on Component- Based Software Engineering (CBSE), Edinburgh, UK (2004)Google Scholar
  4. 4.
    Amza, C., Cecchet, E., Chanda, A., Cox, A.L., Elnikety, S., Gil, R., Marguerite, J., Rajamani, K., Zwaenepoel, W.: Specification and implementation of dynamic web site benchmarks. In: IEEE Annual Workshop on Workload Characterization, Austin, TX, USA (2002)Google Scholar
  5. 5.
    Harbaoui, A., Salmi, N., Dillenseger, B., Vincent, J.: Introducing queuing network-based performance awareness in autonomic systems. In: ICAS, Cancun, Mexico (2010)Google Scholar
  6. 6.
    Tchana, A., Temate, S., Broto, L., Hagimont, D.: Autonomic resource allocation in a J2EE cluster. In: Utility and Cloud Computing, Chennai, India (2010)Google Scholar
  7. 7.
    Amazon Web Services, Amazon EC2 auto-scaling functions, (visited on February 2013)
  8. 8.
    Righscale web site, (visited on February 2013)
  9. 9.
    Openstack web site, (visited on February 2013)
  10. 10.
    Grid’5000 web site, (visited on February 2013)
  11. 11.
    The Apache Software Foundation, Apache JMeter, (visited on February 2013)
  12. 12.
    Rolls, D., Joslin, C., Scholz, S.-B.: Unibench: a tool for automated and collaborative benchmarking. In: ICPC, Braga, Portugal (2010)Google Scholar
  13. 13.
    Almeida, R., Vieira, M.: Benchmarking the resilience of self-adaptive software systems: perspectives and challenges. In: SEAMS, Waikiki, Honolulu, HI, USA (2011)Google Scholar
  14. 14.
    El-Refaey, M.A., Rizkaa, M.A.: CloudGauge: a dynamic cloud and virtualization benchmarking suite. In: WETICE, Larissa, Greece (2010)Google Scholar
  15. 15.
    Jin, H., Cao, W., Yuan, P., Xie, X.: VSCBenchmark: benchmark for dynamic server performance of virtualization technology. In: IFMT, Cairo, Egypt (2008)Google Scholar
  16. 16.
    Makhija, V., Herndon, B., Smith, P., Roderick, L., Zamost, E., Anderson, J.: VMmark: a scalable benchmark for virtualized systems, Technical Report VMware-TR-2006-002, Palo Alto, CA, USA (September 2006)Google Scholar
  17. 17.
    BlazeMeter,Dependability benchmarking project, (visited on February 2013)
  18. 18.
    Neotys, NeoLoad: load test all web and mobile applications, (visited on February 2013)

Copyright information

© IFIP International Federation for Information Processing 2013

Authors and Affiliations

  • Alain Tchana
    • 1
  • Noel De Palma
    • 1
  • Ahmed El-Rheddane
    • 1
  • Bruno Dillenseger
    • 2
  • Xavier Etchevers
    • 2
  • Ibrahim Safieddine
    • 1
  1. 1.LIG Laboratory (UJF/LIG)Joseph Fourier UniversityGrenobleFrance
  2. 2.Orange LabsGrenobleFrance

Personalised recommendations