Self-scalable Benchmarking as a Service with Automatic Saturation Detection

  • Alain Tchana
  • Bruno Dillenseger
  • Noel De Palma
  • Xavier Etchevers
  • Jean-Marc Vincent
  • Nabila Salmi
  • Ahmed Harbaoui
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8275)


Software applications providers have always been required to perform load testing prior to launching new applications. This crucial test phase is expensive in human and hardware terms, and the solutions generally used would benefit from further development. In particular, designing an appropriate load profile to stress an application is difficult and must be done carefully to avoid skewed testing. In addition, static testing platforms are exceedingly complex to set up. New opportunities to ease load testing solutions are becoming available thanks to cloud computing. This paper describes a Benchmark-as-a-Service platform based on: (i) intelligent generation of traffic to the benched application without inducing thrashing (avoiding predefined load profiles), (ii) a virtualized and self-scalable load injection system. This platform was found to reduce the cost of testing by 50% compared to more commonly used solutions. It was experimented on the reference JEE benchmark RUBiS. This involved detecting bottleneck tiers.


Benchmarking as a service Saturation detection Cloud 


  1. 1.
    Amza, C., Cecchet, E., Chanda, A., Cox, A.L., Elnikety, S., Gil, R., Marguerite, J., Rajamani, K., Zwaenepoel, W.: Specification and implementation of dynamic web site benchmarks. In: IEEE Annual Workshop on Workload Characterization, Austin, TX, USA, pp. 3–13 (2002)Google Scholar
  2. 2.
    Dillenseger, B.: CLIF, a framework based on fractal for flexible, distributed load testing. In: Annals of Telecommunications, vol. 64(1-2), pp. 101–120. Springer, Paris (2009)Google Scholar
  3. 3.
    Bruneton, E., Coupaye, T., Leclercq, M., Quema, V., Stefani, J.-B.: An Open Component Model and Its Support in Java. In: Crnković, I., Stafford, J.A., Schmidt, H.W., Wallnau, K. (eds.) CBSE 2004. LNCS, vol. 3054, pp. 7–22. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  4. 4.
    Harbaoui, A., Salmi, N., Dillenseger, B., Vincent, J.: Introducing Queuing Network-Based Performance Awareness in Autonomic Systems. In: Proceedings of the International Conference on Autonomic and Autonomous Systems, Cancun, Mexico, pp. 7–12 (2010)Google Scholar
  5. 5.
    Kleinrock, L.: Queueing Systems. Wiley-Interscience, New York (1975) ISBN 0471491101Google Scholar
  6. 6.
    Stewart, W.: Introduction to the Numerical Solution of Markov Chains. Princeton University Press, Princeton (1994) ISBN 0691036993Google Scholar
  7. 7.
    Jain, R.K.: The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and modelling. John Wiley and Sons, Inc., Canada (1991) ISBN 0471503363Google Scholar
  8. 8.
    Oracle, Java Message Service, (October 2012),
  9. 9.
    Kephart, J.O., Chess, D.M.: The Vision of Autonomic Computing. Computer 36(1), 41–50 (2003)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Grid’5000: a scientific instrument designed to support experiment-driven research (October 2012),
  11. 11.
    Openstack web site (October 2012),
  12. 12.
    Amazon Web Services, Amazon EC2 auto-scaling functions (October 2012),
  13. 13.
    Simic, B.: The performance of web applications: Customers are won or lost in one second. A. R. Library (2008)Google Scholar
  14. 14.
    Wang, Q., Malkowski, S., Jayasinghe, D., Xiong, P., Pu, C., Kanemasa, Y., Kawaba, M., Harada, L.: The impact of soft resource allocation on n-tier application scalability. In: Proceedings of the 2011 IEEE International Parallel & Distributed Processing Symposium, Washington, DC, USA, pp. 1034–1045 (2011)Google Scholar
  15. 15.
    Rolls, D., Joslin, C., Scholz, S.-B.: Unibench: a tool for automated and collaborative benchmarking. In: Proceedings of the IEEE International Conference on Program Comprehension, Braga, Portugal, pp. 50–51 (2010)Google Scholar
  16. 16.
    Almeida, R., Vieira, M.: Benchmarking the resilience of self-adaptive software systems: perspectives and challenges. In: Proceedings of the International Symposium on Software Engineering for Adaptive and Self-Managing Systems, Waikiki, Honolulu, HI, USA, pp. 190–195 (2011)Google Scholar
  17. 17.
    El-Refaey, M.A., Rizkaa, M.A.: CloudGauge: a dynamic cloud and virtualization benchmarking suite. In: Proceedings of the IEEE International Workshops on Enabling Technologies: Infrastructures for Collaborative Enterprises, Larissa, Greece, pp. 66–75 (2010)Google Scholar
  18. 18.
    Jin, H., Cao, W., Yuan, P., Xie, X.: VSCBenchmark: benchmark for dynamic server performance of virtualization technology. In: Proceedings of the International Forum on Next-Generation Multicore/Manycore Technologies, Cairo, Egypt, pp. 1–8 (2008)Google Scholar
  19. 19.
    Makhija, V., Herndon, B., Smith, P., Roderick, L., Zamost, E., Anderson, J.: VMmark: a scalable benchmark for virtualized systems, Technical Report VMware-TR-2006-002, Palo Alto, CA, USA (September 2006)Google Scholar
  20. 20.
    Jayasinghe, D., Swint, G.S., Malkowski, S., Li, J., Park, J., Pu, C.: Expertus: A Generator Approach to Automate Performance Testing in IaaS Clouds. In: Proceedings of the IEEE International Conference on Cloud Computing, Honolulu, HI, USA, pp. 115–122 (June 2012)Google Scholar
  21. 21.
    BlazeMeter, Dependability benchmarking project (October 2012),
  22. 22.
    The Apache Software Foundation, Apache JMeter (October 2012),
  23. 23.
    Neotys, NeoLoad: load test all web and mobile applications (October 2012),

Copyright information

© IFIP International Federation for Information Processing 2013

Authors and Affiliations

  • Alain Tchana
    • 1
  • Bruno Dillenseger
    • 2
  • Noel De Palma
    • 1
  • Xavier Etchevers
    • 2
  • Jean-Marc Vincent
    • 1
  • Nabila Salmi
    • 2
  • Ahmed Harbaoui
    • 2
  1. 1.LIGJoseph Fourier UniversityGrenobleFrance
  2. 2.Orange LabsGrenobleFrance

Personalised recommendations