Benchmarking Heterogeneous Cloud Functions

  • Maciej MalawskiEmail author
  • Kamil Figiela
  • Adam Gajek
  • Adam Zima
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10659)


Cloud Functions, often called Function-as-a-Service (FaaS), pioneered by AWS Lambda, are an increasingly popular method of running distributed applications. As in other cloud offerings, cloud functions are heterogeneous, due to different underlying hardware, runtime systems, as well as resource management and billing models. In this paper, we focus on performance evaluation of cloud functions, taking into account heterogeneity aspects. We developed a cloud function benchmarking framework, consisting of one suite based on Serverless Framework, and one based on HyperFlow. We deployed the CPU-intensive benchmarks: Mersenne Twister and Linpack, and evaluated all the major cloud function providers: AWS Lambda, Azure Functions, Google Cloud Functions and IBM OpenWhisk. We make our results available online and continuously updated. We report on the initial results of the performance evaluation and we discuss the discovered insights on the resource allocation policies.


Cloud computing FaaS Cloud functions Performance evaluation 



This work was supported by the National Science Centre, Poland, grant 2016/21/B/ST6/01497.


  1. 1.
    Balis, B.: HyperFlow: a model of computation, programming approach and enactment engine for complex distributed workflows. Future Gener. Comput. Syst. 55, 147–162 (2016)CrossRefGoogle Scholar
  2. 2.
    Iosup, A., Ostermann, S., Yigitbasi, N., Prodan, R., Fahringer, T., Epema, D.: Performance analysis of cloud computing services for many-tasks scientific computing. IEEE Trans. Parallel Distrib. Syst. 22(6), 931–945 (2011)CrossRefGoogle Scholar
  3. 3.
    Leitner, P., Cito, J.: Patterns in the chaos - a study of performance variation and predictability in public IaaS clouds. ACM Trans. Internet Techn. 16(3), 15:1–15:23 (2016). CrossRefGoogle Scholar
  4. 4.
    Leitner, P., Scheuner, J.: Bursting with possibilities - an empirical study of credit-based bursting cloud instance types. In: 8th IEEE/ACM International Conference on Utility and Cloud Computing, UCC 2015, Limassol, Cyprus, 7–10 December 2015, pp. 227–236 (2015).
  5. 5.
    Malawski, M.: Towards serverless execution of scientific workflows - HyperFlow case study. In: WORKS 2016 Workshop, Workflows in Support of Large-Scale Science, in Conjunction with SC 2016 Conference., Salt Lake City, November 2016Google Scholar
  6. 6.
    Malawski, M., Kuzniar, M., Wojcik, P., Bubak, M.: How to use Google app engine for free computing. IEEE Internet Comput. 17(1), 50–59 (2013)CrossRefGoogle Scholar
  7. 7.
    McGrath, M.G., Short, J., Ennis, S., Judson, B., Brenner, P.R.: Cloud event programming paradigms: applications and analysis. In: 9th IEEE International Conference on Cloud Computing, CLOUD 2016, San Francisco, CA, USA, 27 June – 2 July 2016, pp. 400–406. IEEE Computer Society (2016)Google Scholar
  8. 8.
    Prodan, R., Sperk, M., Ostermann, S.: Evaluating high-performance computing on Google app engine. IEEE Softw. 29(2), 52–58 (2012)CrossRefGoogle Scholar
  9. 9.
    Spillner, J.: Snafu: Function-as-a-Service (FaaS) runtime design and implementation. CoRR abs/1703.07562 (2017).
  10. 10.
    Villamizar, M., Garces, O., Ochoa, L., Castro, H., Salamanca, L., Verano, M., Casallas, R., Gil, S., Valencia, C., Zambrano, A., Lang, M.: Infrastructure cost comparison of running web applications in the cloud using AWS lambda and monolithic and microservice architectures. In: 2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), pp. 179–182, May 2016Google Scholar
  11. 11.
    Wagner, B., Sood, A.: Economics of resilient cloud services. In: 1st IEEE International Workshop on Cyber Resilience Economics, August 2016.

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Computer ScienceAGH University of Science and TechnologyKrakowPoland

Personalised recommendations