Skip to main content
SpringerLink
Log in

Main FaaS Providers Behavior Under High Concurrency: An Evaluation with Orama Framework Distributed Architecture

  • Original Research
  • Published:
SN Computer Science Aims and scope Submit manuscript

Abstract

With the increasing adoption of Function-as-a-Service (FaaS) oriented solutions, there is a growing interest in tools that facilitate benchmarking in these environments. The Orama framework has emerged as a standout solution, offering high configurability and scalability for provisioning, running benchmarks, and performing statistical analysis. This research presents the use of the distributed architecture of the Orama framework in evaluating the main public FaaS providers in situations of high competition. The experimental results demonstrate the framework’s effectiveness in orchestrating load tests with high levels of concurrency. Notably, the study reveals specific characteristics of the providers involved, with Alibaba Function showing excellent performance, registering the lowest average execution time and remaining free of failures. Google Cloud Function and AWS Lambda provide intermediate results at average runtime and report some failures. Meanwhile, Azure Function performs worst in average execution time and cold start. A deeper investigation into the Alibaba Function provider was conducted to verify if its performance maintains a more complex usage scenario, in this case, a use case merging FaaS and Database-as-a-Service.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Gartner: Gartner forecasts worldwide public cloud end-user spending to reach nearly 600 billion in 2023 (Online); 01-Jan-2023; https://tinyurl.com/3z72zebw; 2022.

  2. MELL P, Grance T. The NIST definition of cloud computing. National Institute of Standards and Tecnology; 2011.

  3. Duan Y, Fu G, Zhou N, Sun X, Narendra NC, Hu B. Everything as a service (XaaS) on the cloud: origins. Curr Future Trends. 2015;00:621–8.

    Google Scholar 

  4. Schleier-Smith J, Sreekanti V, Khandelwal A, Carreira J, Yadwadkar NJ, Popa RA, Gonzalez JE, Stoica I, Patterson DA. What serverless computing is and should become: the next phase of cloud computing. ACM. 2021;64(5):76–84.

    Article  Google Scholar 

  5. Copik M, Kwasniewski G, Besta M, Podstawski M, Hoefler T. SeBS: A serverless benchmark suite for function-as-a-service computing, 2021.

  6. Somu N, Daw N, Bellur U, Kulkarni P. Panopticon: a comprehensive benchmarking tool for serverless applications. In: 2020 COMSNETS; 2020. p. 144–51.

  7. Maissen P, Felber P, Kropf P, Schiavoni V. Faasdom. In: Proceedings of the 14th ACM international conference on distributed and event-based systems; 2020.

  8. Grambow M, Pfandzelter T, Burchard L, Schubert C, Zhao M, Bermbach D. BeFaaS: an application-centric benchmarking framework for FaaS platforms; 2021.

  9. Carvalho L, Araujo A. Orama: A benchmark framework for function-as-a-service. In: Proceedings of the 12th CLOSER. SciTePress, Online; 2022. INSTICC. p. 313–22.

  10. de Carvalho LR, Kamienski B, Araujo A. FaaS Benchmarking over Orama Framework’s Distributed Architecture. In: Proceedings of the 13th international conference on cloud computing and services science, vol 1: CLOSER. SciTePress, Prague; 2023. INSTICC. p. 67–77.

  11. Motta, MADC, Reboucas De Carvalho, L, Rosa, MJF, Favacho De Araujo, AP. Comparison of FaaS platform performance in private clouds. In: Proceedings of the 12th CLOSER. SciTePress, Online; 2022. INSTICC. p. 109–20.

  12. HashiCorp: Terraform: Write, Plan, Apply (Online); 11-Aug-2021. https://www.terraform.io; 2021. https://www.terraform.io/. Accessed 2021-05-31.

  13. Gupta PR, Taneja S, Datt A. Using heat and ceilometer for providing autoscaling in openstack. JIMS8I Int J Inform Commun Comput Technol. 2014;2(2):84–9.

    Google Scholar 

  14. Wittig M, Wittig A. Amazon web services in action. New York: Simon and Schuster; 2018.

    Google Scholar 

  15. Carvalho LR, Araujo APF. Performance comparison of terraform and cloudify as multicloud orchestrators. In: 2020 20th IEEE/ACM CCGRID; 2020. p. 380–9.

  16. Erinle B. Performance testing with JMeter 2.9. Packt Publishing Ltd, United Kingdom; 2013. ISBN: 9781782165842.

  17. Sax MJ. In: Sakr, S., Zomaya, A. (eds.) Apache Kafka. Cham: Springer; 2018. p. 1–8. ISBN: 978-3-319-63962-8.

  18. Ibrahim MH, Sayagh M, Hassan AE. A study of how docker compose is used to compose multi-component systems. Empir Softw Eng. 2021;26(6):128.

    Article  Google Scholar 

  19. Jain R. The art of computer systems: Techniques for experimental design, measurement, simulation, and modeling. London: Wiley; 1991. ISBN:13.978-0471503361.

  20. Hsu H, Lachenbruch PA. Paired \(t\) test. London: Wiley; 2014.

    Book  Google Scholar 

  21. Back T, Andrikopoulos V. Using a microbenchmark to compare function as a service solutions. In: ECSOCC. London: Springer; 2018. p. 146–60.

  22. Kuhlenkamp J, Werner S, Borges MC, El Tal K, Tai S. An evaluation of FAAS platforms as a foundation for serverless big data processing. In: Proceedings of the 12th IEEE/ACM. UCC’19. ACM, NY, USA; 2019. p. 1–9.

  23. Barcelona-Pons D, García-López P. Benchmarking parallelism in FaaS platforms. Fut Gener Comput Syst. 2021;124:268–84.

    Article  Google Scholar 

  24. Wen J, Liu Y, Chen Z, Chen J, Ma Y. Characterizing commodity serverless computing platforms; 2020. https://doi.org/10.48550/ARXIV.2012.00992.

Download references

Acknowledgements

Partially funded by CNPq/AWS 64/2022 BioCloud2 (n\(^{\textrm{o}}\) 421828/2022-6).

Author information

Author notes

  1. Leonardo Rebouças de Carvalho, Bruno Kamienski and Aleteia Araujo have contributed equally to this work.

Authors and Affiliations

  1. Department of Computer Science, University of Brasília, Campus Darcy Ribeiro, Brasilia, Federal District, 70910-000, Brazil

    Leonardo Rebouças de Carvalho, Bruno Kamienski & Aleteia Araujo

Authors
  1. Leonardo Rebouças de Carvalho
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bruno Kamienski
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Aleteia Araujo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Leonardo Rebouças de Carvalho.

Ethics declarations

Conflict of interest

The authors declare that there is no Conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article is part of the topical collection “Recent Trends on Cloud Computing and Services Science” guest edited by Claus Pahl and Maarten van Steen.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

de Carvalho, L.R., Kamienski, B. & Araujo, A. Main FaaS Providers Behavior Under High Concurrency: An Evaluation with Orama Framework Distributed Architecture. SN COMPUT. SCI. 5, 541 (2024). https://doi.org/10.1007/s42979-024-02895-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-024-02895-1

Keywords

Search

Navigation