Abstract
Docker and Kubernetes have revolutionized the cloud-native technology ecosystem by offering robust solutions for containerization and orchestration workflows. This combination provides unprecedented speed, scalability, and efficiency in deploying and managing applications in distributed environments. However, when scheduling complex workflows across multi-cluster Kubernetes environments, existing workflow scheduling systems often fail to provide the necessary support. Integrating workflow scheduling algorithms with multi-cluster scheduling algorithms poses a complex and challenging problem. In this paper, we present a comprehensive framework known as the Containerized Workflow Engine (CWE), specifically designed for multi-cluster Kubernetes deployments. The CWE framework employs a two-level scheduling scheme, which combines the benefits of workflow containerization and establishes seamless connections between multi-cluster scheduling algorithms and multi-cluster Kubernetes environments. By integrating workflow scheduling algorithms with Kubernetes schedulers across Kubernetes environments, the CWE framework enables efficient utilization of resources and improved overall workflow performance. Compared to the state-of-the-art Argo workflows, CWE performs better in average task pod execution time and resource utilization.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Apache airflow (2023). https://airflow.apache.org/
Argo-workflows - github (2023). https://github.com/argoproj/argo-workflows
The best free and open source container tools (2023). https://podman.io/
CWE - github (2023). https://github.com/liudy093/CWE
Develop faster. Run anywhere (2023). https://www.docker.com/
Linux man page (2023). https://linux.die.net/man/1/stress
Nextflow (2023). https://www.nextflow.io/
Our mission is to be the trusted cloud native repository for kubernetes (2023). https://goharbor.io/
Production-grade container orchestration (2023). https://kubernetes.io/
Program against your datacenter like it’s a single pool of resources (2023). https://mesos.apache.org/
Volcano - github (2023). https://github.com/volcano-sh/volcano
Adhikari, M., Amgoth, T., Srirama, S.N.: A survey on scheduling strategies for workflows in cloud environment and emerging trends. ACM Comput. Surv. (CSUR) 52(4), 1–36 (2019)
Bernstein, D.: Containers and cloud: from LXC to docker to Kubernetes. IEEE Cloud Comput. 1(3), 81–84 (2014)
Bharathi, S., Chervenak, A., Deelman, E., Mehta, G., Su, M.H., Vahi, K.: Characterization of scientific workflows. In: 2008 Third Workshop on Workflows in Support of Large-Scale Science, pp. 1–10. IEEE (2008)
Deelman, E., Gannon, D., Shields, M., Taylor, I.: Workflows and e-science: an overview of workflow system features and capabilities. Futur. Gener. Comput. Syst. 25(5), 528–540 (2009)
Hobson, T., Yildiz, O., Nicolae, B., Huang, J., Peterka, T.: Shared-memory communication for containerized workflows. In: 2021 IEEE/ACM 21st International Symposium on Cluster, Cloud and Internet Computing (CCGrid), pp. 123–132. IEEE (2021)
Klop, I.: Containerized workflow scheduling (2018)
Pahl, C.: Containerization and the PaaS cloud. IEEE Cloud Comput. 2(3), 24–31 (2015)
Shan, C., Wang, G., Xia, Y., Zhan, Y., Zhang, J.: Containerized workflow builder for Kubernetes. In: 2021 IEEE 23rd International Conference on High Performance Computing & Communications; 7th International Conference on Data Science & Systems; 19th International Conference on Smart City; 7th International Conference on Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys), pp. 685–692. IEEE (2021)
Shan, C., Xia, Y., Zhan, Y., Zhang, J.: KubeAdaptor: a docking framework for workflow containerization on Kubernetes. Futur. Gener. Comput. Syst. 148, 584–599 (2023)
Varghese, B., Buyya, R.: Next generation cloud computing: new trends and research directions. Futur. Gener. Comput. Syst. 79, 849–861 (2018)
Wang, Y.R., Huang, K.C., Wang, F.J.: Scheduling online mixed-parallel workflows of rigid tasks in heterogeneous multi-cluster environments. Futur. Gener. Comput. Syst. 60, 35–47 (2016)
Zheng, C., Tovar, B., Thain, D.: Deploying high throughput scientific workflows on container schedulers with makeflow and mesos. In: 2017 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), pp. 130–139. IEEE (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Liu, D., Xia, Y., Shan, C., Wang, G., Wang, Y. (2023). Scheduling Containerized Workflow in Multi-cluster Kubernetes. In: Chen, E., et al. Big Data. BigData 2023. Communications in Computer and Information Science, vol 2005. Springer, Singapore. https://doi.org/10.1007/978-981-99-8979-9_12
Download citation
DOI: https://doi.org/10.1007/978-981-99-8979-9_12
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8978-2
Online ISBN: 978-981-99-8979-9
eBook Packages: Computer ScienceComputer Science (R0)
