Skip to main content

MiCADO-Edge: Towards an Application-level Orchestrator for the Cloud-to-Edge Computing Continuum


Automated deployment and run-time management of microservices-based applications in cloud computing environments is relatively well studied with several mature solutions. However, managing such applications and tasks in the cloud-to-edge continuum is far from trivial, with no robust, production-level solutions currently available. This paper presents our first attempt to extend an application-level cloud orchestration framework called MiCADO to utilise edge and fog worker nodes. The paper illustrates how MiCADO-Edge can automatically deploy complex sets of interconnected microservices in such multi-layered cloud-to-edge environments. Additionally, it shows how monitoring information can be collected from such services and how complex, user- defined run-time management policies can be enforced on application components running at any layer of the architecture. The implemented solution is demonstrated and evaluated using two realistic case studies from the areas of video processing and secure healthcare data analysis.


  1. 1.

    Gartner forecasts worldwide public cloud revenue to grow 17% in 2020 (2019)., Accessed 5 Oct 2020

  2. 2.

    Marston, S., Li, Z., Bandyopadhyay, S., Ghalsasi, A.: Cloud computing - the business perspective. In: 2011 44th Hawaii International Conference on System Sciences, pp 1–11 (2011)

  3. 3.

    Kubeedge (2020)., Accessed 4 Oct 2020

  4. 4.

    Project eve (2020)., Accessed 4 Oct 2020

  5. 5.

    Goethals, T., De Turck, F., Volckaert, B.: Fledge: Kubernetes compatible container orchestration on low-resource edge devices. In: Internet of vehicles : technologies and services toward smart cities, 6th International Conference, IOV 2019, Proceedings, pp 174–189. Springer (2020)

  6. 6.

    Yousefpour, A., Fung, C., Nguyen, T., Kadiyala, K., Jalali, F., Niakanlahiji, A., Kong, J., Jue, J.P.: All one needs to know about fog computing and related edge computing paradigms: A complete survey. J. Syst Architect. 98, 289–330 (2019).

    Article  Google Scholar 

  7. 7.

    Mercer, D.: Global connected and iot device forecast update. (2019)

  8. 8.

    Columbus, L.: Roundup of internet of things forecasts and market estimates, 2016. Forbes Magazine. (2016)

  9. 9.

    The growth in connected iot devices is expected to generate 79.4zb of data in 2025, according to a new idc forecast (2019).

  10. 10.

    IEEE standard for adoption of openfog reference architecture for fog computing. IEEE Std 1934-2018, pp. 1–176 (2018)

  11. 11.

    Kubernetes : Production-grade container orchestration (2020)., Accessed 4 Oct 2020

  12. 12.

    Docker swarm (2020)., Accessed 4 Oct 2020

  13. 13.

    Apache brooklyn (2020)., Accessed 4 Oct 2020

  14. 14.

    Cloudify orchestration platform - multi cloud, cloud native & edge (2020)., Accessed 4 Oct 2020

  15. 15.

    Cloudiator (2020)., Accessed 4 Oct 2020

  16. 16.

    Alien 4 cloud (2020)., Accessed 4 Oct 2020

  17. 17.

    Modaclouds multi-cloud devops alliance: Modaclouds releases multi-cloud devops toolbox (2020)., Accessed 4 Oct 2020

  18. 18.

    Micadoscale (2020)., Accessed 4 Oct 2020

  19. 19.

    Amazon: Aws cloudformation: Speed up cloud provisioning with infrastructure as code., Accessed 18 Oct 2020 (2020)

  20. 20.

    OpenStack: Openstack orchestration., Accessed 18 Oct 2020 (2020)

  21. 21.

    Azure resource manager (arm) templates (2020)., Accessed 19 Oct 2020

  22. 22.

    Google cloud depyment manager (2020)., Accessed 19 Oct 2020

  23. 23.

    Ghobaei-Arani, M., Souri, A., Rahmanian, A.A.: Resource management approaches in fog computing: a comprehensive review. J. Grid Comput, 1–42 (2019)

  24. 24.

    Digitbrain h2020 project (2020)., Accessed 4 Oct 2020

  25. 25.

    Asclepios eu h2020 project (2020)., Accessed 4 Oct 2020

  26. 26.

    Oasis topology and orchestration specification for cloud applications (2020)., Accessed 4 Oct 2020

  27. 27.

    Pierantoni, G., Kiss, T., Terstyanszky, G., DesLauriers, J., Gesmier, G., Dang, H.-V.: Describing and processing topology and quality of service parameters of applications in the cloud. J. Grid Comput., 1–18 (2020)

  28. 28.

    Cola - cloud orchestration at the level of application, h2020 eu project (2020)., Accessed 4 Oct 2020

  29. 29.

    DesLauriers, J, Kiss, T., Ariyattu, R.C., Dang, H.-V., Ullah, A., Bowden, J., Krefting, D., Pierantoni, G., Terstyánszky, G.: Cloud apps to-go: Cloud portability with tosca and micado. Concurrency and Computation: Practice and Experience, Accepted (2020)

  30. 30.

    Kiss, T., Kacsuk, P., Kovács, J., Rakoczi, B., Hajnal, Á., Farkas, A., Gesmier, G., Terstyánszky, G.: Micado - microservice-based cloud application-level dynamic orchestrator. Fut. Gener. Comput. Syst. 94, 937–946 (2019)

    Article  Google Scholar 

  31. 31.

    Kiss, T., DesLauriers, J., Gesmier, G., Terstyánszky, G, Pierantoni, G., Oun, O.A., Taylor, S.J.E., Anagnostou, A., Kovács, J.: A cloud-agnostic queuing system to support the implementation of deadline-based application execution policies. Future Gener. Comput. Syst. 101, 99–111 (2019).

    Article  Google Scholar 

  32. 32.

    Kovács, J., Kacsuk, P.: Occopus: a multi-cloud orchestrator to deploy and manage complex scientific infrastructures. J. Grid Comput. 16(1), 19–37 (2018)

    Article  Google Scholar 

  33. 33.

    Terraform (2020)., Accessed 4 Oct 2020

  34. 34.

    Docker (2020)., Accessed 4 Oct 2020

  35. 35.

    Prometheus (2020)., Accessed 4 Oct 2020

  36. 36.

    Kovács, J.: Supporting programmable autoscaling rules for containers and virtual machines on clouds. J. Grid Comput. 17(4), 813–829 (2019)

    Article  Google Scholar 

  37. 37.

    Ansible documentation (2020)., Accessed 19 Oct 2020

  38. 38.

    Micado - autoscaling framework for docker services on cloud (2020)., Accessed 19 Oct 2020

  39. 39.

    Oasis (2020)., Accessed 30 Oct 2020

  40. 40.

    Openstack parser (2020)., Accessed 4 Oct 2020

  41. 41.

    Kubeedge (2020)., Accessed 19 Oct 2020

  42. 42.

    Wang, N., Matthaiou, M., Nikolopoulos, D.S., Varghese, B.: Dyverse: Dynamic vertical scaling in multi-tenant edge environments. Future Generation Computer Systems (2020)

  43. 43.

    McChesney, J., Wang, N., Tanwer, A., de Lara, E., Varghese, B.: Defog: fog computing benchmarks. In: Proceedings of the 4th ACM/IEEE Symposium on Edge Computing, pp 47–58 (2019)

  44. 44.

    Dyverse - dynamic vertical scaling in multi-tenant edge environments (2020)., Accessed 19 Oct 2020

  45. 45.

    Real time face detection (fd) demo application (2020)., Accessed 19 Oct 2020

  46. 46.

    Micado tosca adt repository (2020)., Accessed 19 Oct 2020

  47. 47.

    Ullah, A.: Towards a novel biologically-inspired cloud elasticity framework. Ph.D. Thesis, University of Stirling, UK (2017)

  48. 48.

    Costan, V., Devadas, S.: Intel sgx explained. IACR Cryptol. ePrint Arch. 2016(86), 1–118 (2016)

    Google Scholar 

  49. 49.

    Sweeney, L.: k-anonymity: A model for protecting privacy. Int. J. Uncertain. Fuzz. Knowl.-Based Syst. 10(05), 557–570 (2002)

    MathSciNet  Article  Google Scholar 

  50. 50.

    Beier, M., Jansen, C., Mayer, G., Penzel, T., Rodenbeck, A., Siewert, R., Witt, M., Wu, J., Krefting, D.: Multicenter data sharing for collaboration in sleep medicine. Fut. Gener. Comput. Syst. 67, 466–480 (2017)

    Article  Google Scholar 

  51. 51.

    Beier, M., Penzel, T., Krefting, D.: A performant web-based visualization, assessment and collaboration tool for multidimensional biosignals. Front. Neuroinform. 13, 65 (2019)

    Article  Google Scholar 

  52. 52.

    Bakas, A., Michalas, A.: Power range: Forward private multi-client symmetric searchable encryption with range queries support (2020)

  53. 53.

    Sabt, M., Achemlal, M., Bouabdallah, A.: Trusted execution environment: what it is, and what it is not. In: 2015 IEEE Trustcom/BigDataSE/ISPA, vol. 1, pp 57–64, IEEE (2015)

  54. 54.

    Asclepios adt repository (2020)., Accessed 19 Oct 2020

  55. 55.

    Amazon greengrass (2020)., Accessed 4 Oct 2020

  56. 56.

    Microsoft azure iot (2020)., Accessed 4 Oct 2020

  57. 57.

    Google cloud iot (2020)., Accessed 4 Oct 2020

  58. 58.

    Kubernetes federation project (2020)., Accessed 4 Oct 2020

  59. 59.

    Submariner, connected kubernetes overlay networks (2020)., Accessed 4 Oct 2020

  60. 60.

    Edgex foundry (2020)., Accessed 4 Oct 2020

  61. 61.

    Ostberg, P., Byrne, J., Casari, P., Eardley, P., Anta, A.F., Forsman, J., Kennedy, J., Le Duc, T., Marino, M.N., Loomba, R., Lopez Pena, M.A., Veiga, J.L., Lynn, T., Mancuso, V., Svorobej, S., Torneus, A., Wesner, S., Willis, P., Domaschka, J.: Reliable capacity provisioning for distributed cloud/edge/fog computing applications. In: 2017 European Conference on Networks and Communications (EuCNC), pp 1–6 (2017)

  62. 62.

    Velasquez, K., Abreu, D. P., Gonçalves, D., Bittencourt, L., Curado, M., Monteiro, E., Madeira, E.: Service orchestration in fog environments. In: 2017 IEEE 5th International Conference on Future Internet of Things and Cloud (FiCloud), pp 329–336 (2017)

  63. 63.

    de Brito, M.S., Hoque, S., Magedanz, T., Steinke, R., Willner, A., Nehls, D., Keils, O., Schreiner, F.: A service orchestration architecture for fog-enabled infrastructures. In: 2017 Second International Conference on Fog and Mobile Edge Computing (FMEC), pp 127–132 (2017)

  64. 64.

    Alam, M., Rufino, J., Ferreira, J., Ahmed, S. H., Shah, N., Chen, Y.: Orchestration of microservices for iot using docker and edge computing. IEEE Commun. Mag. 56(9), 118–123 (2018)

    Article  Google Scholar 

  65. 65.

    Villari, M., Celesti, A., Tricomi, G., Galletta, A., Fazio, M.: Deployment orchestration of microservices with geographical constraints for edge computing. In: 2017 IEEE Symposium on Computers and Communications (ISCC), pp 633–638 (2017)

  66. 66.

    Pahl, C., Helmer, S., Miori, L., Sanin, J., Lee, B.: A container-based edge cloud paas architecture based on raspberry pi clusters. In: 2016 IEEE 4th International Conference on Future Internet of Things and Cloud Workshops (FiCloudW), pp 117–124 (2016aug)

  67. 67.

    Wang, N., Varghese, B., Matthaiou, M., Nikolopoulos, D.: Enorm: A framework for edge node resource management. IEEE Trans. Serv. Comput. PP (2017)

  68. 68.

    Zanni, A., Forsstrom, S., Jennehag, U., Bellavista, P.: Elastic provisioning of internet of things services using fog computing: An experience report. In: 2018 6th IEEE International Conference on Mobile Cloud Computing, Services, and Engineering (MobileCloud), pp 17–22 (2018)

  69. 69.

    Pizzolli, D., Cossu, G., Santoro, D., Capra, L., Dupont, C., Charalampos, D., De Pellegrini, F., Antonelli, F., Cretti, S.: Cloud4iot: A heterogeneous, distributed and autonomic cloud platform for the iot, pp. 476–479 (2016)

  70. 70.

    Taherizadeh, S., Stankovski, V., Grobelnik, M.: A capillary computing architecture for dynamic internet of things: Orchestration of microservices from edge devices to fog and cloud providers. Sensors 18 (2018)

  71. 71.

    Yigitoglu, E., Mohamed, M., Liu, L., Ludwig, H.: Foggy: A framework for continuous automated iot application deployment in fog computing. In: 2017 IEEE International Conference on AI Mobile Services (AIMS), pp 38–45 (2017)

  72. 72.

    Yigitoglu, E., Liu, L., Looper, M., Pu, C.: Distributed orchestration in large-scale iot systems. In: 2017 IEEE International Congress on Internet of Things (ICIOT), pp 58–65 (2017)

  73. 73.

    Davoli, G., Borsatti, D., Tarchi, D., Cerroni, W.: Forch: An orchestrator for fog computing service deployment. In: 2020 IFIP Networking Conference (Networking), pp 677–678 (2020)

  74. 74.

    Lertsinsrubtavee, A., Ali, A., Molina-Jimenez, C., Sathiaseelan, A., Crowcroft, J.: Picasso: A lightweight edge computing platform. In: 2017 IEEE 6th International Conference on Cloud Networking (CloudNet), pp 1–7 (2017)

  75. 75.

    Cloud application management for platforms version 1.1 (2020)., Accessed 1 Feb 2020

  76. 76.

    Paladi, N., Michalas, A., Dang, H.-V.: Towards secure cloud orchestration for multi-cloud deployments. In: Proceedings of the 5th Workshop on CrossCloud Infrastructures & Platforms, pp 1–6 (2018)

  77. 77.

    Fiware: The open source platform for our smart digital future (2020)., Accessed 19 Oct 2020

Download references


This work was funded by the following projects: ASCLEPIOS – Advanced Secure Cloud Encrypted Platform for Internationally Orchestrated Solutions in Healthcare – project, No. 826093, European Commission (EU H2020); DIGITbrain - Digital twins bringing agility and innovation to manufacturing SMEs, by empowering a network of DIHs with an integrated digital platform that enables Manufacturing as a Service – project, No. 952071, European Commission (EU H2020).

Author information



Corresponding author

Correspondence to Amjad Ullah.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Ullah, A., Dagdeviren, H., Ariyattu, R.C. et al. MiCADO-Edge: Towards an Application-level Orchestrator for the Cloud-to-Edge Computing Continuum. J Grid Computing 19, 47 (2021).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI:


  • Application-level orchestration
  • Cloud-Fog-Edge ecosystems
  • Cloud-to-Edge continuum
  • IoT applications orchestration
  • Orchestration of microservices
  • Deployment and run-time management