Skip to main content

Transparent Orchestration of Task-based Parallel Applications in Containers Platforms


This paper presents a framework to easily build and execute parallel applications in container-based distributed computing platforms in a user-transparent way. The proposed framework is a combination of the COMP Superscalar (COMPSs) programming model and runtime, which provides a straightforward way to develop task-based parallel applications from sequential codes, and containers management platforms that ease the deployment of applications in computing environments (as Docker, Mesos or Singularity). This framework provides scientists and developers with an easy way to implement parallel distributed applications and deploy them in a one-click fashion. We have built a prototype which integrates COMPSs with different containers engines in different scenarios: i) a Docker cluster, ii) a Mesos cluster, and iii) Singularity in an HPC cluster. We have evaluated the overhead in the building phase, deployment and execution of two benchmark applications compared to a Cloud testbed based on KVM and OpenStack and to the usage of bare metal nodes. We have observed an important gain in comparison to cloud environments during the building and deployment phases. This enables better adaptation of resources with respect to the computational load. In contrast, we detected an extra overhead during the execution, which is mainly due to the multi-host Docker networking.

This is a preview of subscription content, access via your institution.


  1. Advanced Multi-layered unification filesystem. Web page at (2017). Accessed April 11 2017

  2. Chameleon Cloud Project. Web page at (2017). Accessed April 11 2017

  3. Chameleon Cloud Project. Web page at (2017). Accessed April 11 2017

  4. Chef. Web page at (2017). Accessed April 11 2017

  5. Chronos Scheduler for Mesos. Web page at (2017). Accessed April 11 2017

  6. COMP Superscalar. Web page at (2017). Accessed April 11 2017

  7. COMPSs Application Repository. Web page at (2017). Accessed April 11 2017

  8. Docker. Web page at (2017). Accessed April 11 2017

  9. Docker Plug-ins. Web page at (2017). Accessed April 11 2017

  10. GUIDANCE: An Integrated Framework for Large-scale Genome and Phenome-Wide Association Studies on Parallel Computing Platforms. Web page at (2017). Accessed April 11 2017

  11. Kubernetes. Web page at (2017). Accessed April 11 2017

  12. MareNostrum supercomputer. Web page at (2017). Accessed April 11 2017

  13. Multiscale Genomics Project. Web page at (2017). Accessed April 11 2017

  14. Puppet. Web page at (2017). Accessed April 11 2017

  15. Shifter. Web page at (2017). Accessed April 11 2017

  16. Singularity. Web page at (2017). Accessed April 11 2017

  17. transPLANT Project. Web page at (2017). Accessed April 11 2017

  18. VM Ware. Web page at (2017). Accessed April 11 2017

  19. Cloud-init. Web page at (2016). Accessed November 15 2016

  20. Nova-Docker driver for OpenStack. Web page at (2016). Accessed November 15 2016

  21. OneDock: Docker driver for Open Nebula. Web page at (2016). Accessed November 15 2016

  22. Amaral, R., Badia, R.M., Blanquer, I., Braga-Neto, R., Candela, L., Castelli, D., Flann, C., De Giovanni, R., Gray, W.A., Jones, A., Lezzi, D., Pagano, P., Perez-Canhos, V., Quevedo, F., Rafanell, R., Rebello, V., Sousa-Baena, M.S., Torres, E.: Supporting biodiversity studies with the eubrazilopenbio hybrid data infrastructure. Concurrency Comput.: Pract. Experience 27 (2), 376–394 (2015).

    Article  Google Scholar 

  23. Anton, V., Ramon-Cortes, C., Ejarque, J., Badia, R.M.: Transparent execution of task-based parallel applications in docker with comp superscalar. pp. 463–467 IEEE. (2017)

  24. Armbrust, M., Fox, A., Griffith, R., Joseph, A., Katz, R., Konwinski, A., Lee, G., Patterson, D., Rabkin, A., Stoica, I., et al.: Above the clouds: a berkeley view of cloud computing. EECS Department, University of California, Berkeley, Tech. Rep UCB/EECS-2009-28 (2009)

  25. Armstrong, D., Espling, D., Tordsson, J., Djemame, K., Elmroth, E.: Contextualization: dynamic configuration of virtual machines. J. Cloud Comput. 4(1), 1 (2015)

    Article  Google Scholar 

  26. Badia, R.M., Conejero, J., Diaz, C., Ejarque, J., Lezzi, D., Lordan, F., Ramon-Cortes, C., Sirvent, R.: Comp superscalar, an interoperable programming framework. SoftwareX 3, 32–36 (2015).

    Article  Google Scholar 

  27. Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T., Ho, A., Neugebauer, R., Pratt, I., Warfield, A.: Xen and the Art of Virtualization. In: ACM SIGOPS Operating Systems Review, vol. 37, pp 164–177. ACM (2003)

  28. Bruneo, D., Fritz, T., Keidar-Barner, S., Leitner, P., Longo, F., Marquezan, C., Metzger, A., Pohl, K., Puliafito, A., Raz, D., et al.: Cloudwave: where adaptive cloud management meets Devops. In: 2014 IEEE Symposium on Computers and Communications (ISCC), pp 1–6. IEEE (2014)

  29. Conejero, J., Corella, S., Badia, R.M., Labarta, J.: Task-based programming in compss to converge from hpc to big data. The International Journal of High Performance Computing Applications 0(0).

  30. Di Tommaso, P., Palumbo, E., Chatzou, M., Prieto, P., Heuer, M.L., Notredame, C.: The impact of Docker containers on the performance of genomic pipelines. PeerJ 3, e1273 (2015).

    Article  Google Scholar 

  31. Ejarque, J., Sulistio, A., Lordan, F., Gilet, P., Sirvent, R., Badia, R.M.: Service construction tools for easy cloud deployment. In: 7th IBERIAN Grid Infrastructure Conference Proceedings, p 119 (2013)

  32. Badia, E.T.R.M., Lea, J.: Pycompss: parallel computational workflows in python. The International Journal of High Performance Computing Applications (IJHPCA) 31, 66–82 (2017).

    Article  Google Scholar 

  33. Felter, W., Ferreira, A., Rajamony, R., Rubio, J.: An Updated Performance Comparison of Virtual Machines and Linux Containers. In: 2015 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pp 171–172. IEEE (2015)

  34. Galante, G., Erpen De Bona, L.C., Mury, A.R., Schulze, B., da Rosa Righi, R.: An analysis of public clouds elasticity in the execution of scientific applications: a survey. J Grid Comput. 14(2), 193–216 (2016).

    Article  Google Scholar 

  35. Gerlach, W., Tang, W., Keegan, K., Harrison, T., Wilke, A., Bischof, J., D–Souza, M., Devoid, S., Murphy-Olson, D., Desai, N., et al.: Skyport: container-based execution environment management for multi-cloud scientific workflows. In: Proceedings of the 5Th International Workshop on Data-Intensive Computing in the Clouds, pp 25–32. IEEE Press (2014)

  36. Hindman, B., Konwinski, A., Zaharia, M., Ghodsi, A., Joseph, A.D., Katz, R., Shenker, S., Stoica, I.: Mesos: a platform for fine-grained resource sharing in the data center. In: Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation, NSDI–11, pp. 295–308. USENIX Association, Berkeley (2011).

  37. Katsaros, G., Menzel, M., Lenk, A., Revelant, J.R., Skipp, R., Eberhardt, J.: Cloud application portability with Tosca, Chef and Openstack. In: 2014 IEEE International Conference on Cloud Engineering (IC2E), pp 295–302. IEEE (2014)

  38. Kivity, A., Kamay, Y., Laor, D., Lublin, U., Liguori, A.: Kvm: the Linux Virtual Machine Monitor. In: Proceedings of the Linux Symposium, vol. 1, pp 225–230 (2007)

  39. Krishnan, S., Gonzalez, J.L.U.: Google compute engine. In: Building Your Next Big Thing with Google Cloud Platform, pp 53–81. Springer (2015)

  40. Lordan, F., Tejedor, E., Ejarque, J., Rafanell, R., Alvarez, J., Marozzo, F., Lezzi, D., Sirvent, R., Talia, D., Badia, R.M.: Servicess: an interoperable programming framework for the cloud. J Grid Comput. 12(1), 67–91 (2014).

    Article  Google Scholar 

  41. Lorido-Botran, T., Miguel-Alonso, J., Lozano, J.A.: A review of auto-scaling techniques for elastic applications in cloud environments. J Grid Comput. 12(4), 559–592 (2014)

    Article  Google Scholar 

  42. Meng, H., Thain, D.: Umbrella: a Portable Environment Creator for Reproducible Computing on Clusters, Clouds, and Grids. In: Proceedings of the 8th International Workshop on Virtualization Technologies in Distributed Computing, VTDC ’15, pp. 23–30. ACM, New York (2015).

  43. Merkel, D.: Docker: lightweight linux containers for consistent development and deployment. Linux Journal 2014(239), 2 (2014)

    Google Scholar 

  44. Peinl, R., Holzschuher, F., Pfitzer, F.: Docker cluster management for the cloud - survey results and own solution. J Grid Comput. 14(2), 265–282 (2016).

    Article  Google Scholar 

  45. Sánchez-Expósito, S., Martín, P., Ruiz, J.E., Verdes-Montenegro, L., Garrido, J., Sirvent, R., Falcó, A.R., Badia, R.M., Lezzi, D.: Web services as building blocks for science gateways in astrophysics. J Grid Comput. 14(4), 673–685 (2016).

    Article  Google Scholar 

  46. Sefraoui, O., Aissaoui, M., Eleuldj, M.: Openstack: toward an open-source solution for cloud computing. Int. J Comput. Appl. 55(3), 38–42 (2012)

    Google Scholar 

  47. Sotomayor, B., Montero, R.S., Llorente, I.M., Foster, I.: Virtual infrastructure management in private and hybrid clouds. IEEE Internet comput. 13(5), 14–22 (2009)

    Article  Google Scholar 

  48. Zheng, C., Thain, D.: Integrating containers into workflows: a case study using Makeflow, Work Queue, and Docker. In: Proceedings of the 8Th International Workshop on Virtualization Technologies in Distributed Computing, pp 31–38. ACM (2015)

Download references


This work is partly supported by the Spanish Government through Programa Severo Ochoa (SEV-2015-0493), by the Spanish Ministry of Science and Technology through TIN2015-65316 project, by the Generalitat de Catalunya under contracts 2014-SGR-1051 and 2014-SGR-1272, and by the European Union through the Horizon 2020 research and innovation program under grant 690116 (EUBra-BIGSEA Project). Results presented in this paper were obtained using the Chameleon testbed supported by the National Science Foundation.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Cristian Ramon-Cortes.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Ramon-Cortes, C., Serven, A., Ejarque, J. et al. Transparent Orchestration of Task-based Parallel Applications in Containers Platforms. J Grid Computing 16, 137–160 (2018).

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI:


  • Cloud computing
  • Containers orchestration
  • Linux containers
  • Distributed systems
  • Parallel programming models