Abstract
A dynamic version of Cloudbook is presented in this work, a new tool for automatically and unattendedly parallelizing codes which also lately distributes the tasks dynamically. Cloudbook is designed for Python codes and, above all, makes the parallelization in a way in which the number and main characteristics of the available infrastructure is taken into account for optimizing the execution (performance, bandwidth connection, etc.) in a dynamic way. Cloudbook is designed to allow developers to get the technical benefits of automated distribution and parallelization of programs with a very low learning cost. It only requires labelling the original code with a reduced set of pragmas located at function headers. Results of the tests carried out with Cloudbook with several codes on a real infrastructure are presented as well.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Barve, A., Khandelwal, S., Khan, N., Keshatiwar, S., Botre, S.: Serial to parallel code converter tools: a review. Int. J. Res. Advent Tech. Special Issue National Conference “NCPCI-2016” (2016)
Varsha, K.R.: Automatic parallelization tools: a review. IJESC 7(3), 5780–5784 (2017)
Barve, A., Khomane, S., Kulkarni, B., Katare, S., Ghadage, S.: A serial to parallel C++ code converter for multi-core machines. In: Proceedings of the International Conference on ICT in Business Industry & Government (2016)
Alsubhi, K.: An architecture for translating sequential code to parallel. In: Proceedings of the 2nd International Conference on Information System and Data Mining, pp. 88–92, April 2018
André, F., Le Fur, M., Mahéo, Y., Pazat, J.-L.: The Pandore data-parallel compiler and its portable runtime. In: Hertzberger, B., Serazzi, G. (eds.) HPCN-Europe 1995. LNCS, vol. 919, pp. 176–183. Springer, Heidelberg (1995). https://doi.org/10.1007/BFb0046627
Chapman, B., Jost, G., van der Pas, R.: Using OpenMP. The MIT Press, Cambridge (2008)
Chavarria-Miranda, D., Mellor-Crummey, J.: An evaluation of data-parallel compiler support for line-sweep applications. In: Proceedings of the International Conference on Parallel Architectures and Compilation Techniques, pp. 7–17. IEEE, New York (2002)
Chavarría-Miranda, D., Mellor-Crummey, J., Sarang, T.: Data-parallel compiler support for multipartitioning. In: Sakellariou, R., Gurd, J., Freeman, L., Keane, J. (eds.) Euro-Par 2001. LNCS, vol. 2150, pp. 241–253. Springer, Heidelberg (2001). https://doi.org/10.1007/3-540-44681-8_36
Di Martino, B., Esposito, A., D’Angelo, S., Maisto, S.A., Nacchia, S.: A compiler for agnostic programming and deployment of big data analytics on multiple platforms. IEEE Trans. Parallel Distrib. Syst. 30(9), 1920–1931 (2019). https://doi.org/10.1109/TPDS.2019.2901488
Hsu, A.W.: A Data Parallel Compiler Hosted on the GPU. Indiana University, Bloomington (2019)
Kotsifakou, M.: HPVM: heterogeneous parallel virtual machine. In: Proceedings of the 23rd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pp. 68–80. ACM Digital Library (2018)
Iserte, S., et al.: Dynamic management of resource allocation for OmpSs jobs. In: Proceedings of the First Ph.D. Symposium on Sustainable Ultrascale Computing Systems, NESUS COST Action, Timisoara (2016)
Becker, T., Karl, W., Schüle, T.: Evaluating dynamic task scheduling in a task-based runtime system for heterogeneous architectures. In: Schoeberl, M., Hochberger, C., Uhrig, S., Brehm, J., Pionteck, T. (eds.) ARCS 2019. LNCS, vol. 11479, pp. 142–155. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-18656-2_11
Ramírez-Velarde, R., Tchernykh, A., Barba-Jimenez, C., Hirales-Carbajal, A., Nolazco-Flores, J.: Adaptive resource allocation with job runtime uncertainty. J. Grid Comput. 15(4), 415–434 (2017). https://doi.org/10.1007/s10723-017-9410-6
Becker, T., Busse, P., Schuele, T.: Evaluation of dynamic task scheduling algorithms in a runtime system for heterogeneous architectures. In: 31st International Conference on Architecture of Computing Systems, Braunschweig, pp. 1–8 (2018)
Rodriguez-Pascual, M., Mayo-García, R.M., Llorente, I.M.: Montera: a framework for efficient execution of Monte Carlo codes on grid infrastructure. Comput. Inform. 32, 113–144 (2013)
Rubio-Montero, A.J., Rodríguez-Pascual, M.A., Mayo-García, R.: A simple model to exploit reliable algorithms in cloud federations. Soft. Comput. 21, 4543–4555 (2017). https://doi.org/10.1007/s00500-016-2143-9
Kumar, A., Shorey, R.: Performance analysis and scheduling of stochastic fork-join jobs in a multicomputer system. IEEE Trans. Parallel Distrib. Syst. 4, 1147–1164 (1993). https://doi.org/10.1109/71.246075
Heggie, D.C.: The Classical Gravitational N-Body Problem. Encyclopaedia of Mathematical Physics, Elsevier (2006)
Romik, D.: Shortest paths in the Tower of Hanoi graph and finite automata. SIAM J. Discret. Math. 20, 610–622 (2006)
Demo of the N-Body proof of concept and how it performs. https://drive.google.com/open?id=193f30luFq22cy8QUjzzWHgMA8zKfUAv4
Rodríguez-Pascual, M.A., et al.: Superconducting vortex lattice configurations on periodic potentials: simulation and experiment. J. Supercond. Nov. Magn. 25, 2127–2130 (2012). https://doi.org/10.1007/s10948-012-1636-8
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
García-Aranda, J.J. et al. (2021). Dynamically Distributing Tasks from an Unattended Parallel Compiler with Cloudbook. In: Nesmachnow, S., Castro, H., Tchernykh, A. (eds) High Performance Computing. CARLA 2020. Communications in Computer and Information Science, vol 1327. Springer, Cham. https://doi.org/10.1007/978-3-030-68035-0_1
Download citation
DOI: https://doi.org/10.1007/978-3-030-68035-0_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-68034-3
Online ISBN: 978-3-030-68035-0
eBook Packages: Computer ScienceComputer Science (R0)