Skip to main content

Runtime and energy constrained work scheduling for heterogeneous systems

The Journal of Supercomputing (2022)Cite this article

Abstract

Heterogeneous hardware systems consisting of CPUs and different types of accelerators are wide-spread nowadays for large supercomputers as well as smaller cluster systems in the field of high-performance computing (HPC). A fundamental problem for such systems is the distribution of the workload of data-parallel HPC applications onto heterogeneous compute devices. The distribution of the workload tries to achieve (1) a well-balanced and runtime efficient program execution and (2) energy efficiency. However, typically both goals are contradicting objectives resulting in a challenging bi-criteria optimization problem. In this paper, we present an efficient scheduling algorithm that assigns work bundles to heterogeneous compute devices and determines an optimal solution for minimizing the makespan of a task under a given energy constraint. Work bundles are equal-sized, medium-grained data chunks that are obtained by partitioning the workload of data-parallel applications. Energy consumption and execution time for processing a single work bundle varies depending on the respective compute device and is essential for beneficial scheduling strategies. We formulate our optimization problem as an Integer Linear Program and devise an efficient bisection algorithm, which computes optimal solutions with logarithmic-time complexity. Experiments emphasize the efficiency of our algorithm. Further we investigate the two-dimensional optimization space and sketch an algorithm for Strong Pareto Optimal solutions.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price includes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

References

  1. Raca V, Mehofer E (2020) Clustercl: comprehensive support for multi-kernel data-parallel applications in heterogeneous asymmetric clusters. J. Supercomput. 76(12):9976–10008. https://doi.org/10.1007/s11227-020-03234-w

    Article  Google Scholar 

  2. Raca V, Mehofer E (2015)Device-sensitive framework for handling heterogeneous asymmetric clusters efficiently. In: 27th International Symposium on Computer Architecture and High Performance Computing, SBAC-PAD 2015, Florianópolis, Brazil, October 17–21, 2015, pp 178–185. https://doi.org/10.1109/SBAC-PAD.2015.15

  3. Raca V, Mehofer E, Hudec M (2016) Optimal time and energy efficient work distributions in heterogeneous systems. In: 24th Euromicro International Conference on Parallel, Distributed, and Network-Based Processing (PDP), Heraklion, Greece, pp 440– 447. https://doi.org/10.1109/PDP.2016.68

  4. Liu Q, Luk W (2012) Heterogeneous systems for energy efficient scientific computing. In: 8th International Symposium, ARC 2012, Hong Kong, China, March 19-23, 2012. Proceedings, vol 7199, pp 64–75. https://doi.org/10.1007/978-3-642-28365-9_6

  5. Bertsimas D, Weismantel R (2005) Optimization over integers. Athena Scientific, US

    Google Scholar 

  6. Makhorin A (2012) GLPK - GNU Project - Free Software Foundation (FSF). https://www.gnu.org/software/glpk/#TOCintroduction. Accessed 14 May 2022

  7. Hochbaum DS, Shmoys DB (1987) Using dual approximation algorithms for scheduling problems theoretical and practical results. J ACM 34(1):144–162. https://doi.org/10.1145/7531.7535

    MathSciNet  Article  Google Scholar 

  8. Kleinberg JM, Tardos É (2006) Algorithm design. Addison-Wesley, US

    Google Scholar 

  9. Branke J, Deb K, Miettinen K, Slowinski R (2008) Multiobjective optimization: interactive and evolutionary approaches. Springer, Germany

    Book  Google Scholar 

  10. Marler RT, Arora JS (2004) Survey of multi-objective optimization methods for engineering. Struct Multidiscip Optim 26(6):369–395. https://doi.org/10.1007/s00158-003-0368-6

    MathSciNet  Article  MATH  Google Scholar 

  11. Deb K (2014) Multiobjective optimization. In: Burke EK, Kendall G (eds) Search methodologies: introductory tutorials in optimization and decision support techniques. Springer, Boston, pp 403–449. https://doi.org/10.1007/978-1-4614-6940-7_15

    Chapter  Google Scholar 

  12. Shmoys DB, Tardos E (1993)Scheduling unrelated machines with costs. In: Proceedings of the Fourth Annual ACM-SIAM Symposium on Discrete Algorithms. SODA ’93, pp. 448– 454. Society for Industrial and Applied Mathematics, Philadelphia, PA, USA. http:dl-acm-org.uaccess.univie.ac.at/citation.cfm?id=313559.313851

  13. Freeh VW, Lowenthal DK, Pan F, Kappiah N, Springer R, Rountree BL, Femal ME (2007) Analyzing the energy-time trade-off in high-performance computing applications. IEEE Trans Parallel Distrib Syst 18(6):835–848

    Article  Google Scholar 

  14. He Y, Liu F, Cao H-j, Li C-b (2005) A bi-objective model for job-shop scheduling problem to minimize both energy consumption and makespan. J Central South Univ Technol 12(2):167–171

    Article  Google Scholar 

  15. Durillo JJ, Nae V, Prodan R (2013) Multi-objective workflow scheduling: an analysis of the energy efficiency and makespan tradeoff. In: 2013 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, pp 203– 210. https://doi.org/10.1109/CCGrid.2013.62

  16. Hajiamini S, Shirazi B, Crandall A, Ghasemzadeh H (2020) A dynamic programming framework for dvfs-based energy-efficiency in multicore systems. IEEE Trans Sustain Comput 5(1):1–12

    Article  Google Scholar 

  17. D’Amico M, Gonzalez, JC (2021) Energy hardware and workload aware job scheduling towards interconnected hpc environments. IEEE Trans Parallel Distrib Syst 1. https://doi.org/10.1109/TPDS.2021.3090334

  18. Kumar N, Vidyarthi DP (2021) A novel energy-efficient scheduling model for multi-core systems. Clust Comput 24(2):643–666. https://doi.org/10.1007/s10586-020-03143-w

    Article  Google Scholar 

  19. Pineau J-F, Robert Y, Vivien F (2011) Energy-aware scheduling of bag of tasks applications on master worker platforms. Concurrency Comput Practice Exp 23(2):145–157

    Article  Google Scholar 

  20. Li D, Wu J (2012) Energy-aware scheduling for frame-based tasks on heterogeneous multiprocessor platforms. In: 2012 41st International Conference on Parallel Processing, pp 430– 439. https://doi.org/10.1109/ICPP.2012.26

  21. Tang X, Fu Z (2020) Cpu-gpu utilization aware energy-efficient scheduling algorithm on heterogeneous computing systems. IEEE Access 8:58948–58958

    Article  Google Scholar 

  22. Li Y, Liu Y, Qian D (2009) A heuristic energy-aware scheduling algorithm for heterogeneous clusters. In: 2009 15th International Conference on Parallel and Distributed Systems, pp 407– 413. https://doi.org/10.1109/ICPADS.2009.33

  23. Wang G, Ren X (2010) Power-efficient work distribution method for cpu-gpu heterogeneous system. In: International Symposium on Parallel and Distributed Processing with Applications, pp 122– 129 ( 2010). https://doi.org/10.1109/ISPA.2010.22

  24. Cabrera A, Acosta A, Almeida F, Blanco V (2020) A dynamic multi objective approach for dynamic load balancing in heterogeneous systems. IEEE Trans Parallel Distrib Syst 31(10):2421–2434. https://doi.org/10.1109/TPDS.2020.2989869

    Article  Google Scholar 

  25. Balaprakash P, Tiwari A, Wild SM (2014) Multi objective optimization of HPC kernels for performance, power, and energy. In: Jarvis SA, Wright SA, Hammond SD (eds) High performance computing systems. performance modeling, benchmarking and simulation. Springer International Publishing, Cham, p 239–260. isbn: 978-3-319-10214-6

    Chapter  Google Scholar 

  26. Friese R, Khemka B, Maciejewski AA, Siegel HJ, Koenig GA, Powers S, Hilton M, Rambharos J, Okonski G, Poole SW (2013) An analysis framework for investigating the trade-offs between system performance and energy consumption in a heterogeneous computing environment. In: 2013 IEEE International Symposium on Parallel Distributed Processing, Workshops and Phd Forum, pp 19–30. https://doi.org/10.1109/IPDPSW.2013.142

  27. Young BD, Apodaca J, Briceño LD, Smith J, Pasricha S, Maciejewski AA, Siegel HJ, Khemka B, Bahirat S, Ramirez A, Zou Y (2013) Deadline and energy constrained dynamic resource allocation in a heterogeneous computing environment. J Supercomput 63(2):326–347. https://doi.org/10.1007/s11227-012-0740-7

    Article  Google Scholar 

  28. Tarplee KM, Friese R, Maciejewski AA, Siegel HJ, Chong EKP (2016) Energy and makespan tradeoffs in heterogeneous computing systems using efficient linear programming techniques. IEEE Trans Parallel Distrib Syst 27(6):1633–1646. https://doi.org/10.1109/TPDS.2015.2456020

    Article  Google Scholar 

  29. Li W, Liu X, Cai X, Zhang X (2019) Approximation algorithm for the energy-aware profit maximizing problem in heterogeneous computing systems. J Parall Distrib Comput 124:70–77. https://doi.org/10.1016/j.jpdc.2018.10.013

    Article  Google Scholar 

  30. Yang P, Wong C, Marchal P, Catthoor F, Desmet D, Verkest D, Lauwereins R (2001) Energy-aware runtime scheduling for embedded-multiprocessor socs. IEEE Des Test Comput 18(5):46–58. https://doi.org/10.1109/54.953271

    Article  Google Scholar 

  31. Nesmachnow S, Dorronsoro B, Pecero JE, Bouvry P (2013) Energy-aware scheduling on multicore heterogeneous grid computing systems. J Grid Comput 11(4):653–680. https://doi.org/10.1007/s10723-013-9258-3

    Article  Google Scholar 

  32. Dongarra JJ, Jeannot E, Saule E, Shi Z (2007) Bi-objective scheduling algorithms for optimizing makespan and reliability on heterogeneous systems. In: Proceedings of the Nineteenth Annual ACM Symposium on Parallel Algorithms and Architectures. SPAA ’07, pp. 280–288. ACM, New York, NY, USA (2007). https://doi.org/10.1145/1248377.1248423

Download references

Author information

Author notes

  1. Seeun William Umboh, Eduard Mehofer and Bernhard Scholz have contributed equally to this work.

Authors and Affiliations

  1. Faculty of Electrical and Computer Engineering, University of Prishtina, Prishtina, Kosovo

    Valon Raca

  2. School of Computer Science, University of Sydney, Sydney, Australia

    Seeun William Umboh & Bernhard Scholz

  3. Faculty of Computer Science, University of Vienna, Vienna, Austria

    Eduard Mehofer

Authors
  1. Valon Raca
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Seeun William Umboh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Eduard Mehofer
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bernhard Scholz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Valon Raca.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Raca, V., Umboh, S.W., Mehofer, E. et al. Runtime and energy constrained work scheduling for heterogeneous systems. J Supercomput (2022). https://doi.org/10.1007/s11227-022-04556-7

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11227-022-04556-7

Keywords

  • High performance computing
  • Runtime and energy efficiency
  • Data-parallel applications
  • Heterogeneous systems
  • Pareto efficiency