Advertisement

The Journal of Supercomputing

, Volume 72, Issue 3, pp 985–1012 | Cite as

Cost-aware DAG scheduling algorithms for minimizing execution cost on cloud resources

  • Moïse W. Convolbo
  • Jerry ChouEmail author
Article

Abstract

Directed acyclic graph (DAG) scheduling is a well-known problem, because a DAG can be used to describe a wide range of complex applications, including scientific applications and parallel computing jobs. Most DAG scheduling algorithms were proposed to minimize the job makespan (i.e., execution time) on a multiprocessor computer or cluster. However, as the cost-driven public cloud services have become an attractive and popular platform for providing computing resources, cost minimization emerges as a new critical issue. Therefore, the objective of this work is to propose and solve the cost optimization problem for scheduling DAGs on an IaaS cloud platform where task scheduling must cope with resource provisioning to achieve the optimal solution. In this paper, we proposed both optimal and heuristic scheduling algorithms, and we evaluated them across a variety of DAGs using the price model from EC2. Comparing to other cost-oblivious DAG schedules that aim to minimize makespan or resource usage, the results show that our cost-aware heuristic algorithm can reduce cost by 20–50 % and achieve a cost within x1.16 of the optimal one.

Keywords

DAG scheduling Virtual machine Cloud computing  Cost optimization 

References

  1. 1.
    Adam TL, Chandy KM, Dickson JR (1974) A comparison of list schedules for parallel processing systems. Commun ACM 17(12):685–690CrossRefzbMATHGoogle Scholar
  2. 2.
    Ahmad I, Yu-Kwong K (1998) On exploiting task duplication in parallel program scheduling. IEEE Trans Parallel Distrib Syst 9(9):872–892CrossRefGoogle Scholar
  3. 3.
    Almeida VAF, Vasconcelos IMM, Arabe J, Menasce D (1992) Using random task graphs to investigate the potential benefits of heterogeneity in parallel systems. In: ACM/IEEE conference on supercomputing, pp 683–691Google Scholar
  4. 4.
    Arabnejad H, Barbosa J (2014) List scheduling algorithm for heterogeneous systems by an optimistic cost table. IEEE Trans Parallel Distrib Syst 25(3):682–694CrossRefGoogle Scholar
  5. 5.
    Armbrust M, Fox A, Griffith R, Joseph AD, Katz R, Konwinski A, Lee G, Patterson D, Rabkin A, Stoica I, Zaharia M (2010) A view of cloud computing. Commun ACM 53(4):50–58CrossRefGoogle Scholar
  6. 6.
    Bajaj R, Agrawal D (2004) Improving scheduling of tasks in a heterogeneous environment. IEEE Trans Parallel Distrib Syst 15(2):107–118CrossRefGoogle Scholar
  7. 7.
    Bittencourt L, Madeira E (2011) HCOC: a cost optimization algorithm for workflow scheduling in hybrid clouds. J Internet Serv Appl 2(3):207–227CrossRefGoogle Scholar
  8. 8.
    Canon L, Jeannot E (2010) Evaluation and optimization of the robustness of dag schedules in heterogeneous environments. IEEE Trans Parallel Distrib Syst 21(4):532–546CrossRefGoogle Scholar
  9. 9.
    Casavant TL, Kuhl JG (1988) A taxonomy of scheduling in general-purpose distributed computing systems. IEEE Trans Softw Eng 14(2):141–154CrossRefGoogle Scholar
  10. 10.
    Chen G (2010) Simplified particle swarm optimization algorithm based on particles classification. Int Conf Nat Comput 5:2701–2705Google Scholar
  11. 11.
    Chen H, Shirazi B, Kavi K, Hurson A (1993) Static scheduling using linear clustering and task duplication. In: Parallel and distributed computing and systems, pp 285–290Google Scholar
  12. 12.
    Colin J, Chretienne P (1991) C.P.M. scheduling with small computation delays and task duplication. Oper Res 39:680–684CrossRefzbMATHGoogle Scholar
  13. 13.
    Cordeiro D, Mounié G, Perarnau S, Trystram D, Vincent J-M, Wagner F (2010) Random graph generation for scheduling simulations. In: Proceedings of the 3rd international ICST conference on simulation tools and techniques, pp 60:1–60:10Google Scholar
  14. 14.
    Darbha S, Agrawal DP (1998) Optimal scheduling algorithm for distributed-memory machines. IEEE Trans Parallel Distrib Syst 9(1):87–95CrossRefGoogle Scholar
  15. 15.
    Dean J, Ghemawat S (2008) MapReduce: simplified data processing on large clusters. Commun ACM 51(1):107–113CrossRefGoogle Scholar
  16. 16.
    Fujimoto N, Hagihara K (2003) Near-optimal dynamic task scheduling of precedence constrained coarse-grained tasks onto a computational grid. In: Proceedings of international conference on parallel and distributed computing, pp 80–87Google Scholar
  17. 17.
    Graham R, Lawler L, Lenstra J, Kan A (1979) Optimization and approximation in deterministic sequencing and scheduling: a survey. Ann Discrete Math 5:287–326MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Hu T (1961) Parallel sequencing and assembly line problems. Oper Res 9(6):841–848MathSciNetCrossRefGoogle Scholar
  19. 19.
    Juve G, Chervenak A, Deelman E, Bharathi S, Mehta G, Vahi K (2013) Characterizing and profiling scientific workflows. Future Gener Comput Syst 29(3):682–692CrossRefGoogle Scholar
  20. 20.
    Kvas A, Ojstersek M, Zumer V (1994) Evaluation of static program allocation schemes for macro data-flow computer. In: Proceedings of the 20th EUROMICRO conference, pp 573–580Google Scholar
  21. 21.
    Kwok Y-K, Ahmad I (1996) Dynamic critical-path scheduling: an effective technique for allocating task graphs to multiprocessors. IEEE Trans Parallel Distrib Syst 7(5):506–521CrossRefGoogle Scholar
  22. 22.
    Mao M, Humphrey M (2011) Auto-scaling to minimize cost and meet application deadlines in cloud workflows. ACM/IEEE Conf Supercomput 49(1–49):12Google Scholar
  23. 23.
    Olston C, Reed B, Srivastava U, Kumar R, Tomkins A (2008) Pig latin: a not-so-foreign language for data processing. In: ACM SIGMOD, pp 1099–1110Google Scholar
  24. 24.
    Radulescu A, van Gemund A (2000) Fast and effective task scheduling in heterogeneous systems. In: Proceedings of 9th heterogeneous computing workshop, 2000 (HCW 2000), pp 229–238Google Scholar
  25. 25.
    Raju R, Amudhavel J, Pavithra M, Anuja S, Abinaya B (2014) A heuristic fault tolerant “MapReduce framework for minimizing makespan in hybrid cloud environment. In: International conference on green computing communication and electrical engineering (ICGCCEE), pp 1–4Google Scholar
  26. 26.
    Rodriguez MA, Buyya R (2014) Deadline based resource provisioningand scheduling algorithm for scientific workflows on clouds. IEEE Trans Cloud Comput 2(2):222–235CrossRefGoogle Scholar
  27. 27.
    Shirazi B, Wang M, Pathak G (1990) Analysis and evaluation of heuristic methods for static task scheduling. J Parallel Distrib Comput 10(3):222–2232CrossRefGoogle Scholar
  28. 28.
    Thusoo A, Sarma JS, Jain N, Shao Z, Chakka P, Anthony S, Liu H, Wyckoff P, Murthy R (2009) Hive: a warehousing solution over a Map-Reduce framework. Proc VLDB Endow 2(2):1626–1629CrossRefGoogle Scholar
  29. 29.
    Topcuouglu H, Hariri S, Wu M-Y (2002) Performance-effective and low-complexity task scheduling for heterogeneous computing. IEEE Trans Parallel Distrib Syst 13(3):260–274CrossRefGoogle Scholar
  30. 30.
    Wu M-Y, Gajski D (1990) Hypertool: a programming aid for message-passing systems. IEEE Trans Parallel Distrib Syst 1(3):330–343CrossRefGoogle Scholar
  31. 31.
    Yang C-H, Lee P, Chung Y-C (2007) Improving static task scheduling in heterogeneous and homogeneous computing systems. In: ICPP, pp 45–45Google Scholar
  32. 32.
    Yang T, Gerasoulis A (1994) DSC: scheduling parallel tasks on an unbounded number of processors. IEEE Trans Parallel Distrib Syst 5(9):951–967CrossRefGoogle Scholar
  33. 33.
    Yu WH (1984) Lu decomposition on a multiprocessing system with communications delay, Ph.D. thesis. University of California, Berkeley. AAI8427141Google Scholar
  34. 34.
    Yu Y, Isard M, Fetterly D, Budiu M, Erlingsson U, Gunda PK, Currey J (2008) DryadLINQ: a system for general-purpose distributed data-parallel computing using a high-level language. In: Proceedings of the USENIX conference on operating systems design and implementation, pp 1–14Google Scholar
  35. 35.
    Zaharia M, Chowdhury M, Franklin MJ, Shenker S, Stoica I (2010) Spark: cluster computing with working sets. In: Proceedings of the 2Nd USENIX conference on hot topics in cloud computing, HotCloud’10. USENIX Association, Berkeley, p 10Google Scholar
  36. 36.
    Zhang C, De Sterck H (2010) Cloudbatch: a batch job queuing system on clouds with hadoop and hbase. In: IEEE international conference on cloud computing technology and science (CloudCom), pp 368–375Google Scholar
  37. 37.
    Zhou AC, He B (2014) Transformation-based monetary cost optimizations for workflows in the cloud. IEEE Trans Cloud Comput 2(1):85–98CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.Institute of Information Systems and ApplicationsNational Tsing Hua UniversityHsinchuTaiwan
  2. 2.Department of Computer ScienceNational Tsing Hua UniversityHsinchuTaiwan

Personalised recommendations