Cost Optimization of Execution of Multi-level Deadline-Constrained Scientific Workflows on Clouds

  • Maciej Malawski
  • Kamil Figiela
  • Marian Bubak
  • Ewa Deelman
  • Jarek Nabrzyski
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8384)

Abstract

This paper introduces a cost optimization model for scientific workflows on IaaS clouds such as Amazon EC2 or RackSpace. We assume multiple IaaS clouds with heterogeneous VM instances, with limited number of instances per cloud and hourly billing. Input and output data are stored on a Cloud Object Store such as Amazon S3. Applications are scientific workflows modeled as DAGs as in the Pegasus Workflow Management System. We assume that tasks in the workflows are grouped into levels of identical tasks. Our model is specified in AMPL modeling language and allows us to minimize the cost of workflow execution under deadline constraints. We present results obtained using our model and the benchmark workflows representing real scientific applications such as Montage, Epigenomics, LIGO. We indicate how this model can be used for scenarios that require resource planning for scientific workflows and their ensembles.

Keywords

AMPL optimization Cloud computing Scientific workflows 

References

  1. 1.
    Abrishami, S., Naghibzadeh, M., Epema, D.H.: Deadline-constrained workflow scheduling algorithms for infrastructure as a service clouds. Future Gener. Comput. Syst. 29(1), 158–169 (2013). http://www.sciencedirect.com/science/article/pii/S0167739X12001008
  2. 2.
    AWS: AWS public datasets. http://aws.amazon.com/publicdatasets/ (2013)
  3. 3.
    Barrionuevo, J.J.D., Fard, H.M., Prodan, R.: Moheft: a multi-objective list-based method for workflow scheduling. In: 4th IEEE International Conference on Cloud Computing Technology and Science Proceedings, CloudCom 2012, Taipei, Taiwan, 3–6 December 2012, pp. 185–192 (2012)Google Scholar
  4. 4.
    Bharathi, S., Chervenak, A., Deelman, E., Mehta, G., Su, M.H., Vahi, K.: Characterization of scientific workflows. In: Third Workshop on Workflows in Support of Large-Scale Science, WORKS 2008, pp. 1–10. IEEE (2008). http://dx.doi.org/10.1109/WORKS.2008.4723958
  5. 5.
    Bittencourt, L.F., Madeira, E.R.M.: Hcoc: a cost optimization algorithm for workflow scheduling in hybrid clouds. J. Internet Serv. Appl. 2(3), 207–227 (2011)CrossRefGoogle Scholar
  6. 6.
    den Bossche, R.V., Vanmechelen, K., Broeckhove, J.: Online cost-efficient scheduling of deadline-constrained workloads on hybrid clouds. Future Gener. Comput. Syst. 29(4), 973–985 (2013). http://www.sciencedirect.com/science/article/pii/S0167739X12002324
  7. 7.
    Bubak, M., Kasztelnik, M., Malawski, M., Meizner, J., Nowakowski, P., Varma, S.: Evaluation of cloud providers for VPH applications. In: CCGrid2013 - 13th IEEE/ACM International Symposium on Cluster, Cloud and Grid, Computing, May 2013. http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6546092
  8. 8.
    Chen, J., Wang, C., Zhou, B.B., Sun, L., Lee, Y.C., Zomaya, A.Y.: Tradeoffs between profit and customer satisfaction for service provisioning in the cloud. In: Proceedings of the 20th International Symposium on High Performance Distributed Computing, HPDC ’11, pp. 229–238. ACM, New York (2011)Google Scholar
  9. 9.
    CloudHarmony: Benchmarks. http://cloudharmony.com/benchmarks (2011)
  10. 10.
    Deelman, E., Juve, G., Malawski, M., Nabrzyski, J.: Hosted science: managing computational workflows in the cloud. Parallel Process. Lett. 23(2), June 2013. http://www.worldscientific.com/doi/abs/10.1142/S0129626413400045
  11. 11.
    Duan, R., Prodan, R., Li, X.: A sequential cooperative game theoretic approach to storage-aware scheduling of multiple large-scale workflow applications in grids. In: 2012 ACM/IEEE 13th International Conference on Grid Computing (GRID), pp. 31–39. IEEE (2012). http://dx.doi.org/10.1109/Grid.2012.14
  12. 12.
    Fourer, R., Gay, D.M., Kernighan, B.W.: AMPL: A Modeling Language for Mathematical Programming. Duxbury Press, Belmont (2002)Google Scholar
  13. 13.
    IBM: IBM ILOG CPLEX Optimization Studio - CPLEX User’s Manual. http://pic.dhe.ibm.com/infocenter/cosinfoc/v12r5/index.jsp (2013)
  14. 14.
    Kim, H., El-Khamra, Y., Rodero, I., Jha, S., Parashar, M.: Autonomic management of application workflows on hybrid computing infrastructure. Sci. Program. 19, 75–89 (2011)Google Scholar
  15. 15.
    Malawski, M., Figiela, K., Nabrzyski, J.: Cost minimization for computational applications on hybrid cloud infrastructures. Future Gener. Comput. Syst. 29(7), 1786–1794 (2013). http://www.sciencedirect.com/science/article/pii/S0167739X13000186
  16. 16.
    Malawski, M., Juve, G., Deelman, E., Nabrzyski, J.: Cost- and deadline-constrained provisioning for scientific workflow ensembles in IaaS clouds. In: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, SC ’12. IEEE Computer Society Press (2012). http://portal.acm.org/citation.cfm?id=2389026
  17. 17.
    Mao, M., Humphrey, M.: Auto-scaling to minimize cost and meet application deadlines in cloud workflows. In: Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’11. ACM, New York (2011). http://dx.doi.org/10.1145/2063384.2063449
  18. 18.
    Pandey, S., Barker, A., Gupta, K.K., Buyya, R.: Minimizing execution costs when using globally distributed cloud services. In: 24th IEEE International Conference on Advanced Information Networking and Applications, pp. 222–229. IEEE (2010)Google Scholar
  19. 19.
    Tolosana-Calasanz, R., Banares, J.A., Pham, C., Rana, O.F.: Enforcing QoS in scientific workflow systems enacted over cloud infrastructures. J. Comput. Syst. Sci. 78(5), 1300–1315 (2012). http://www.sciencedirect.com/science/article/pii/S0022000011001607

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Maciej Malawski
    • 1
  • Kamil Figiela
    • 1
  • Marian Bubak
    • 1
    • 2
  • Ewa Deelman
    • 3
  • Jarek Nabrzyski
    • 4
  1. 1.Department of Computer ScienceAGHKrakówPoland
  2. 2.ACC CYFRONET AGHKrakówPoland
  3. 3.USC Information Sciences InstituteMarina Del ReyUSA
  4. 4.Center for Research ComputingUniversity of Notre DameNotre DameUSA

Personalised recommendations