Skip to main content
SpringerLink
Log in

On Efficiency of Multi-job Grid Allocation Based on Statistical Trace Data

  • Published:
Journal of Grid Computing Aims and scope Submit manuscript

Abstract

The ever growing number of computation-intensive applications calls for utilizing large-scale, potentially interoperable distributed infrastructures. Nowadays, such distributed systems enable the management of heterogeneous scientific workflows of considerable sizes, where job scheduling and resource management is a crucial issue. In this paper we focus on the challenges of scheduling parameter sweep applications, a specific and commonly used type of workflows where ordering of job executions is irrelevant. A parameter sweep has a large set of independent job instances, called a multi-job, submitted for execution in a single step. In order to cope with the high uncertainty and unpredictable load of resources, and the simultaneous submissions of multi-job instances, we propose a statistics-based brokering approach for allocating jobs to resources so that the makespan is minimised. Earlier studies claim that users’ predictions on job runtime are inaccurate and unusable for scheduling. Our aim is to examine, whether statistical trace data for the same purpose is efficient compared to randomized allocation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Constantini, A.: RWavePR workflow at GASuC. Online: http://www.lpds.sztaki.hu/gasuc/index.php?-m=7&s=12 (2012). Accessed 1 Oct 2012

  2. Wiggins, A.: Success-Abandonment-Classification workflow at myExperiment. Online: http://www.myexperiment.org/workflows/140.html (2012). Accessed 1 Oct 2012

  3. Buyya, R., Murshed, M., Abramson, D.: Gridsim: a toolkit for the modeling and simulation of distributed resource management and scheduling for Grid computing. In: Journal of Concurrency and Computation: Practice and Experience, pp. 1175–1220 (2002)

  4. Casanova, H., et al.: Heuristics for scheduling parameter sweep applications in Grid environments. In: Proceedings 9th Heterogeneous Computing Workshop, (HCW 2000). IEEE, Press, Piscataway (2000)

    Google Scholar 

  5. Cirne, W., Paranhos, D., Costa, L., Santos-Neto, E., Brasileiro, F., Sauve, J., Silva, F.A.B., Barros, C.O., Silveira, C.: Running bag-of-tasks applications on computational Grids: the mygrid approach. In: International Conference on Parallel Processing, pp. 407–416. IEEE Press, Piscataway (2003)

    Google Scholar 

  6. Da Silva, D.P., Cirne, W., Vilar Brasileiro F.: Trading cycles for information: using replication to schedule bag-of-tasks applications on computational Grids. Euro-Par 2003 Parallel Processing, pp. 169–180. Springer Berlin Heidelberg (2003)

  7. European Grid Infrastructure. Online: http://www.egi.eu/ (2012). Accessed 1 Oct 2012

  8. Garey, M.R., Johnson D.S.: Computers and Intractability; a Guide to the Theory of Np-Completeness. W. H. Freeman & Co., New York (1979)

    MATH  Google Scholar 

  9. Goble, C.A., Bhagat, J., Aleksejevs, S., Cruickshank, D., Michaelides, D., Newman, D., Borkum, M., Bechhofer, S., Roos, M., Li, P., De Roure, D.: myExperiment: a repository and social network for the sharing of bioinformatics workflows. Nucleic. Acids Res. 38(suppl 2), W677–W682 (2010)

    Article  Google Scholar 

  10. The Grid Workloads Archive website. Online: http://gwa.ewi.tudelft.nl (2010). Accessed 1 Oct 2012

  11. Hirales-Carbajal, A., Tchernykh, A., Yahyapour, R., Gonzalez-Garcia, J.L., Roblitz, T., Ramirez-Alcaraz, J.M.: Multiple workflow scheduling strategies with user run time estimates on a Grid. J. Grid Comput. 10(2), 325–346 (2012)

    Article  Google Scholar 

  12. Howell, F., McNab, R.: SimJava: a discrete event simulation library for Java. In: Proc. of the International Conference on Web-Based Modeling and Simulation, San Diego, USA (1998)

  13. Iosup, A., Li, H., Jan, M., Anoep, S., Dumitrescu, C., Wolters, L., Epema, D.H.J.: The Grid workloads archive. Futur. Gener. Comput. Syst. 24(7), 672–686 (2008)

    Article  Google Scholar 

  14. Kacsuk, P., Farkas, Z., Kozlovszky, M., Hermann, G., Balasko, A., Karoczkai, K., Marton, I.: WS-PGRADE/gUSE Generic DCI gateway framework for a large variety of user communities. J. Grid Comput. 9(4), 479–499 (2012)

    Google Scholar 

  15. Kwok, Y-K., Ahmad. I.: Static scheduling algorithms for allocating directed task graphs to multiprocessors. ACM Comput. Surv. (CSUR) 31(4), 406–471 (1999)

    Article  Google Scholar 

  16. Lee, C.B., Schwartzman, Y., Hardy, J., Snavely, A.: Are user runtime estimates inherently inaccurate? Springer LNCS, vol. 3277, pp. 253–263 (2005)

  17. Maheswaran, M., Ali, S., Siegal, H.J., Hensgen, D., Freund, RF.: Dynamic matching and scheduling of a class of independent tasks onto heterogeneous computing systems. In: Proceedings Heterogeneous Computing Workshop, (HCW’99), pp. 30–44. IEEE (1999)

  18. Parallel workloads archive website. Online: http://www.cs.huji.ac.il/labs/parallel/workload (2009). Accessed 1 Oct 2012

  19. Ramirez-Alcaraz, J.M., Tchernykh, A., Yahyapour, R., Schwiegelshohn, U., Quezada-Pina, A., Gonzalez-Garcia, J.L., Hirales-Carbajal, A.: Job allocation strategies with user run-time estimates for online scheduling in hierarchical Grids. J. Grid Computing 9(1), 95–116 (2011)

    Article  Google Scholar 

  20. Oprescu, A., Kielmann, T.: Bag-of-Tasks Scheduling under Budget Constraints. CloudCom, pp. 351–359 (2010)

  21. Saha, D., Menasce, D., Porto, S.: Static and dynamic processor scheduling disciplines in heterogeneous parallel architectures. J. Parallel Distrib. Comput. 28.1, 1–18 (1995)

    Google Scholar 

  22. Schwiegelshohn, U., Tchernykh, A., Yahyapour, R.: Online scheduling in Grids. In: 22nd IEEE International Symposium on Parallel and Distributed Processing (IPDPS 2008), pp. 1–10 (2008)

  23. SHaring Interoperable Workflows for large-scale scientific simulations on Available DCIs (SHIWA) Eu FP7 project. Online: http://www.shiwa-workflow.eu/ (2012). Accessed 1 Oct 2012

  24. Building a European Research Community through Interoperable Workflows and Data (ER-flow) Eu FP7 project. Online: http://www.erflow.eu/ (2013). Accessed 1 Oct 2012

  25. Silberstein, M., Sharov, A., Geiger, D., Schuster, A.: GridBot, execution of bags of tasks in multiple Grids. In: Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis (SC ’09) (2009)

  26. Ullman, J.D.: NP-complete scheduling problems. J. Comput. Syst. Sci. 10(3), 384–393 (1975)

    Article  MATH  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. MTA SZTAKI, P.O. Box 63, 1518, Budapest, Hungary

    Gábor Bacsó, Ádám Visegrádi, Attila Kertesz & Zsolt Németh

Authors
  1. Gábor Bacsó
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ádám Visegrádi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Attila Kertesz
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zsolt Németh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ádám Visegrádi.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bacsó, G., Visegrádi, Á., Kertesz, A. et al. On Efficiency of Multi-job Grid Allocation Based on Statistical Trace Data. J Grid Computing 12, 169–186 (2014). https://doi.org/10.1007/s10723-013-9274-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10723-013-9274-3

Keywords

Search

Navigation