Skip to main content
SpringerLink
Log in

A Meta-Brokering Framework for Science Gateways

  • Published:
Journal of Grid Computing Aims and scope Submit manuscript

Abstract

Recently scientific communities produce a growing number of computation-intensive applications, which calls for the interoperation of distributed infrastructures including Clouds, Grids and private clusters. The European SHIWA and ER-flow projects have enabled the combination of heterogeneous scientific workflows, and their execution in a large-scale system consisting of multiple Distributed Computing Infrastructures. One of the resource management challenges of these projects is called parameter study job scheduling. A parameter study job of a workflow generally has a large number of input files to be consumed by independent job instances. In this paper we propose a meta-brokering framework for science gateways to support the execution of such workflows. In order to cope with the high uncertainty and unpredictable load of the utilized distributed infrastructures, we introduce the so called resource priority services. These tools are capable of determining and dynamically updating priorities of the available infrastructures to be selected for job instances. Our evaluations show that this approach implies an efficient distribution of job instances among the available computing resources resulting in shorter makespan for parameter study workflows.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Wiggins, A.: Success-Abandonment-Classification workflow at myExperiment. Online: http://www.myexperiment.org/workflows/140.html (2012)

  2. SHaring Interoperable Workflows for large-scale scientific simulations on Available DCIs (SHIWA) EU FP7 project.Online: http://www.shiwa-workflow.eu/ (2012)

  3. Building a European Research Community through Interoperable Workflows and Data (ER-flow) Eu FP7 project. Online: http://www.erflow.eu/ (2013)

  4. Rubio-Montero, A.J., Huedo, E., Castejon, F., Mayo-Garcia, R.: GWpilot: Enabling multi-level scheduling in distributed infrastructures with GridWay and pilot jobs. Fut. Gener. Comput. Syst. 45, 25–52 (2015)

    Article  Google Scholar 

  5. Hirales-Carbajal, A., Tchernykh, A., Yahyapour, R., Gonzalez-Garcia, J.L., Roblitz, T., Ramirez-Alcaraz, J.M.: Multiple workflow scheduling strategies with user run time estimates on a grid. Journal of Grid Computing (2012)

  6. Oprescu, A., Kielmann, T.: Bag-of-tasks scheduling under budget constraints. CloudCom, 351–359 (2010)

  7. Silberstein, M., Sharov, A., Geiger, D., Schuster, A.: GridBot, execution of bags of tasks in multiple grids. In: Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis (SC ’09) (2009)

  8. Kacsuk, P., Farkas, Z., Kozlovszky, M., Hermann, G., Balasko, Á., Karóczkai, K., Márton, I.: WS-PGRADE/gUSE generic DCI gateway framework for a large variety of user communities. J. Grid Comput. 10(4), 601–630 (2012)

    Article  Google Scholar 

  9. Lee, C.B., Schwartzman, Y., Hardy, J., Snavely, A.: Are user runtime estimates inherently inaccurate? Springer LNCS 3277, 253–263 (2005)

    Google Scholar 

  10. Fan, Y., Pamidighantam, S., Smith, W.: Incorporating job predictions into the SEAGrid science gateway. ACM, NY, USA (2014)

  11. Resource Priority Service for gUSE. Online: http://sourceforge.net/projects/priorityservice.guse.p/ (2015)

  12. Ullman, J.D.: NP-complete scheduling problems. J. Comput. Syst. Sci. 10(3), 384–393 (1975)

    Article  MathSciNet  MATH  Google Scholar 

  13. Schwiegelshohn, U., Tchernykh, A., Yahyapour, R.: Online scheduling in grids. 22nd IEEE International Symposium on Parallel and Distributed Processing (IPDPS 2008), pp. 1–10 (2008)

  14. Casanova, H., et al.: Heuristics for scheduling parameter sweep applications in grid environments. Heterogeneous Computing Workshop, 2000. (HCW 2000) Proceedings. 9th. IEEE (2000)

  15. Muthucumaru, M., Ali, S., Siegal, H.J., Hensgen, D., Freund, R.F.: Dynamic matching and scheduling of a class of independent tasks onto heterogeneous computing systems. In: Heterogeneous Computing Workshop, 1999. (HCW’99) Proceedings, pp 30–44. IEEE (1999)

  16. Lucas-Simarro, J.L., Moreno-Vozmediano, R., Montero, R.S., Llorente, I.M.: Scheduling strategies for optimal service deployment across multiple clouds. Future Generation Computer Systems, 10.1016/j.future.2012.01.007 (2012)

  17. Kertesz, A., Kacsuk, P.: GMBS: A new middleware service for making grids interoperable. Fut. Gener. Comput. Syst. 16, 542–553 (2010)

    Article  Google Scholar 

  18. Assuncao, M.D., Buyya, R., Venugopal, S.: InterGrid: A case for internetworking islands of grids. Concurrency and Computation: Practice and Experience (CCPE) (2007)

  19. Buyya, R., Ranjan, R., Calheiros, R.N.: InterCloud: Utility-oriented federation of cloud computing environments for scaling of application services. Lect. Notes Comput. Sci. Algorithm. Architectures Parallel Process. 6081 (2010)

  20. Buyya, R., Ranjan, R.: Special section: Federated resource management in grid and cloud computing systems. Fut. Gener. Comput. Syst. 26, 1189–1191 (2010)

    Article  Google Scholar 

  21. Kertesz, A., Maros, G., Dombi, J.D.: Multi-job meta-brokering in distributed computing infrastructures using pliant logic. In: Proceedings of the 22th Euromicro International Conference on Parallel, Distributed and Network-Based Computing (PDP’14), pp 138–145. IEEE CS, Turin, Italy (2014)

  22. Karoczkai, K., Kertesz, A., Kacsuk, P.: Brokering solution for science gateways using multiple distributed computing infrastructures. In: 7th International Workshop on Science Gateways (IWSG). doi:10.1109/IWSG.2015.12, pp 28–33, 3–5 (2015)

  23. Bacso, G., Kis, T., Visegradi, A., Kertesz, A., Nemeth, Z.S.: A set of successive job allocation models in distributed computing infrastructures (2015). doi:doi:10.1007/s10723-015-9347-6 10.1007/s10723-015-9347-6

  24. DCI-Bridge User Manual. http://sourceforge.net/projects/dcibridge/files/DCI-Bridge-3.5.2/Documentation/DCIBridgeManualv3.5.2.pdf (2015)

  25. Korkhov, V., Krefting, D., Kukla, T., et al.: Exploring workflow interoperability for neuroimage analysis on the SHIWA platform. J Grid Comput. 11, 505 (2013). doi:10.1007/s10723-013-9262-7

    Article  Google Scholar 

  26. Kiss, T.: Science gateways for the broader take-up of distributed computing infrastructures. J Grid Comput. 10, 599 (2012). doi:10.1007/s10723-012-9245-0

    Article  Google Scholar 

  27. Liu, J., Pacitti, E., Valduriez, P., et al.: A survey of data-intensive scientific workflow management. J Grid Comput. 13, 457 (2015). doi:10.1007/s10723-015-9329-8

    Article  Google Scholar 

  28. Kacsuk, P. (ed.): Science Gateways for Distributed Computing Infrastructures: Development Framework and Exploitation by Scientific User Communities, Springer. pp. 301 (2014)

Download references

Author information

Authors and Affiliations

  1. Laboratory of Parallel and Distributed Systems, MTA SZTAKI, Budapest, Hungary

    Krisztian Karoczkai, Attila Kertesz & Peter Kacsuk

Authors
  1. Krisztian Karoczkai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Attila Kertesz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Peter Kacsuk
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Attila Kertesz.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Karoczkai, K., Kertesz, A. & Kacsuk, P. A Meta-Brokering Framework for Science Gateways. J Grid Computing 14, 687–703 (2016). https://doi.org/10.1007/s10723-016-9378-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10723-016-9378-7

Keywords

Search

Navigation