Workflow Fairness Control on Online and Non-clairvoyant Distributed Computing Platforms

  • Rafael Ferreira da Silva
  • Tristan Glatard
  • Frédéric Desprez
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8097)

Abstract

Fairly allocating distributed computing resources among workflow executions is critical to multi-user platforms. However, this problem remains mostly studied in clairvoyant and offline conditions, where task durations on resources are known, or the workload and available resources do not vary along time. We consider a non-clairvoyant, online fairness problem where the platform workload, task costs and resource characteristics are unknown and not stationary. We propose a fairness control loop which assigns task priorities based on the fraction of pending work in the workflows. Workflow characteristics and performance on the target resources are estimated progressively, as information becomes available during the execution. Our method is implemented and evaluated on 4 different applications executed in production conditions on the European Grid Infrastructure. Results show that our technique reduces slowdown variability by 3 to 7 compared to first-come-first-served.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    N’Takpe, T., Suter, F.: Concurrent scheduling of parallel task graphs on multi-clusters using constrained resource allocations. In: IPDPS 2009, pp. 1–8 (2009)Google Scholar
  2. 2.
    Zhao, H., Sakellariou, R.: Scheduling multiple DAGs onto heterogeneous systems. In: IPDPS 2006, p. 159 (2006)Google Scholar
  3. 3.
    Casanova, H., Desprez, F., Suter, F.: On cluster resource allocation for multiple parallel task graphs. J. of Par. and Dist. Computing 70(12), 1193–1203 (2010)MATHCrossRefGoogle Scholar
  4. 4.
    Glatard, T., et al.: A virtual imaging platform for multi-modality medical image simulation. IEEE Trans. Med. Imaging 32, 110–118 (2013)CrossRefGoogle Scholar
  5. 5.
    Shahand, S., et al.: Front-ends to Biomedical Data Analysis on Grids. In: Proceedings of HealthGrid 2011, Bristol, UK (June 2011)Google Scholar
  6. 6.
    Kacsuk, P.: P-GRADE Portal Family for Grid Infrastructures. Concurrency and Computation: Practice and Experience 23(3), 235–245 (2011)CrossRefGoogle Scholar
  7. 7.
    Roberto: Supporting e-science applications on e-infrastructures: Some use cases from latin america. In: Grid Computing, pp. 33–55 (2011)Google Scholar
  8. 8.
    Hsu, C.C., Huang, K.C., Wang, F.J.: Online scheduling of workflow applications in grid environments. Fut. Gen. Computer Systems 27(6), 860–870 (2011)CrossRefGoogle Scholar
  9. 9.
    Arabnejad, H., Barbosa, J.: Fairness resource sharing for dynamic workflow scheduling on heterogeneous systems. In: ISPA 2012, pp. 633–639 (July 2012)Google Scholar
  10. 10.
    Sabin, G., Kochhar, G., Sadayappan, P.: Job fairness in non-preemptive job scheduling. In: ICPP 2004, pp. 186–194 (2004)Google Scholar
  11. 11.
    Hirales-Carbajal, A., et al.: Multiple workflow scheduling strategies with user run time estimates on a grid. Journal of Grid Computing 10, 325–346 (2012)CrossRefGoogle Scholar
  12. 12.
    Ferreira da Silva, R., Glatard, T., Desprez, F.: Self-healing of operational workflow incidents on distributed computing infrastructures. In: CCGrid 2012, pp. 318–325 (2012)Google Scholar
  13. 13.
    Ferreira da Silva, R., Glatard, T.: A Science-Gateway Workload Archive to Study Pilot Jobs, User Activity, Bag of Tasks, Task Sub-Steps, and Workflow Executions. In: CoreGRID/ERCIM Workshop on Grids, Clouds and P2P Computing (2012)Google Scholar
  14. 14.
    Montagnat, J., et al.: A data-driven workflow language for grids based on array programming principles. In: WORKS 2009, Portland, USA, pp. 1–10. ACM (2009)Google Scholar
  15. 15.
    Glatard, T., et al.: Flexible and Efficient Workflow Deployment of Data-Intensive Applications on Grids with MOTEUR. IJHPCA 22(3), 347–360 (2008)Google Scholar
  16. 16.
    Tsaregorodtsev, A., et al.: DIRAC3. The New Generation of the LHCb Grid Software. Journal of Physics: Conference Series 219(6), 062029 (2009)CrossRefGoogle Scholar
  17. 17.
    Jan, S., et al.: GATE V6: a major enhancement of the GATE simulation platform enabling modelling of CT and radiotherapy. Phys. Med. Biol. 56(4), 881–901 (2011)CrossRefGoogle Scholar
  18. 18.
    Jensen, J., Svendsen, N.B.: Calculation of pressure fields from arbitrarily shaped, apodized, and excited ultrasound transducers. IEEE UFFC 39(2), 262–267 (1992)CrossRefGoogle Scholar
  19. 19.
    Reilhac, A., et al.: PET-SORTEO: Validation and Development of Database of Simulated PET Volumes. IEEE Trans. on Nuclear Science 52, 1321–1328 (2005)CrossRefGoogle Scholar
  20. 20.
    da Silva, R.F., Glatard, T., Desprez, F.: On-line, non-clairvoyant optimization of workflow activity granularity on grids. In: Wolf, F., Mohr, B., an Mey, D. (eds.) Euro-Par 2013. LNCS, vol. 8097, pp. 255–266. Springer, Heidelberg (2013)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Rafael Ferreira da Silva
    • 1
  • Tristan Glatard
    • 1
  • Frédéric Desprez
    • 2
  1. 1.CNRS, INSERM, CREATISUniversity of LyonVilleurbanneFrance
  2. 2.INRIA, University of Lyon, LIP, ENS LyonLyonFrance

Personalised recommendations