Advertisement

Cluster Computing

, Volume 13, Issue 4, pp 421–434 | Cite as

Failure-aware workflow scheduling in cluster environments

  • Zhifeng Yu
  • Chenjia Wang
  • Weisong Shi
Article

Abstract

The goal of workflow application scheduling is to achieve minimal makespan for each workflow. Scheduling workflow applications in high performance cluster environments is an NP-Complete problem, and becomes more complicated when potential resource failures are considered. While more research on failure prediction has been witnessed in recent years to improve system availability and reliability, very few of them attack the problem in the context of workflow application scheduling. In this paper, we study how a workflow scheduler benefits from failure prediction and propose FLAW, a failure-aware workflow scheduling algorithm. We propose two important definitions on accuracy, Application Oblivious Accuracy (AOA) and Application Aware Accuracy (AAA), from the perspectives of system and scheduling respectively, as we observe that the prediction accuracy defined conventionally imposes different performance implications on different applications and fails to measure how that improves scheduling effectiveness. The comprehensive evaluation results using real failure traces show that FLAW performs well with practically achievable prediction accuracy by reducing the average makespan, the loss time and the number of job rescheduling.

Keywords

Failure-aware Workflow scheduling Cluster computing 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Garey, M., Johnson, D.: Computers and Intractibility: A Guide to the Theory of NP-completeness. Freeman, San Francisco (1979) zbMATHGoogle Scholar
  2. 2.
    Open science grid. [Online]. Available: http://www.opensciencegrid.org/
  3. 3.
    Nsf taragrid. [Online]. Available: http://www.teragrid.org/
  4. 4.
    Yang, L., Schopf, J., Foster, I.: Anomaly detection and diagnosis in grid environments. In: SC’07: Proceedings of the 2007 ACM/IEEE Conference on Supercomputing. IEEE Computer Society, Washington (2007) Google Scholar
  5. 5.
    Liang, Y., Sivasubramaniam, A., Moreira, J.: Filtering failure logs for a bluegene/l prototype. In: Proceedings of the 2005 International Conference on Dependable Systems and Networks (DSN’05), pp. 476–485. IEEE Computer Society, Washington (2005) CrossRefGoogle Scholar
  6. 6.
    Fu, S., Xu, C.: Exploring event correlation for failure prediction in coalitions of clusters. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC’07) (2007) Google Scholar
  7. 7.
    Oppenheimer, D., et al.: Service placement in shared wide-area platforms. In: Proceedings of the Twentieth ACM Symposium on Operating Systems Principles (SOSP’05), p. 1. ACM, New York (2005) Google Scholar
  8. 8.
    Zhang, Y., et al.: Performance implications of failures in large-scale cluster scheduling. In: Proceedings of 10th International WorkshopJob Scheduling Strategies for Parallel Processing (JSSPP’04), pp. 233–252 (2004) Google Scholar
  9. 9.
    Schroeder, B., Gibson, G.: A large-scale study of failures in high-performance computing systems. In: Proceedings of the International Conference on Dependable Systems and Networks (DSN’06), pp. 249–258. IEEE Computer Society, Washington (2006) CrossRefGoogle Scholar
  10. 10.
    Yalagandula, P., et al.: Beyond availability: Towards a deeper understanding of machine failure characteristics in large distributed systems. In: Proceedings of the Workshop on Real, Large Distributed Systems (WORLDS’04) (2004) Google Scholar
  11. 11.
    Ren, X., et al.: Prediction of resource availability in fine-grained cycle sharing systems empirical evaluation. J. Grid Comput. 5(2), 173–195 (2007) CrossRefGoogle Scholar
  12. 12.
    Salfner, F., Schieschke, M., Malek, M.: Predicting failures of computer systems: a case study for a telecommunication system. In: Proceedings of the 20th International Parallel and Distributed Processing Symposium (IPDPS 2006) (2006) Google Scholar
  13. 13.
    Li, Y., Lan, Z.: Exploit failure prediction for adaptive fault-tolerance in cluster computing. In: Proceedings of the Sixth IEEE International Symposium on Cluster Computing and the Grid (CCGRID’06), pp. 531–538. IEEE Computer Society, Washington (2006) Google Scholar
  14. 14.
    Li, Y., et al.: Fault-driven re-scheduling for improving system-level fault resilience. In: Proceedings of the 2007 International Conference on Parallel Processing (ICPP’07), p. 39. IEEE Computer Society, Washington (2007) CrossRefGoogle Scholar
  15. 15.
    Oliner, A., et al.: Fault-aware job scheduling for bluegene/l systems. In: Proceedings of the 18th International Parallel and Distributed Processing Symposium (IPDPS’04). IEEE Computer Society, Washington (2004) Google Scholar
  16. 16.
    Hwang, S., Kesselman, C.: Gridworkflow: A flexible failure handling framework for the grid. In: Proceedings of the 12th IEEE International Symposium on High Performance Distributed Computing (HPDC’03), p. 126. IEEE Computer Society, Washington (2003) CrossRefGoogle Scholar
  17. 17.
    Abawajy, J.H.: Fault-tolerant scheduling policy for grid computing systems. In: Proceedings of the 18th International Parallel and Distributed Processing Symposium (IPDPS’04). IEEE Computer Society, Washington (2004) Google Scholar
  18. 18.
    Dagman. [Online]. Available: http://www.cs.wisc.edu/condor/dagman/
  19. 19.
    Dogan, A., Özgüner, F.: Matching and scheduling algorithms for minimizing execution time and failure probability of applications in heterogeneous computing. IEEE Trans. Parallel Distrib. Syst. 13(3), 308–323 (2002) CrossRefGoogle Scholar
  20. 20.
    Deelman, E., Blythe, J., Gil, Y., Kesselman, C.: Workflow management in griphyn. In: Grid Resource Management: State of the Art and Future Trends, pp. 99–116. Kluwer Academic, Norwell (2004) Google Scholar
  21. 21.
    Planet lab. [Online]. Available: http://www.planet-lab.org
  22. 22.
    Yu, Z., Shi, W.: A planner-guided scheduling strategy for multiple grid workflow applications. In: Proceeding of Fourth International Workshop on Scheduling and Resource Management for Parallel and Distributed Systems (SRMPDS ’08), Portland, Oregon, USA, September 2008 Google Scholar
  23. 23.
    Los Alamos National Laboratory. Operational data to support and enable computer science research (2006). [Online]. Available: http://institutes.lanl.gov/data/fdata/
  24. 24.
    Salfner, F., Malek, M.: Proactive fault handling for system availability enhancement. In: Proceedings of the 19th IEEE International Parallel and Distributed Processing Symposium (IPDPS’05)—Workshop 16, p. 281.1. IEEE Computer Society, Washington (2005) Google Scholar
  25. 25.
    Yu, Z., Shi, W.: An adaptive rescheduling strategy for grid workflow applications. In: Proceeding of 21st International Parallel and Distributed Processing Symposium (IPDPS’07), Long Beach, Florida, USA, March 2007 Google Scholar
  26. 26.
    Topcuouglu, H., Hariri, S., Wu, M.: Performance-effective and low-complexity task scheduling for heterogeneous computing. IEEE Trans. Parallel Distrib. Syst. 13(3), 260–274 (2002) CrossRefGoogle Scholar
  27. 27.
    Oliner, A., Sahoo, R., Moreira, J., Gupta, M.: Performance implications of periodic checkpointing on large-scale cluster systems. In: Proceedings of the 19th IEEE International Parallel and Distributed Processing Symposium (IPDPS’05), p. 299.2. IEEE Computer Society, Washington (2005) Google Scholar
  28. 28.
    Schroeder, B., Gibson, G.: Understanding failures in petascale computers. J. Phys., Condens. Matter 19(45) (2007) Google Scholar
  29. 29.
    Hönig, U., Schiffmann, W.: A comprehensive test bench for the evaluation of scheduling heuristics. In: Proceedings of the 16th International Conference on Parallel and Distributed Computing and Systems (PDCS’04). IEEE, New York (2004) Google Scholar
  30. 30.
    Canon, L.-C., Jeannot, E., Sakellariou, R., Zheng, W.: Comparative evaluation of the robustness of dag scheduling heuristics. In: Integration Research in Grid Computing, CoreGRID Integration Workshop, pp. 63–74. Crete University Press, Heraklion (2008) Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2010

Authors and Affiliations

  1. 1.Wayne State UniversityDetroitUSA

Personalised recommendations