On Grid Performance Evaluation Using Synthetic Workloads

  • Alexandru Iosup
  • Dick H. J. Epema
  • Carsten Franke
  • Alexander Papaspyrou
  • Lars Schley
  • Baiyi Song
  • Ramin Yahyapour
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4376)

Abstract

Grid computing is becoming a common platform for solving large scale computing tasks. However, a number of major technical issues, including the lack of adequate performance evaluation approaches, hinder the grid computing’s further development. The requirements herefore are manifold; adequate approaches must combine appropriate performance metrics, realistic workload models, and flexible tools for workload generation, submission, and analysis. In this paper we present an approach to tackle this complex problem. First, we introduce a set of grid performance objectives based on traditional and grid-specific performance metrics. Second, we synthesize the requirements for realistic grid workload modeling, e.g. co-allocation, data and network management, and failure modeling. Third, we show how GrenchMark, an existing framework for generating, running, and analyzing grid workloads, can be extended to implement the proposed modeling techniques. Our approach aims to be an initial and necessary step towards a common performance evaluation framework for grid environments.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    The Parallel Workloads Archive Team. The parallel workloads archive logs (June 2006), Available at http://www.cs.huji.ac.il/labs/parallel/workload/logs.html
  2. 2.
    Berman, F., et al.: Adaptive computing on the grid using apples. IEEE Trans. Parallel Distrib. Syst. 14(4), 369–382 (2003)CrossRefGoogle Scholar
  3. 3.
    Beume, N., et al.: Scheduling Algorithm Development based on Complex Owner Defined Objectives. Technical Report CI-190/05, University of Dortmund (January 2005)Google Scholar
  4. 4.
    Bucur, A.I.D., Epema, D.H.J.: The performance of processor co-allocation in multicluster systems. In: Proc. of the 3rd IEEE Int’l. Symp. on Cluster Computing and the Grid (CCGrid), pp. 302–309. IEEE Computer Society Press, Los Alamitos (2003)CrossRefGoogle Scholar
  5. 5.
    Bucur, A.I.D., Epema, D.H.J.: Trace-based simulations of processor co-allocation policies in multiclusters. In: Proc. of the 12th Intl. Symposium on High-Performance Distributed Computing (HPDC), pp. 70–79 (2003)Google Scholar
  6. 6.
    Buyya, R., Abramson, D., Venugopal, S.: The grid economy. In: Special Issue of the Proceedings of the IEEE on Grid Computing. IEEE Press, Los Alamitos (To appear, 2005)Google Scholar
  7. 7.
    Calzarossa, M., Serazzi, G.: A characterization of the variation in time of workload arrival patterns. IEEE Trans. Comput. 34(2), 156–162 (1985)CrossRefGoogle Scholar
  8. 8.
    Chapin, S.J., et al.: Benchmarks and standards for the evaluation of parallel job schedulers. In: Feitelson, D.G., Rudolph, L. (eds.) JSSPP 1999, IPPS-WS 1999, and SPDP-WS 1999. LNCS, vol. 1659, pp. 67–90. Springer, Heidelberg (1999)CrossRefGoogle Scholar
  9. 9.
    Chun, B.N., et al.: Mirage: A microeconomic resource allocation system for sensornet testbeds. In: Proc. of 2nd IEEE Workshop on Embedded Networked Sensors (EmNetsII), IEEE Computer Society Press, Los Alamitos (2005)Google Scholar
  10. 10.
    Chun, G., et al.: Benchmark probes for grid assessment. In: Proc. of the 18th International Parallel and Distributed Processing Symposium (IPDPS) (2004)Google Scholar
  11. 11.
    Cirne, W., Berman, F.: A Comprehensive Model of the Supercomputer Workload. In: 4th Workshop on Workload Characterization (December 2001)Google Scholar
  12. 12.
    Cirne, W., Berman, F.: A model for moldable supercomputer jobs. In: Proc. of the 15th International Parallel and Distributed Processing Symposium (IPDPS), pp. 59–79 (2001)Google Scholar
  13. 13.
    Downey, A.B.: A parallel workload model and its implications for processor allocation. Cluster Computing 1(1), 133–145 (1998)CrossRefGoogle Scholar
  14. 14.
    Dumitrescu, C., Raicu, I., Foster, I.T.: Experiences in running workloads over grid3. In: Zhuge, H., Fox, G.C. (eds.) GCC 2005. LNCS, vol. 3795, pp. 274–286. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  15. 15.
    Ernemann, C., Yahyapour, R.: Applying Economic Scheduling Methods to Grid Environments. In: Ernemann, C., Yahyapour, R. (eds.) Grid Resource Management - State of the Art and Future Trends, pp. 491–506. Kluwer Academic Publishers, Dordrecht (2003)Google Scholar
  16. 16.
    Ernemann, C., et al.: On advantages of grid computing for parallel job scheduling. In: Proc. of the 2nd IEEE Int’l. Symp. on Cluster Computing and the Grid (CCGrid), pp. 39–49. IEEE Computer Society Press, Los Alamitos (2002)CrossRefGoogle Scholar
  17. 17.
    Ernemann, C., Hamscher, V., Yahyapour, R.: Benefits of global grid computing for job scheduling. In: Proceedings of the 5th IEEE/ACM International Workshop on Grid Computing, Pittsburgh, November 2004, IEEE Computer Society, Los Alamitos (2004)Google Scholar
  18. 18.
    Ernemann, C., Song, B., Yahyapour, R.: Scaling of Workload Traces. In: Feitelson, D.G., Rudolph, L., Schwiegelshohn, U. (eds.) JSSPP 2003. LNCS, vol. 2862, pp. 166–183. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  19. 19.
    Feitelson, D.G.: Packing Schemes for Gang Scheduling. In: Feitelson, D.G., Rudolph, L. (eds.) Job Scheduling Strategies for Parallel Processing. LNCS, vol. 1162, pp. 89–110. Springer, Heidelberg (1996)CrossRefGoogle Scholar
  20. 20.
    Feitelson, D.G.: The forgotten factor: facts on performance evaluation and its dependence on workloads. In: Monien, B., Feldmann, R.L. (eds.) Euro-Par 2002. LNCS, vol. 2400, pp. 49–60. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  21. 21.
    Feitelson, D.G.: Workload Modeling for Performance Evaluation. In: Calzarossa, M.C., Tucci, S. (eds.) Performance 2002. LNCS, vol. 2459, pp. 114–141. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  22. 22.
    Feitelson, D.G.: Metric and workload effects on computer systems evaluation. IEEE Computer 36(9), 18–25 (2003)Google Scholar
  23. 23.
    Feitelson, D.G.: Experimental analysis of the root causes of performance evaluation results: a backfilling case study. IEEE Transactions on Parallel and Distributed Systems 16(2), 175–182 (2005)CrossRefGoogle Scholar
  24. 24.
    Feitelson, D.G., Rudolph, L.: Metrics and benchmarking for parallel job scheduling. In: Feitelson, D.G., Rudolph, L. (eds.) Job Scheduling Strategies for Parallel Processing. LNCS, vol. 1459, pp. 1–24. Springer, Heidelberg (1998)CrossRefGoogle Scholar
  25. 25.
    Feitelson, D.G., et al.: Theory and Practice in Parallel Job Scheduling. In: Feitelson, D.G., Rudolph, L. (eds.) Job Scheduling Strategies for Parallel Processing. LNCS, vol. 1291, pp. 1–34. Springer, Heidelberg (1997)Google Scholar
  26. 26.
    Foster, I., et al.: End-to-end quality of service for high-end applications. Computer Communications 27(14), 1375–1388 (2004)CrossRefGoogle Scholar
  27. 27.
    Frumkin, M.A., Van der Wijngaart, R.F.: Nas grid benchmarks: A tool for grid space exploration. In: Proc. of the 10th Intl. Symposium on High-Performance Distributed Computing (HPDC), pp. 315–326 (2001)Google Scholar
  28. 28.
    Graham, R.L., et al.: Optimization and approximation in deterministic sequencing and scheduling: A survey. Annals of Discrete Mathematics 15, 287–326 (1979)MathSciNetCrossRefGoogle Scholar
  29. 29.
    Hwang, S., Kesselman, C.: Gridworkflow: A flexible failure handling framework for the grid. In: Proc. of the 12th Intl. Symposium on High-Performance Distributed Computing (HPDC), pp. 126–137 (2003)Google Scholar
  30. 30.
    Iosup, A., et al.: How are real grids used? the analysis of four grid traces and its implications. In: The 7th IEEE/ACM International Conference on Grid Computing (Grid), Barcelona, ES, Sept. 28-29 (accepted, 2006)Google Scholar
  31. 31.
    Iosup, A., Epema, D.H.J.: GrenchMark: A framework for analyzing, testing, and comparing grids. In: Proc. of the 6th IEEE/ACM Int’l. Symp. on Cluster Computing and the GRID (CCGrid), May (accepted, 2006)Google Scholar
  32. 32.
    Iosup, A., et al.: Synthetic grid workloads with Ibis, KOALA, and GrenchMark. In: Dumke, R.R., Abran, A. (eds.) IWSM 2000. LNCS, vol. 2006, Springer, Heidelberg (2001)Google Scholar
  33. 33.
    Jackson, D., Snell, Q., Clement, M.: Core algorithms of the Maui scheduler. In: Feitelson, D.G., Rudolph, L. (eds.) JSSPP 2001. LNCS, vol. 2221, pp. 87–102. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  34. 34.
    Jain, R.: The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling (Winner of “1991 Best Advanced How-To Book, Systems” award from the Computer Press Association). Wiley-Interscience, New York (May 1991)MATHGoogle Scholar
  35. 35.
    Kenyon, C., Cheliotis, G.: Architecture requirements for commercializing grid resources. In: Proc. of the 11th Intl. Symposium on High-Performance Distributed Computing (HPDC), pp. 215–224 (2002)Google Scholar
  36. 36.
    Kola, G., Kosar, T., Livny, M.: Phoenix: Making data-intensive grid applications fault-tolerant. In: Buyya, R. (ed.) GRID, pp. 251–258. IEEE Computer Society Press, Los Alamitos (2004)Google Scholar
  37. 37.
    Kondo, D., et al.: Characterizing and evaluating desktop grids: An empirical study. In: Proc. of the 18th International Parallel and Distributed Processing Symposium (IPDPS) (2004)Google Scholar
  38. 38.
    Li, H., Groep, D., Wolters, L.: Workload characteristics of a multi-cluster supercomputer. In: Feitelson, D.G., Rudolph, L., Schwiegelshohn, U. (eds.) JSSPP 2004. LNCS, vol. 3277, pp. 176–194. Springer, Heidelberg (2005)Google Scholar
  39. 39.
    Lublin, U., Feitelson, D.G.: The workload on parallel supercomputers: Modeling the characteristics of rigid jobs. Journal of Parallel and Distributed Computing 63(20), 1105–1122 (2003)MATHCrossRefGoogle Scholar
  40. 40.
    Medeiros, R., et al.: Faults in grids: Why are they so bad and what can be done about it? In: Stockinger, H. (ed.) GRID, pp. 18–24. IEEE Computer Society Press, Los Alamitos (2003)Google Scholar
  41. 41.
    Medernach, E.: Workload analysis of a cluster in a grid environment. In: Feitelson, D.G., et al. (eds.) JSSPP 2005. LNCS, vol. 3834, pp. 36–61. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  42. 42.
    Mohamed, H.H., Epema, D.H.J.: Experiences with the koala co-allocating scheduler in multiclusters. In: Proc. of the 5th IEEE/ACM Int’l Symp. on Cluster Computing and the GRID (CCGrid), May 2005, ACM Press, New York (2005)Google Scholar
  43. 43.
    Pinedo, M.: Scheduling: Theory, Algorithms, and Systems, 2nd edn. Prentice-Hall, Englewood Cliffs (2002)MATHGoogle Scholar
  44. 44.
    Schroeder, W., et al.: Analysis of HPSS Performance Based on Per-file Transfer Logs. In: Proc. of the 16th IEEE Mass Storage Systems Symposium, San Diego, March 1999, pp. 103–115. IEEE Computer Society Press, Los Alamitos (1999)Google Scholar
  45. 45.
    Schwiegelshohn, U., Yahyapour, R.: Fairness in parallel job scheduling. Journal of Scheduling 3(5), 297–320 (2000)MATHCrossRefMathSciNetGoogle Scholar
  46. 46.
    Shan, H., Oliker, L., Biswas, R.: Job superscheduler architecture and performance in computational grid environments. In: SC, pp. 44–54. ACM Press, New York (2003)Google Scholar
  47. 47.
    Song, B., Ernemann, C., Yahyapour, R.: Parallel Computer Workload Modeling with Markov Chains. In: Feitelson, D.G., Rudolph, L., Schwiegelshohn, U. (eds.) JSSPP 2004. LNCS, vol. 3277, pp. 47–62. Springer, Heidelberg (2005)Google Scholar
  48. 48.
    Song, B., Ernemann, C., Yahyapour, R.: User Group-based Workload Analysis and Modeling. In: Proc. of the 5th Int’l. Symp. on Cluster Computing and the Grid (CCGrid), IEEE Computer Society Press, Los Alamitos (2005)Google Scholar
  49. 49.
    Thain, D., et al.: Pipeline and batch sharing in grid workloads. In: Proc. of the 12th Intl. Symposium on High-Performance Distributed Computing (HPDC), pp. 152–161 (2003)Google Scholar
  50. 50.
    Theys, M.D., et al.: A Mathematical Model, Heuristic and Simulation Study for a Basic Data Staging Problem in a Heterogenous Networking Environment. In: Proc. of the 7th Heterogeneous Computing Workshop, Orlando, March 1998, pp. 115–122. IEEE, Los Alamitos (1998)Google Scholar
  51. 51.
    Tsouloupas, G., Dikaiakos, M.D.: GridBench: A workbench for grid benchmarking. In: Sloot, P.M.A., et al. (eds.) EGC 2005. LNCS, vol. 3470, pp. 211–225. Springer, Heidelberg (2005)Google Scholar
  52. 52.
    Wolski, R., Spring, N., Hayes, J.: The network weather service: A distributed resource performance forecasting service for metacomputing. Journal of Future Generation Computing Systems 15(5-6), 757–768 (1999)CrossRefGoogle Scholar

Copyright information

© Springer Berlin Heidelberg 2007

Authors and Affiliations

  • Alexandru Iosup
    • 1
  • Dick H. J. Epema
    • 1
  • Carsten Franke
    • 2
  • Alexander Papaspyrou
    • 2
  • Lars Schley
    • 2
  • Baiyi Song
    • 2
  • Ramin Yahyapour
    • 2
  1. 1.Faculty of Electrical Engineering, Mathematics, and Computer Science, Delft University of TechnologyThe Netherlands
  2. 2.Information Technology Section, Robotics Research Institute, Dortmund UniversityGermany

Personalised recommendations