The Forgotten Factor: Facts on Performance Evaluation and Its Dependence on Workloads

  • Dror G. Feitelson
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2400)

Abstract

The performance of a computer system depends not only on its design and implementation, but also on the workloads it has to handle. Indeed, in some cases the workload can sway performance evaluation results. It is therefore crucially important that representative workloads be used for performance evaluation. This can be done by analyzing and modeling existing workloads. However, as more sophisticated workload models become necessary, there is an increasing need for the collection of more detailed data about workloads. This has to be done with an eye for those features that are really important.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    S-H. Chiang and M. K. Vernon, “Characteristics of a large shared memory production workload”. In Job Scheduling Strategies for Parallel Processing, D. G. Feitelson and L. Rudolph (eds.), pp. 159–187, Springer Verlag, 2001. Lect. Notes Comput. Sci. vol. 2221.CrossRefGoogle Scholar
  2. 2.
    W. Cirne and F. Berman, “A comprehensive model of the supercomputer workload”. In 4th Workshop on Workload Characterization, Dec 2001.Google Scholar
  3. 3.
    W. Cirne and F. Berman, “A model for moldable supercomputer jobs”. In 15th Intl. Parallel & Distributed Processing Symp., Apr 2001.Google Scholar
  4. 4.
    M. E. Crovella, “Performance evaluation with heavy tailed distributions”. In Job Scheduling Strategies for Parallel Processing, D. G. Feitelson and L. Rudolph (eds.), pp. 1–10, Springer Verlag, 2001. Lect. Notes Comput. Sci. vol. 2221.CrossRefGoogle Scholar
  5. 5.
    R. Cypher, A. Ho, S. Konstantinidou, and P. Messina, “A quantitative study of parallel scientific applications with explicit communication”. J. Supercomput. 10(1), pp. 5–24, 1996.CrossRefGoogle Scholar
  6. 6.
    A. B. Downey, “A parallel workload model and its implications for processor allocation”. In 6th Intl. Symp. High Performance Distributed Comput., Aug 1997.Google Scholar
  7. 7.
    A. B. Downey and D. G. Feitelson, “The elusive goal of workload characterization”. Performance Evaluation Rev. 26(4), pp. 14–29, Mar 1999.Google Scholar
  8. 8.
    D. G. Feitelson, Analyzing the Root Causes of Performance Evaluation Results. Technical Report 2002-4, School of Computer Science and Engineering, Hebrew University, Mar 2002.Google Scholar
  9. 9.
    D. G. Feitelson, “The effect of workloads on performance evaluation”. In Performance Evaluation of Complex Systems: Techniques and Tools, M. Calzarossa (ed.), Springer-Verlag, Sep 2002. Lect. Notes Comput. Sci. Tutorials.Google Scholar
  10. 10.
    D. G. Feitelson, “Memory usage in the LANL CM-5 workload”. In Job Scheduling Strategies for Parallel Processing, D. G. Feitelson and L. Rudolph (eds.), pp. 78–94, Springer Verlag, 1997. Lect. Notes Comput. Sci. vol. 1291.Google Scholar
  11. 11.
    D. G. Feitelson and B. Nitzberg, “Job characteristics of a production parallel scientific workload on the NASA Ames iPSC/860”. In Job Scheduling Strategies for Parallel Processing, D. G. Feitelson and L. Rudolph (eds.), pp. 337–360, Springer-Verlag, 1995. Lect. Notes Comput. Sci. vol. 949.Google Scholar
  12. 12.
    D. G. Feitelson and L. Rudolph, “Evaluation of design choices for gang scheduling using distributed hierarchical control”. J. Parallel & Distributed Comput. 35(1), pp. 18–34, May 1996.Google Scholar
  13. 13.
    D. G. Feitelson and L. Rudolph, “Metrics and benchmarking for parallel job scheduling”. In Job Scheduling Strategies for Parallel Processing, D. G. Feitelson and L. Rudolph (eds.), pp. 1–24, Springer-Verlag, 1998. Lect. Notes Comput. Sci. vol. 1459.CrossRefGoogle Scholar
  14. 14.
    S. Hotovy, “Workload evolution on the Cornell Theory Center IBM SP2”. In Job Scheduling Strategies for Parallel Processing, D. G. Feitelson and L. Rudolph (eds.), pp. 27–40, Springer-Verlag, 1996. Lect. Notes Comput. Sci. vol. 1162.CrossRefGoogle Scholar
  15. 15.
    J. Jann, P. Pattnaik, H. Franke, F. Wang, J. Skovira, and J. Riodan, “Modeling of workload in MPPs”. In Job Scheduling Strategies for Parallel Processing, D. G. Feitelson and L. Rudolph (eds.), pp. 95–116, Springer Verlag, 1997. Lect. Notes Comput. Sci. vol. 1291.Google Scholar
  16. 16.
    P. Krueger, T-H. Lai, and V. A. Dixit-Radiya, “Job scheduling is more important than processor allocation for hypercube computers”. IEEE Trans. Parallel & Distributed Syst. 5(5), pp. 488–497, May 1994.Google Scholar
  17. 17.
    V. Lo, J. Mache, and K. Windisch, “A comparative study of real workload traces and synthetic workload models for parallel job scheduling”. In Job Scheduling Strategies for Parallel Processing, D. G. Feitelson and L. Rudolph (eds.), pp. 25–46, Springer Verlag, 1998. Lect. Notes Comput. Sci. vol. 1459.CrossRefGoogle Scholar
  18. 18.
    U. Lublin and D. G. Feitelson, The Workload on Parallel Supercomputers: Modeling the Characteristics of Rigid Jobs. Technical Report 2001-12, Hebrew University, Oct 2001.Google Scholar
  19. 19.
    A. W. Mu’alem and D. G. Feitelson, “Utilization, predictability, workloads, and user runtime estimates in scheduling the IBM SP2 with backfilling”. IEEE Trans. Parallel & Distributed Syst. 12(6), pp. 529–543, Jun 2001.Google Scholar
  20. 20.
    T. D. Nguyen, R. Vaswani, and J. Zahorjan, “Parallel application characterization for multiprocessor scheduling policy design”. In Job Scheduling Strategies for Parallel Processing, D. G. Feitelson and L. Rudolph (eds.), pp. 175–199, Springer-Verlag, 1996. Lect. Notes Comput. Sci. vol. 1162.CrossRefGoogle Scholar
  21. 21.
    N. Nieuwejaar, D. Kotz, A. Purakayastha, C. S. Ellis, and M. L. Best, “File-access characteristics of parallel scientific workloads”. IEEE Trans. Parallel & Distributed Syst. 7(10), pp. 1075–1089, Oct 1996.Google Scholar
  22. 22.
    Parallel workloads archive. URL http://www.cs.huji.ac.il/labs/parallel/workload/.
  23. 23.
    J. P. Singh, J. L. Hennessy, and A. Gupta, “Scaling parallel programs for multiprocessors: methodology and examples”. Computer 26(7), pp. 42–50, Jul 1993.Google Scholar
  24. 24.
    E. Smirni and D. A. Reed, “Workload characterization of input/output intensive parallel applications”. In 9th Intl. Conf. Comput. Performance Evaluation, pp. 169–180, Springer-Verlag, Jun 1997. Lect. Notes Comput. Sci. vol. 1245.Google Scholar
  25. 25.
    J. S. Vetter and F. Mueller, “Communication characteristics of large-scale scientific applications for contemporary cluster architectures”. In 16th Intl. Parallel & Distributed Processing Symp., May 2002.Google Scholar
  26. 26.
    K. Y. Wang and D. C. Marinescu, “Correlation of the paging activity of individual node programs in the SPMD execution model”. In 28th Hawaii Intl. Conf. System Sciences, vol. I, pp. 61–71, Jan 1995.Google Scholar
  27. 27.
    K. Windisch, V. Lo, R. Moore, D. Feitelson, and B. Nitzberg, “A comparison of workload traces from two production parallel machines”. In 6th Symp. Frontiers Massively Parallel Comput., pp. 319–326, Oct 1996.Google Scholar
  28. 28.
    A. Wong, L. Oliker, W. Kramer, T. Kaltz, and D. Bailey, “System utilization benchmark on the Cray T3E and IBM SP2”. In Job Scheduling Strategies for Parallel Processing, D. G. Feitelson and L. Rudolph (eds.), pp. 56–67, Springer Verlag, 2000. Lect. Notes Comput. Sci. vol. 1911.CrossRefGoogle Scholar
  29. 29.
    S. C. Woo, M. Ohara, E. Torrie, J. P. Singh, and A. Gupta, “The SPLASH-2 programs: characterization and methodological considerations”. In 22nd Ann. Intl. Symp. Computer Architecture Conf. Proc., pp. 24–36, Jun 1995.Google Scholar
  30. 30.
    P. H. Worley, “The effect of time constraints on scaled speedup”. SIAM J. Sci. Statist. Comput. 11(5), pp. 838–858, Sep 1990.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2002

Authors and Affiliations

  • Dror G. Feitelson
    • 1
  1. 1.School of Computer Science and EngineeringThe Hebrew UniversityJerusalemIsrael

Personalised recommendations