Skip to main content
SpringerLink
Log in

Workload Characteristics of a Multi-cluster Supercomputer

  • Conference paper
Job Scheduling Strategies for Parallel Processing (JSSPP 2004)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 3277))

Included in the following conference series:

Abstract

This paper presents a comprehensive characterization of a multi-cluster supercomputer workload using twelve-month scientific research traces. Metrics that we characterize include system utilization, job arrival rate and interarrival time, job cancellation rate, job size (degree of parallelism), job runtime, memory usage, and user/group behavior. Correlations between metrics (job runtime and memory usage, requested and actual runtime, etc) are identified and extensively studied. Differences with previously reported workloads are recognized and statistical distributions are fitted for generating synthetic workloads with the same characteristics. This study provides a realistic basis for experiments in resource management and evaluations of different scheduling strategies in a multi-cluster research environment.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Calzarossa, M., Serazzi, G.: Workload characterization: A survey. Proc. IEEE 81(8), 1136–1150 (1993)

    Article  Google Scholar 

  2. Feitelson, D.G.: Workload modeling for performance evaluation. In: Calzarossa, M.C., Tucci, S. (eds.) Performance 2002. LNCS, vol. 2459, pp. 114–141. Springer, Heidelberg (2002)

    Chapter  Google Scholar 

  3. Parallel Workload Archive, http://www.cs.huji.ac.il/labs/parallel/workload/

  4. Chiang, S.-H., Vernon, M.K.: Characteristics of a large shared memory production workload. In: Feitelson, D.G., Rudolph, L. (eds.) JSSPP 2001. LNCS, vol. 2221, pp. 159–187. Springer, Heidelberg (2001)

    Chapter  Google Scholar 

  5. Feitelson, D., Nitzberg, B.: Job characteristics of a production parallel scientific workload on the NASA ames iPSC/860. In: Feitelson, D.G., Rudolph, L. (eds.) IPPS-WS 1995 and JSSPP 1995. LNCS, vol. 949, pp. 337–360. Springer, Heidelberg (1995)

    Google Scholar 

  6. Windisch, K., Lo, V., Moore, R., Feitelson, D., Nitzberg, B.: A comparison of workload traces from two production parallel machines. In: 6th Symp. Frontiers Massively Parallel Comput., pp. 319–326 (1996)

    Google Scholar 

  7. Lublin, U., Feitelson, D.G.: The workload on parallel supercomputers: modeling the characteristics of rigid jobs. J. Parallel and Distributed Comput. 63(11), 1105–1122 (2003)

    Article  MATH  Google Scholar 

  8. Jann, J., Pattnaik, P., Franke, H., Wang, F., Skovira, J., Riodan, J.: Modeling of workload in MPPs. In: Feitelson, D.G., Rudolph, L. (eds.) IPPS-WS 1997 and JSSPP 1997. LNCS, vol. 1291, pp. 95–116. Springer, Heidelberg (1997)

    Google Scholar 

  9. Cirne, W., Berman, F.: A comprehensive model of the supercomputer workload. In: IEEE 4th Annual Workshop on Workload Characterization (2001)

    Google Scholar 

  10. Chapin, S.J., Cirne, W., Feitelson, D.G., Jones, J.P., Leutenegger, S.T., Schwiegelshohn, U., Smith, W., Talby, D.: Benchmarks and standards for the evaluation of parallel job schedulers. In: Feitelson, D.G., Rudolph, L. (eds.) JSSPP 1999, IPPS-WS 1999, and SPDP-WS 1999. LNCS, vol. 1659, pp. 67–90. Springer, Heidelberg (1999)

    Chapter  Google Scholar 

  11. Downey, B., Feitelson, D.G.: The elusive goal of workload characterization. Perf. Eval. Rev. 26(4), 14–29 (1999)

    Article  Google Scholar 

  12. The DAS-2 Supercomputer, http://www.cs.vu.nl/das2

  13. Banen, S., Bucur, A., Epema, D.H.J.: A Measurement-Based Simulation Study of Processor Co-Allocation in Multicluster Systems. In: Feitelson, D.G., Rudolph, L., Schwiegelshohn, U. (eds.) JSSPP 2003. LNCS, vol. 2862, pp. 105–128. Springer, Heidelberg (2003)

    Chapter  Google Scholar 

  14. Portable Batch System, http://www.openpbs.org

  15. The Maui Scheduler, http://www.supercluster.org

  16. The Globus project, http://www.globus.org

  17. Allen, O.: Probability, Statistics, and Queueing Theory with Computer Science Applications. Academic Press, London (1978)

    MATH  Google Scholar 

  18. Khayari, R.E.A., Sadre, R., Haverkort, B.R.: Fitting world-wide web request traces with the EM-algorithm. Performance Evaluation 52, 175–191 (2003)

    Article  Google Scholar 

  19. Matlab, http://www.mathworks.com

  20. Dataplot, http://www.itl.nist.gov/div898/software/dataplot/

  21. The EMpht programme, http://www.maths.lth.se/matstat/staff/asmus/pspapers.html

  22. Downey, A.B.: Using Queue Time Predictions for Processor Allocation. In: Feitelson, D.G., Rudolph, L. (eds.) IPPS-WS 1997 and JSSPP 1997. LNCS, vol. 1291, pp. 35–57. Springer, Heidelberg (1997)

    Google Scholar 

  23. Feitelson, D.G.: Memory usage in the LANL CM-5 Workload. In: Feitelson, D.G., Rudolph, L. (eds.) IPPS-WS 1997 and JSSPP 1997. LNCS, vol. 1291, pp. 78–94. Springer, Heidelberg (1997)

    Google Scholar 

  24. Calzarossa, M., Serazzi, G.: Construction and use of multiclass workload models. Performance Evaluation 19(4), 341–352 (1994)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Leiden Institute of Advanced Computer Science (LIACS), Leiden University, The Netherlands

    Hui Li & Lex Wolters

  2. National Institute for Nuclear and High Energy Physics (NIKHEF), The Netherlands

    David Groep

Authors
  1. Hui Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. David Groep
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lex Wolters
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Computer Science, The Hebrew University of Jerusalem,  

    Dror G. Feitelson

  2. Massachusetts Institute of Technology, 77 Massachusetts Avenue, MA 02139, Cambridge, USA

    Larry Rudolph

  3. No Affiliations,  

    Uwe Schwiegelshohn

Rights and permissions

Reprints and permissions

Copyright information

© 2005 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Li, H., Groep, D., Wolters, L. (2005). Workload Characteristics of a Multi-cluster Supercomputer. In: Feitelson, D.G., Rudolph, L., Schwiegelshohn, U. (eds) Job Scheduling Strategies for Parallel Processing. JSSPP 2004. Lecture Notes in Computer Science, vol 3277. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11407522_10

Download citation

  • DOI: https://doi.org/10.1007/11407522_10

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-25330-3

  • Online ISBN: 978-3-540-31795-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation