Benchmarking

  • Reinhold Weicker
Conference paper
First Online:
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2459)

Abstract

After a definition (list of properties) of a benchmark, the major benchmarks currently used are classified according to several criteria: Ownership, definition of the benchmark, pricing, area covered by the benchmark. The SPEC Open Systems Group, TPC and SAP benchmarks are discussed in more detail. The use of benchmarks in academic research is discussed. Finally, some current issues in benchmarking are listed that users of benchmark results should be aware of.

Keywords

Benchmark Suite Benchmark Result Software Vendor Secure Socket Layer Mail Server 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
Download to read the full conference paper text

References

  1. 1.
    Yin Chan, Ashok Sundarsanam, and Andrew Wolfe: The Effect of Compiler-Flag Tuning in SPEC Benchmark Performance. ACM Computer Architecture News 22,4 (Sept. 1994), 60–70CrossRefGoogle Scholar
  2. 2.
    Jeffrey D. Gee, Mark D. Hill, Dionisios N. Pnevmatikos, and Alan Jay Smith: Cache Performance of the SPEC92 Benchmark Suite. IEEE Micro, August 1993, 17–27Google Scholar
  3. 3.
    Jim Gray: The Benchmark Handbook for Database and Transaction Processing Systems. Morgan Kaufmann, San Mateo, 2nd Edition, 1993, 592 pagesMATHGoogle Scholar
  4. 4.
    John Hennessy and David Patterson: Computer Architecture. A Quantitative Approach. 2nd Edition, Morgan Kaufmann, San Francisco 1996, 760 pages plus appendixesMATHGoogle Scholar
  5. 5.
    John L. Henning: SPEC CPU2000: Measuring CPU Performance in the New Millenium. Computer (IEEE) 33,7 (July 2000), 28–35CrossRefGoogle Scholar
  6. 6.
    Dennis C. Lee, et al: Execution Characteristics of Desktop Applications on Windows NT. 25th Annual Symposium on Computer Architecture, = ACM Computer Architecture News 26,3 June 1998), 27–38Google Scholar
  7. 7.
    John McCalpin: http://www.cs.virginia.edu/stream/ref.html: FAQ list (Frequently Asked Questions) about Stream
  8. 8.
    Nikki Mirghafori, Margret Jacoby, and David Patterson: Truth in SPEC Benchmarks. ACM Computer Architecture News 23,5 (Dec. 1995), 34–42CrossRefGoogle Scholar
  9. 9.
    Shubhendu S. Mukherjee: New Challenges in Benchmarking Future Processors. Fifth Workshop on Computer Architecture Evaluation using Commercial Workloads, 2002Google Scholar
  10. 11.
    Reinhold P. Weicker: An Overview of Common Benchmarks. Computer (IEEE) 23, 12 (Dec. 1990), 65–75CrossRefGoogle Scholar
  11. 12.
    Reinhold Weicker: On the Use of SPEC Benchmarks in Computer Architecture Research. ACM Computer Architecture News 25,1 (March 1997), 19–22CrossRefGoogle Scholar
  12. 13.
    Reinhold Weicker: Point: Why SPEC Should Burn (Almost) All Flags, and Walter Bays. Counterpoint: Defending the Flag. SPEC Newsletter 7,4, Dec. 1995, 5–6Google Scholar
  13. 14.
    Craig B. Zilles: Benchmark HEALTH Considered Harmful. ACM Computer Architecture News 29,3 (June 2001), 4–5CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2002

Authors and Affiliations

  • Reinhold Weicker
    • 1
  1. 1.Fujitsu Siemens ComputersPaderbornGermany

Personalised recommendations