Abstract
After a definition (list of properties) of a benchmark, the major benchmarks currently used are classified according to several criteria: Ownership, definition of the benchmark, pricing, area covered by the benchmark. The SPEC Open Systems Group, TPC and SAP benchmarks are discussed in more detail. The use of benchmarks in academic research is discussed. Finally, some current issues in benchmarking are listed that users of benchmark results should be aware of.
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Yin Chan, Ashok Sundarsanam, and Andrew Wolfe: The Effect of Compiler-Flag Tuning in SPEC Benchmark Performance. ACM Computer Architecture News 22,4 (Sept. 1994), 60–70
Jeffrey D. Gee, Mark D. Hill, Dionisios N. Pnevmatikos, and Alan Jay Smith: Cache Performance of the SPEC92 Benchmark Suite. IEEE Micro, August 1993, 17–27
Jim Gray: The Benchmark Handbook for Database and Transaction Processing Systems. Morgan Kaufmann, San Mateo, 2nd Edition, 1993, 592 pages
John Hennessy and David Patterson: Computer Architecture. A Quantitative Approach. 2nd Edition, Morgan Kaufmann, San Francisco 1996, 760 pages plus appendixes
John L. Henning: SPEC CPU2000: Measuring CPU Performance in the New Millenium. Computer (IEEE) 33,7 (July 2000), 28–35
Dennis C. Lee, et al: Execution Characteristics of Desktop Applications on Windows NT. 25th Annual Symposium on Computer Architecture, = ACM Computer Architecture News 26,3 June 1998), 27–38
John McCalpin: http://www.cs.virginia.edu/stream/ref.html: FAQ list (Frequently Asked Questions) about Stream
Nikki Mirghafori, Margret Jacoby, and David Patterson: Truth in SPEC Benchmarks. ACM Computer Architecture News 23,5 (Dec. 1995), 34–42
Shubhendu S. Mukherjee: New Challenges in Benchmarking Future Processors. Fifth Workshop on Computer Architecture Evaluation using Commercial Workloads, 2002
Reinhold P. Weicker: An Overview of Common Benchmarks. Computer (IEEE) 23, 12 (Dec. 1990), 65–75
Reinhold Weicker: On the Use of SPEC Benchmarks in Computer Architecture Research. ACM Computer Architecture News 25,1 (March 1997), 19–22
Reinhold Weicker: Point: Why SPEC Should Burn (Almost) All Flags, and Walter Bays. Counterpoint: Defending the Flag. SPEC Newsletter 7,4, Dec. 1995, 5–6
Craig B. Zilles: Benchmark HEALTH Considered Harmful. ACM Computer Architecture News 29,3 (June 2001), 4–5
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2002 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Weicker, R. (2002). Benchmarking. In: Calzarossa, M.C., Tucci, S. (eds) Performance Evaluation of Complex Systems: Techniques and Tools. Performance 2002. Lecture Notes in Computer Science, vol 2459. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-45798-4_9
Download citation
DOI: https://doi.org/10.1007/3-540-45798-4_9
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-44252-3
Online ISBN: 978-3-540-45798-5
eBook Packages: Springer Book Archive