Abstract
In this chapter, we provide an overview of benchmark standardization efforts in the area of computer systems benchmarking. We focus on SPEC and TPC, the two most prominent benchmark standardization bodies in the area of computer systems and information technology.
The reputation of current ‘benchmarketing’ claims regarding system performance is on par with the promises made by politicians during elections. (1993)
—Kaivalya M. Dixit (1942–2004), Long-time SPEC President
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Cao, P., Gowda, B., Lakshmi, S., Narasimhadevara, C., Nguyen, P., Poelman, J., et al. (2016). From BigBench to TPCx-BB: Standardization of a big data benchmark. In R. O. Nambiar & M. Poess (Eds.), Performance Evaluation and Benchmarking: Traditional – Big Data – Internet of Things. 8th TPC Technology Conference (TPCTC 2016), Revised Selected Papers. Lecture Notes in Computer Science (Vol. 10080, pp. 24–44). Berlin: Springer.
Dongarra, J. J., Luszczek, P., & Petitet, A. (2003). The LINPACK benchmark: Past, present and future. In Concurrency and Computation: Practice and Experience, 15(9), 803–820. John Wiley & Sons: Hoboken, New Jersey.
Gibson, J. C. (1970). The Gibson mix (Tech. rep. TR 00.2043). Poughkeepsie, NY: IBM Corp., Systems Development Division.
Gray, J. (Ed.). (1993). The benchmark handbook: For database and transaction systems. The Morgan Kaufmann series in data management systems (2nd ed.). Burlington, MA: Morgan Kaufmann.
Hennessy, J. L., & Patterson, D. A. (1990). Computer architecture: A quantitative approach. The Morgan Kaufmann series in computer architecture and design. Burlington, MA: Morgan Kaufmann.
Hogan, T. (2009). Overview of TPC benchmark E: The next generation of OLTP benchmarks. In R. O. Nambiar, M. Poess (Eds.), Performance Evaluation and Benchmarking – First TPC Technology Conference (TPCTC 2009), Revised Selected Papers. Lecture Notes in Computer Science (Vol. 5895, pp. 84–98). Berlin: Springer.
Huppler, K., & Johnson, D. (2014). TPC express – A new path for TPC benchmarks. In R. O. Nambiar, M. Poess (Eds.), Performance Characterization and Benchmarking – 5th TPC Technology Conference (TPCTC 2013), Revised Selected Papers. Lecture Notes in Computer Science (Vol. 8391, pp. 48–60). Berlin: Springer.
Lange, K.-D. (2009). Identifying shades of green: The SPECpower benchmarks. Computer, 42(3), 95–97. IEEE: Piscataway, NJ.
Lange, K.-D., & Tricker, M. G. (2011). The design and development of the server efficiency rating tool (SERT). In Proceedings of the 2nd ACM/SPEC International Conference on Performance Engineering (ICPE 2011), Karlsruhe (pp. 145–150). New York, NY: ACM.
Longbottom, R. (2004). Whetstone benchmark history and results. http://www.roylongbottom.org.uk/whetstone.htm
McCalpin, J. D. (1995). Memory bandwidth and machine balance in current high performance computers. IEEE Computer Society Technical Committee on Computer Architecture (TCCA) Newsletter (pp. 19–25).
Nambiar, R. O., & Poess, M. (2006). The making of TPC-DS. In Proceedings of the 32nd International Conference on Very Large Data Bases (VLDB 2006), Seoul. VLDB Endowment (pp. 1049–1058).
Nambiar, R. O., Poess, M., Dey, A., Cao, P., Magdon-Ismail, T., Ren, D., et al. (2014). Introducing TPCx-HS: The first industry standard for benchmarking big data systems. In R. O. Nambiar, M. Poess (Eds.), Performance Characterization and Benchmarking: Traditional to Big Data – 6th TPC Technology Conference (TPCTC 2014), Revised Selected Papers. Lecture Notes in Computer Science (Vol. 8904, pp. 1–12). Berlin: Springer.
Poess, M. (2017). Methodologies for a comprehensive approach to measuring the performance of decision support systems (Dissertation). Munich: Technische Universität München.
Poess, M., & Floyd, C. (2000). New TPC benchmarks for decision support and web commerce. ACM SIGMOD Record, 29(4), 64–71. ACM: New York, NY.
Poess, M., Nambiar, R. O., Kulkarni, K., Narasimhadevara, C., Rabl, T., & Jacobsen, H. (2018). Analysis of TPCx-IoT: The first industry standard benchmark for IoT gateway systems. In Proceedings of the 34th IEEE International Conference on Data Engineering (ICDE 2018), Paris (pp. 1519–1530). IEEE: Piscataway, New Jersey.
Poess, M., Rabl, T., & Jacobsen, H.-A. (2017). Analysis of TPC-DS: The first standard benchmark for SQL-based big data systems. In Proceedings of the 2017 Symposium on Cloud Computing (SoCC 2017), Santa Clara, CA (pp. 573–585). ACM: New York, NY.
Poess, M., Rabl, T., Jacobsen, H.-A., & Caufield, B. (2014). TPC-DI: The first industry benchmark for data integration. Proceedings of the VLDB Endowment (PVLDB), 7(13), 1367–1378. VLDB Endowment.
Raab, F. (1993). TPC-C: The standard benchmark for online transaction processing (OLTP). In J. Gray (Ed.), The benchmark handbook: For database and transaction systems. The Morgan Kaufmann series in data management systems (2nd ed.) Burlington, MA: Morgan Kaufmann.
Sachs, K., Kounev, S., Bacon, J., & Buchmann, A. (2009). Performance evaluation of message-oriented middleware using the SPECjms2007 benchmark. Performance Evaluation, 66(8), 410–434. Elsevier Science: Amsterdam.
Sethuraman, P. & Taheri, H. R. (2010). TPC-V: A benchmark for evaluating the performance of database applications in virtual environments. In R. O. Nambiar & M. Poess (Eds.), Performance Evaluation, Measurement and Characterization of Complex Systems – Second TPC Technology Conference (TPCTC 2010), Revised Selected Papers. Lecture Notes in Computer Science (Vol. 6417, pp. 121–135). Berlin: Springer.
Shanley, K. (1998). History and overview of the TPC. http://www.tpc.org/information/about/history.asp
Taheri, H. R., Little, G., Desai, B., Bond, A., Johnson, D., & Kopczynski, G. (2018). Characterizing the performance and resilience of HCI clusters with the TPCx-HCI benchmark. In R. O. Nambiar & M. Poess (Eds.), Performance Evaluation and Benchmarking for the Era of Artificial Intelligence – 10th TPC Technology Conference (TPCTC 2018), Revised Selected Papers. Lecture Notes in Computer Science (Vol. 11135, pp. 58–70). Berlin: Springer.
Wang, K., Bian, B., Cao, P., & Riess, M. (2017). Experiences and lessons in practice using TPCx-BB benchmarks. In R. O. Nambiar & M. Poess (Eds.), Performance Evaluation and Benchmarking for the Analytics Era – 9th TPC Technology Conference (TPCTC 2017), Revised Selected Papers. Lecture Notes in Computer Science (Vol. 10661, pp. 93–102). Berlin: Springer.
Weiss, A. R. (2002). Dhrystone benchmark: History, analysis, “scores” and recommendations. www.eembc.org/techlit/datasheets/dhrystone_wp.pdf
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Kounev, S., Lange, KD., Kistowski, J.v. (2020). Standardization. In: Systems Benchmarking. Springer, Cham. https://doi.org/10.1007/978-3-030-41705-5_9
Download citation
DOI: https://doi.org/10.1007/978-3-030-41705-5_9
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-41704-8
Online ISBN: 978-3-030-41705-5
eBook Packages: Computer ScienceComputer Science (R0)