Advertisement

© 2020

Systems Benchmarking

For Scientists and Engineers

  • Provides theoretical and practical foundations as well as an in-depth look at modern benchmarks and benchmark development

  • May serve both as a textbook and a handbook on benchmarking of systems and components used as building blocks of modern information and communication technologies applications

  • Includes concrete applications and case studies based on input from consortia such as the Standard Performance Evaluation Corporation (SPEC) and the Transaction Processing Performance Council (TPC)

  • Complemented by a web site with supplementary teaching materials and information about new developments

Textbook

Table of contents

  1. Applications

    1. Samuel Kounev, Klaus-Dieter Lange, Jóakim von Kistowski
      Pages 389-421
  2. Back Matter
    Pages 397-402

About this book

Introduction

This book serves as both a textbook and handbook on the benchmarking of systems and components used as building blocks of modern information and communication technology applications. It provides theoretical and practical foundations as well as an in-depth exploration of modern benchmarks and benchmark development.

The book is divided into two parts: foundations and applications. The first part introduces the foundations of benchmarking as a discipline, covering the three fundamental elements of each benchmarking approach: metrics, workloads, and measurement methodology. The second part focuses on different application areas, presenting contributions in specific fields of benchmark development. These contributions address the unique challenges that arise in the conception and development of benchmarks for specific systems or subsystems, and demonstrate how the foundations and concepts in the first part of the book are being used in existing benchmarks. Further, the book presents a number of concrete applications and case studies based on input from leading benchmark developers from consortia such as the Standard Performance Evaluation Corporation (SPEC) and the Transaction Processing Performance Council (TPC).

Providing both practical and theoretical foundations, as well as a detailed discussion of modern benchmarks and their development, the book is intended as a handbook for professionals and researchers working in areas related to benchmarking. It offers an up-to-date point of reference for existing work as well as latest results, research challenges, and future research directions. It also can be used as a textbook for graduate and postgraduate students studying any of the many subjects related to benchmarking. While readers are assumed to be familiar with the principles and practices of computer science, as well as software and systems engineering, no specific expertise in any subfield of these disciplines is required.

“This book should be required reading for anyone interested in making good benchmarks.” – from the Foreword by David Patterson, 2017 ACM A.M. Turing Award Laureate.


Keywords

Benchmarking Metrics Workloads Measurements Experimental Design SPEC VIRT SPEC SFS VMmark SPEC CPU TPC

Authors and affiliations

  1. 1.University of WürzburgWürzburgGermany
  2. 2.Hewlett Packard EnterpriseHoustonUSA
  3. 3.DATEV eGNürnbergGermany

About the authors

Samuel Kounev is a Professor of Computer Science and Chair of Software Engineering at the University of Würzburg (Germany). He has been actively involved in the Standard Performance Evaluation Corporation (SPEC), the largest standardization consortium in the area of computer systems benchmarking, since 2002. He serves as the elected chair of the SPEC Research Group, which he initiated in 2010 with the goal of providing a platform for collaborative research efforts between academia and industry in the area of quantitative system evaluation. Samuel is also co-founder and Steering Committee Co-chair of several conferences in the field, including the ACM/SPEC International Conference on Performance Engineering (ICPE) and the IEEE International Conference on Autonomic Computing and Self-Organizing Systems (ACSOS). He has published extensively in the area of systems benchmarking, modeling, and evaluation of performance, energy efficiency, reliability, and security.

Klaus-Dieter Lange is a technologist at Hewlett Packard Enterprise (HPE), where he started his professional career in 1997, with a focus on performance and workload characterization, industry-standard benchmark development, server efficiency, and the design of secure enterprise solutions. He has been active in several of the SPEC Steering and Sub-committees since 2005, serves on the SPEC Board of Directors, and has been on the ICPE Steering Committee since its inception. Klaus is the founding chair of the SPECpower Committee, which developed and maintains under his technical leadership among others the SPECpower_ssj2008 benchmark, the SPEC PTDaemon, and the Server Efficiency Rating Tool (SERT) suite.

Jóakim von Kistowski is a software architect at DATEV eG, where he focuses on software performance and load testing, driving benchmarking method adoption, software architecture, software testing and evaluation. Jóakim has a strong SPEC background, actively contributing to the SPECpower Committee and serving as elected chair of the SPEC RG Power Working Group.

Bibliographic information

  • Book Title Systems Benchmarking
  • Book Subtitle For Scientists and Engineers
  • Authors Samuel Kounev
    Klaus-Dieter Lange
    Jóakim von Kistowski
  • DOI https://doi.org/10.1007/978-3-030-41705-5
  • Copyright Information Springer Nature Switzerland AG 2020
  • Publisher Name Springer, Cham
  • eBook Packages Computer Science Computer Science (R0)
  • Hardcover ISBN 978-3-030-41704-8
  • Softcover ISBN 978-3-030-41707-9
  • eBook ISBN 978-3-030-41705-5
  • Edition Number 1
  • Number of Pages XXVII, 427
  • Number of Illustrations 46 b/w illustrations, 49 illustrations in colour
  • Topics System Performance and Evaluation
    Software Engineering
    Software Management

Reviews

“This book should be required reading for anyone interested in making good benchmarks.” – from the Foreword by David Patterson, 2017 ACM A.M. Turing Award Laureate.

“This is a fine book by experts. It offers many good lessons and is well worth the time to study.” – from the Foreword by John R. Mashey, SPEC Co-Founder and Former Silicon Graphics VP/Chief Scientist.