Metrics for Measuring the Performance of the Mixed Workload CH-benCHmark

  • Florian Funke
  • Alfons Kemper
  • Stefan Krompass
  • Harumi Kuno
  • Raghunath Nambiar
  • Thomas Neumann
  • Anisoara Nica
  • Meikel Poess
  • Michael Seibold
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7144)

Abstract

Advances in hardware architecture have begun to enable database vendors to process analytical queries directly on operational database systems without impeding the performance of mission-critical transaction processing too much. In order to evaluate such systems, we recently devised the mixed workload CH-benCHmark, which combines transactional load based on TPC-C order processing with decision support load based on TPC-H-like query suite run in parallel on the same tables in a single database system. Just as the data volume of actual enterprises tends to increase over time, an inherent characteristic of this mixed workload benchmark is that data volume increases during benchmark runs, which in turn may increase response times of analytic queries. For purely transactional loads, response times typically do not depend that much on data volume, as the queries used within business transactions are less complex and often indexes are used to answer these queries with point-wise accesses only. But for mixed workloads, the insert throughput metric of the transactional component interferes with the response-time metric of the analytic component. In order to address the problem, in this paper we analyze the characteristics of CH-benCHmark queries and propose normalized metrics which account for data volume growth.

Keywords

mixed workload real-time business intelligence 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Bog, A., Sachs, K., Zeier, A.: Benchmarking Database Design for Mixed OLTP and OLAP Workloads. In: Second Joint WOSP/SIPEW International Conference on Performance Engineering (2011)Google Scholar
  2. 2.
    Bog, A., Schaffner, J., Krüger, J.: A Composite Benchmark for Online Transaction Processing and Operational Reporting. In: IEEE Symposium on Advanced Management of Information for Globalized Enterprises (2008)Google Scholar
  3. 3.
    Cole, R., Funke, F., Giakoumakis, L., Guy, W., Kemper, A., Krompass, S., Kuno, H., Nambiar, R., Neumann, T., Poess, M., Sattler, K.-U., Seibold, M., Simon, E., Waas, F.: The mixed workload CH-benCHmark. In: ACM International Workshop on Testing Database Systems (ACM DBTEST), ACM Special Interest Group on Management of Data (ACM SIGMOD) Conference Workshops (2011)Google Scholar
  4. 4.
    Doppelhammer, J., Höppler, T., Kemper, A., Kossmann, D.: Database Performance in the Real World - TPC-D and SAP R/3. In: ACM SIGMOD International Conference on Management of Data (1997)Google Scholar
  5. 5.
    Plattner, H.: A common database approach for OLTP and OLAP using an in-memory column database. In: ACM SIGMOD International Conference on Management of Data (2009)Google Scholar
  6. 6.
    Poess, M., Nambiar, R.: Building Enterprise Class Real-Time Energy Efficient Decision Support Systems. In: Fourth International Workshop on Enabling Real-Time Business Intelligence (BIRTE), International Conference on Very Large Data Bases (VLDB) Workshops (2010)Google Scholar
  7. 7.
    Transaction Processing Performance Council. TPC-C specification (2010), www.tpc.org/tpcc/spec/TPC-C_v5-11.pdf
  8. 8.
    Transaction Processing Performance Council. TPC-H specification (2011), http://www.tpc.org/tpch/spec/tpch2.14.0.pdf
  9. 9.
    VoltDB. TPC-C-like Benchmark Comparison - Benchmark Description, http://community.voltdb.com/node/134

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Florian Funke
    • 1
  • Alfons Kemper
    • 1
  • Stefan Krompass
    • 1
  • Harumi Kuno
    • 2
  • Raghunath Nambiar
    • 3
  • Thomas Neumann
    • 1
  • Anisoara Nica
    • 4
  • Meikel Poess
    • 5
  • Michael Seibold
    • 1
  1. 1.Technische Universität MünchenGarching bei MünchenGermany
  2. 2.HP LabsPalo AltoUSA
  3. 3.Cisco Systems, Inc.San JoseUSA
  4. 4.Sybase, An SAP CompanyWaterlooCanada
  5. 5.Oracle CorporationRedwood ShoresUSA

Personalised recommendations