Towards an Extensible Middleware for Database Benchmarking

  • David Bermbach
  • Jörn Kuhlenkamp
  • Akon DeyEmail author
  • Sherif Sakr
  • Raghunath Nambiar
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8904)


Today’s database benchmarks are designed to evaluate a particular type of database. Furthermore, popular benchmarks, like those from TPC, come without a ready-to-use implementation requiring database benchmark users to implement the benchmarking tool from scratch. The result of this is that there is no single framework that can be used to compare arbitrary database systems. The primary reason for this, among others, being the complexity of designing and implementing distributed benchmarking tools.

In this paper, we describe our vision of a middleware for database benchmarking which eliminates the complexity and difficulty of designing and running arbitrary benchmarks: workload specification and interface mappers for the system under test should be nothing but configuration properties of the middleware. We also sketch out an architecture for this benchmarking middleware and describe the main components and their requirements.


Database System Database Benchmark Workload Model Operation Trace Synthetic Workload 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Alexandrov, A., Brücke, C., Markl, V.: Issues in big data testing and benchmarking. In: Proceedings of the Sixth International Workshop on Testing Database Systems, DBTest 2013, pp. 1:1–1:5. ACM, New York (2013)Google Scholar
  2. 2.
    Alexandrov, A., Schiefer, B., Poelman, J., Ewen, S., Bodner, T.O., Markl, V.: Myriad: parallel data generation on shared-nothing architectures. In: Proceedings of the 1st Workshop on Architectures and Systems for Big Data, ASBD 2011, pp. 30–33. ACM, New York (2011)Google Scholar
  3. 3.
    Alexandrov, A., Tzoumas, K., Markl, V.: Myriad: scalable and expressive data generation. Proc. VLDB Endowment 5(12), 1890–1893 (2012)CrossRefGoogle Scholar
  4. 4.
    Anderson, E., Li, X., Shah, M.A., Tucek, J., Wylie, J.J.: What consistency does your key-value store actually provide? In: Proceedings of the 6th Workshop on Hot Topics in System Dependability (HOTDEP), HotDep 2010, pp. 1–16. USENIX Association, Berkeley (2010)Google Scholar
  5. 5.
    Armstrong, T.G., Ponnekanti, V., Borthakur, D., Callaghan, M.: LinkBench: a database benchmark based on the facebook social graph. In: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, SIGMOD 2013, pp. 1185–1196. ACM, New York (2013)Google Scholar
  6. 6.
    Baru, C., Bhandarkar, M., Nambiar, R., Poess, M., Rabl, T.: Setting the direction for big data benchmark standards. In: Nambiar, R., Poess, M. (eds.) TPCTC 2012. LNCS, vol. 7755, pp. 197–208. Springer, Heidelberg (2013) CrossRefGoogle Scholar
  7. 7.
    Beitch, A., Liu, B., Yung, T., Griffith, R., Fox, A., Patterson, D.A.: Rain: a workload generation toolkit for cloud computing applications. Technical report, University of California at Berkeley (2010)Google Scholar
  8. 8.
    Bermbach, D.: Benchmarking eventually consistent distributed storage systems. Ph.D. thesis, Karlsruhe Institute of Technology, Germany, February 2014, to be publishedGoogle Scholar
  9. 9.
    Bermbach, D., Kuhlenkamp, J.: Consistency in distributed storage systems. In: Gramoli, V., Guerraoui, R. (eds.) NETYS 2013. LNCS, vol. 7853, pp. 175–189. Springer, Heidelberg (2013) CrossRefGoogle Scholar
  10. 10.
    Bermbach, D., Tai, S.: Eventual consistency: how soon is eventual? An evaluation of amazon S3’s consistency behavior. In: Proceedings of the 6th Workshop on Middleware for Service Oriented Computing (MW4SOC), MW4SOC 2011, pp. 1:1–1:6. ACM, New York (2011)Google Scholar
  11. 11.
    Bermbach, D., Tai, S.: Benchmarking eventual consistency: lessons learned from long-term experimental studies. In: Proceedings of the 2nd International Conference on Cloud Engineering (IC2E). IEEE (2014)Google Scholar
  12. 12.
    Bermbach, D., Zhao, L., Sakr, S.: Towards comprehensive measurement of consistency guarantees for cloud-hosted data storage services. In: Nambiar, R., Poess, M. (eds.) TPCTC 2013. LNCS, vol. 8391, pp. 32–47. Springer, Heidelberg (2014) CrossRefGoogle Scholar
  13. 13.
    Cattell, R.: Scalable SQL and NoSQL data stores. SIGMOD Record 39(4), 12–27 (2010)CrossRefGoogle Scholar
  14. 14.
    Chang, F., Dean, J., Ghemawat, S., Hsieh, W.C., Wallach, D.A., Burrows, M., Chandra, T., Fikes, A., Gruber, R.E.: Bigtable: a distributed storage system for structured data. In: Proceedings of the 7th Symposium on Operating Systems Design and Implementation (OSDI), OSDI 2006, pp. 205–218. USENIX Association, Berkeley (2006)Google Scholar
  15. 15.
    Cooper, B.F., Silberstein, A., Tam, E., Ramakrishnan, R., Sears, R.: Benchmarking cloud serving systems with YCSB. In: Proceedings of the 1st Symposium on Cloud Computing (SOCC), SOCC 2010, pp. 143–154. ACM, New York (2010)Google Scholar
  16. 16.
    T. P. P. Council. TPC benchmark DS: standard specification version 1.1.0. Technical report, Transaction Processing Performance Council (2012)Google Scholar
  17. 17.
    T. P. P. Council. TPC benchmark e: standard specification version 1.13.0. Technical report, Transaction Processing Performance Council (2014)Google Scholar
  18. 18.
    T. T. P. Council. TPC benchmark c: standard specification revision 5.11. Technical report, The Transaction Processing Council (2010)Google Scholar
  19. 19.
    DeCandia, G., Hastorun, D., Jampani, M., Kakulapati, G., Lakshman, A., Pilchin, A., Sivasubramanian, S., Vosshall, P., Vogels, W.: Dynamo: Amazon’s highly available key-value store. In: Proceedings of 21st Symposium on Operating Systems Principles (SOSP), SOSP 2007, pp. 205–220. ACM, New York (2007)Google Scholar
  20. 20.
    Dey, A., Fekete, A., Nambiar, R., Röhm, U.: YCSB+T: benchmarking web-scale transactional databases. In: 2014 IEEE 30th International Conference on Data Engineering Workshops (ICDEW), pp. 223–230, March 2014Google Scholar
  21. 21.
    Difallah, D., Pavlo, A.: OLTP-bench: an extensible testbed for benchmarking relational databases. Proc. VLDB Endowment 7(4), 277–288 (2013)Google Scholar
  22. 22.
    Dory, T., Mej, B., Roy, P.V.: Measuring elasticity for cloud databases. In: Proceedings of the Second International Conference on Cloud Computing, GRIDs, and Virtualization (CLOUD COMPUTING 2011), pp. 154–160 (2011)Google Scholar
  23. 23.
    Florescu, D., Kossmann, D.: Rethinking cost and performance of database systems. SIGMOD Record 38(1), 43–48 (2009)CrossRefGoogle Scholar
  24. 24.
    Folkerts, E., Alexandrov, A., Sachs, K., Iosup, A., Markl, V., Tosun, C.: Benchmarking in the cloud: what it should, can, and cannot be. In: Nambiar, R., Poess, M. (eds.) TPCTC 2012. LNCS, vol. 7755, pp. 173–188. Springer, Heidelberg (2013) CrossRefGoogle Scholar
  25. 25.
    Ghazal, A., Rabl, T., Hu, M., Raab, F., Poess, M., Crolotte, A., Jacobsen, H.-A.: BigBench: towards an industry standard benchmark for big data analytics. In: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, SIGMOD 2013, pp. 1197–1208. ACM, New York (2013)Google Scholar
  26. 26.
    Golab, W., Li, X., Shah, M.A.: Analyzing consistency properties for fun and profit. In: Proceedings of the 30th Symposium on Principles of Distributed Computing (PODC), PODC 2011, pp. 197–206. ACM, New York (2011)Google Scholar
  27. 27.
    Gray, J., Sundaresan, P., Englert, S., Baclawski, K., Weinberger, P.J.: Quickly generating billion-record synthetic databases. In: ACM SIGMOD Record, vol. 23, pp. 243–252. ACM (1994)Google Scholar
  28. 28.
    Hunt, P., Konar, M., Junqueira, F., Reed, B.: ZooKeeper: wait-free coordination for Internet-scale systems. In: USENIX ATC (2010)Google Scholar
  29. 29.
    Huppler, K.: The art of building a good benchmark. In: Nambiar, R., Poess, M. (eds.) TPCTC 2009. LNCS, vol. 5895, pp. 18–30. Springer, Heidelberg (2009) Google Scholar
  30. 30.
    Kuhlenkamp, J., Klems, M., Röss, O.: Benchmarking scalability and elasticity of distributed database systems. PVLDB 7(12), 1219–1230 (2014)Google Scholar
  31. 31.
    Lakshman, A., Malik, P.: Cassandra: a decentralized structured storage system. SIGOPS Operating Syst. Rev. 44(2), 35–40 (2010)CrossRefGoogle Scholar
  32. 32.
    Li, C., Berry, R.: CEPBen: a benchmark for complex event processing systems. In: Nambiar, R., Poess, M. (eds.) TPCTC 2013. LNCS, vol. 8391, pp. 125–142. Springer, Heidelberg (2014) CrossRefGoogle Scholar
  33. 33.
    Massie, M.L., Chun, B.N., Culler, D.E.: The ganglia distributed monitoring system: design, implementation, and experience. Parallel Comput. 30(7), 817–840 (2004)CrossRefGoogle Scholar
  34. 34.
    Müller, S., Bermbach, D., Tai, S., Pallas, F.:Benchmarking the performance impact of transport layer security in cloud database systems. In: Proceedings of the 2nd International Conference on Cloud Engineering (IC2E). IEEE (2014)Google Scholar
  35. 35.
    Nambiar, R., Poess, M.: Keeping the TPC relevant!. Proc. VLDB Endowment 6(11), 1186–1187 (2013)CrossRefGoogle Scholar
  36. 36.
    Patil, S., Polte, M., Ren, K., Tantisiriroj, W., Xiao, L., López, J., Gibson, G., Fuchs, A., Rinaldi, B.: YCSB++: benchmarking and performance debugging advanced features in scalable table stores. In: Proceedings of the 2nd Symposium on Cloud Computing (SOCC), SOCC 2011, pp. 9:1–9:14. ACM, New York (2011)Google Scholar
  37. 37.
    Poess, M.: TPC’s benchmark development model: making the first industry standard benchmark on big data a success. In: Rabl, T., Poess, M., Baru, C., Jacobsen, H.-A. (eds.) WBDB 2012. LNCS, vol. 8163, pp. 1–10. Springer, Heidelberg (2014) CrossRefGoogle Scholar
  38. 38.
    Rabl, T., Frank, M., Sergieh, H.M., Kosch, H.: A data generator for cloud-scale benchmarking. In: Nambiar, R., Poess, M. (eds.) TPCTC 2010. LNCS, vol. 6417, pp. 41–56. Springer, Heidelberg (2011) CrossRefGoogle Scholar
  39. 39.
    Rabl, T., Gómez-Villamor, S., Sadoghi, M., Muntés-Mulero, V., Jacobsen, H.-A., Mankovskii, S.: Solving big data challenges for enterprise application performance management. Proc. VLDB Endowment 5(12), 1724–1735 (2012)CrossRefGoogle Scholar
  40. 40.
    Rabl, T., Jacobsen, H.-A.: Big data generation. In: Rabl, T., Poess, M., Baru, C., Jacobsen, H.-A. (eds.) WBDB 2012. LNCS, vol. 8163, pp. 20–27. Springer, Heidelberg (2014) CrossRefGoogle Scholar
  41. 41.
    Sakr, S.: Cloud-hosted databases: technologies, challenges and opportunities. Cluster Comput. 17(2), 487–502 (2014)CrossRefGoogle Scholar
  42. 42.
    Sakr, S., Casati, F.: Liquid benchmarks: towards an online platform for collaborative assessment of computer science research results. In: Nambiar, R., Poess, M. (eds.) TPCTC 2010. LNCS, vol. 6417, pp. 10–24. Springer, Heidelberg (2011) CrossRefGoogle Scholar
  43. 43.
    Sakr, S., Liu, A.: Is your cloud-hosted database truly elastic? In: SERVICES, pp. 444–447 (2013)Google Scholar
  44. 44.
    Sakr, S., Liu, A., Batista, D.M., Alomari, M.: A survey of large scale data management approaches in cloud environments. IEEE Commun. Surv. Tutorials 13(3), 311–336 (2011)CrossRefGoogle Scholar
  45. 45.
    Schroeder, B., Wierman, A., Harchol-Balter, M.: Open versus closed: a cautionary tale. In: Proceedings of the 3rd Conference on Networked Systems Design & Implementation, NSDI 2006, vol. 3, pp. 18–18. USENIX Association, Berkeley (2006)Google Scholar
  46. 46.
    Smith, W.D.: Characterizing cloud performance with TPC benchmarks. In: Nambiar, R., Poess, M. (eds.) TPCTC 2012. LNCS, vol. 7755, pp. 189–196. Springer, Heidelberg (2013) CrossRefGoogle Scholar
  47. 47.
    Vieira, M., Madeira, H.: A dependability benchmark for OLTP application environments. In: Proceedings of the 29th International Conference on Very Large Data Bases, VLDB 2003, vol. 29, pp. 742–753. VLDB Endowment (2003)Google Scholar
  48. 48.
    Wada, H., Fekete, A., Zhao, L., Lee, K., Liu, A.: Data consistency properties and the trade-offs in commercial cloud storages: the consumers’ perspective. In: Proceedings of the 5th Conference on Innovative Data Systems Research (CIDR), pp. 134–143, January 2011Google Scholar
  49. 49.
    Wyatt, L., Caufield, B., Pol, D.: Principles for an ETL benchmark. In: Nambiar, R., Poess, M. (eds.) TPCTC 2009. LNCS, vol. 5895, pp. 183–198. Springer, Heidelberg (2009) Google Scholar
  50. 50.
    Zellag, K., Kemme, B.: How consistent is your cloud application? In: Proceedings of the 3rd Symposium on Cloud Computing (SOCC), SOCC 2012, pp. 6:1–6:14. ACM, New York (2012)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • David Bermbach
    • 1
  • Jörn Kuhlenkamp
    • 1
  • Akon Dey
    • 2
    Email author
  • Sherif Sakr
    • 3
  • Raghunath Nambiar
    • 4
  1. 1.Information Systems Engineering GroupTU BerlinBerlinGermany
  2. 2.School of Information TechnologiesUniversity of SydneySydneyAustralia
  3. 3.King Saud Bin Abdulaziz University for Health SciencesRiyadhSaudi Arabia
  4. 4.Cisco Systems, Inc.San JoseUSA

Personalised recommendations