MemTest: A Novel Benchmark for In-memory Database

  • Qiangqiang Kang
  • Cheqing JinEmail author
  • Zhao Zhang
  • Aoying Zhou
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8807)


With the rapid development of hardware, a stand-alone computer can employ a memory which has large amounts of volumes. Several industries and research institutions have devoted more resources to develop several in-memory databases, which preload the data into memory for further processing. With the boom of in-memory databases, there emerges requirements to evaluate and compare the performance of these systems impartially and objectively. In this paper, we proposed MemTest, a novel benchmark considering the main characteristics of an in-memory database. This benchmark constructs particular metrics, which cover CPU usage, cache miss, compression ratio, minimal memory space and response time of an in-memory database and are also the core of our benchmark. We design a data model based on inter-bank transaction applications, around which a data generator is devised to support the data distributions of uniform and skew. The MemTest workload includes a set of queries and transactions against the metrics and data model. In the end, we illustrate the efficacy of MemTest through implementations on three different in-memory databases.


Memory In-memory database Benchmark Finance 



Our research is supported by the 973 program of China (No. 2012CB316203), NSFC (No. 61370101), Shanghai Knowledge Service Platform Project (No. ZF1213), Innovation Program of Shanghai Municipal Education Commission (14ZZ045) and the Natural Science Foundation of Shanghai (14ZR1 412600).


  1. 1.
    Fedorova, A., Seltzer, M.I., Small, C., Nussbaum, D.: Performance of multithreaded chip multiprocessors and implications for operating system design. In: USENIX Annual Technical Conference, General Track, pp. 395–398 (2005)Google Scholar
  2. 2.
    Färber, F., Cha, S.K., Primsch, J., Bornhövd, C., Sigg, S., Lehner, W.: Sap hana database: data management for modern business applications. In: SIGMOD Record, pp. 45–51 (2011)Google Scholar
  3. 3.
    Lahiri, T., Neimat, M.A., Folkman, S.: Oracle timesten: an in-memory database for enterprise applications. IEEE Data Eng. Bull. 36, 6–13 (2013)Google Scholar
  4. 4.
    Diaconu, C., Freedman, C., Ismert, E., Larson, P.A., Mittal, P., Stonecipher, R., Verma, N., Zwilling, M.: Hekaton: sql server’s memory-optimized oltp engine. In: SIGMOD Conference, pp. 1243–1254 (2013)Google Scholar
  5. 5.
    HyperSQL: Hypersql.
  6. 6.
  7. 7.
    MemSql: Memsql.
  8. 8.
    Monetdb: Monetdb.
  9. 9.
    Carey, M.J., DeWitt, D.J., Naughton, J.F.: The oo7 benchmark. In: Buneman, P., Jajodia, S. (eds.): SIGMOD Conference, pp. 12–21. ACM Press (1993)Google Scholar
  10. 10.
    Carey, M.J., DeWitt, D.J., Naughton, J.F., Asgarian, M., Brown, P., Gehrke, J., Shah, D.: The bucky object-relational benchmark (experience paper). In: SIGMOD Conference, pp. 135–146 (1997)Google Scholar
  11. 11.
    Schmidt, A., Waas, F., Kersten, M.L., Carey, M.J., Manolescu, I., Busse, R.: Xmark: a benchmark for xml data management. In: VLDB, pp. 974–985 (2002)Google Scholar
  12. 12.
    Carey, M.J., Ling, L., Nicola, M., Shao, L.: EXRT: towards a simple benchmark for XML readiness testing. In: Nambiar, R., Poess, M. (eds.) TPCTC 2010. LNCS, vol. 6417, pp. 93–109. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  13. 13.
    Cooper, B.F., Silberstein, A., Tam, E., Ramakrishnan, R., Sears, R.: Benchmarking cloud serving systems with ycsb. In: SoCC, pp. 143–154 (2010)Google Scholar
  14. 14.
    Ghazal, A., Rabl, T., Hu, M., Raab, F., Poess, M., Crolotte, A., Jacobsen, H.A.: Bigbench: towards an industry standard benchmark for big data analytics. In: SIGMOD Conference, 1197–1208 (2013)Google Scholar
  15. 15.
    Gupta, M.K., Verma, V., Verma, M.S.: In-memory database systems - a paradigm shift, pp. 333–336 (2014)Google Scholar
  16. 16.
    Gray, J.: Benchmark Handbook: For Database and Transaction Processing Systems. Morgan Kaufmann Publishers Inc., San Francisco (1992)Google Scholar
  17. 17.
    Cole, R., Funke, F., Giakoumakis, L., Guy, W., Kemper, A., Krompass, S., Kuno, H.A., et al.: The mixed workload ch-benchmark. In: DBTest, pp. 8:1–8:6 (2011)Google Scholar
  18. 18.
    Rabl, T., Poess, M., Jacobsen, H.A., O’Neil, P.E., O’Neil, E.J.: Variations of the star schema benchmark to test the effects of data skew on query performance. In: ICPE, pp. 361–372 (2013)Google Scholar
  19. 19.
    Tözün, P., Pandis, I., Kaynak, C., Jevdjic, D., Ailamaki, A.: From a to e: analyzing tpc’s oltp benchmarks: the obsolete, the ubiquitous, the unexplored. In: EDBT, pp. 17–28 (2013)Google Scholar
  20. 20.
    Liu, D., Luan, H., Wang, S., Qin, B.: Main memory database TPC-H workload characterization on modern process. J. Softw. 19, 2573–2584 (2008)CrossRefGoogle Scholar
  21. 21.
    Pirk, H., Funke, F., Grund, M., Neumann, T., Leser, U., Manegold, S., Kemper, A., Kersten, M.L.: Cpu and cache efficient management of memory-resident databases. In: ICDE, pp. 14–25 (2013)Google Scholar
  22. 22.
    Müller, I., Ratsch, C., Frber, F.: Adaptive string dictionary compression in in-memory column-store database systems. In: EDBT, pp. 283–294 (2014)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Qiangqiang Kang
    • 1
  • Cheqing Jin
    • 1
    Email author
  • Zhao Zhang
    • 1
  • Aoying Zhou
    • 1
  1. 1.Institute for Data Science and EngineeringSoftware Engineering Institute, East China Normal UniversityShanghaiChina

Personalised recommendations