Compression-Aware In-Memory Query Processing: Vision, System Design and Beyond

  • Juliana Hildebrandt
  • Dirk HabichEmail author
  • Patrick Damme
  • Wolfgang Lehner
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10195)


In-memory database systems have to keep base data as well as intermediate results generated during query processing in main memory. In addition, the effort to access intermediate results is equivalent to the effort to access the base data. Therefore, the optimization of intermediate results is interesting and has a high impact on the performance of the query execution. For this domain, we propose the continuous use of lightweight compression methods for intermediate results and have the aim of developing a balanced query processing approach based on compressed intermediate results. To minimize the overall query execution time, it is important to find a balance between the reduced transfer times and the increased computational effort. This paper provides an overview and presents a system design for our vision. Our system design addresses the challenge of integrating a large and evolving corpus of lightweight data compression algorithms in an in-memory column store. In detail, we present our model-driven approach and describe ongoing research topics to realize our compression-aware query processing vision.


Query Processing Intermediate Result Compression Algorithm Query Optimization Parameter Definition 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Abadi, D., Boncz, P.A., Harizopoulos, S., Idreos, S., Madden, S.: The design and implementation of modern column-oriented database systems. Found. Trends Databases 5(3), 197–280 (2013)CrossRefGoogle Scholar
  2. 2.
    Abadi, D.J., Madden, S.R., Ferreira, M.C.: Integrating compression and execution in column-oriented database systems. In: SIGMOD, pp. 671–682 (2006)Google Scholar
  3. 3.
    Anh, V.N., Moffat, A.: Inverted index compression using word-aligned binary codes. Inf. Retr. 8(1), 151–166 (2005)CrossRefGoogle Scholar
  4. 4.
    Arroyuelo, D., González, S., Oyarzún, M., Sepulveda, V.: Document identifier reassignment and run-length-compressed inverted indexes for improved search performance. In: SIGIR, pp. 173–182 (2013)Google Scholar
  5. 5.
    Boncz, P.A., Kersten, M.L., Manegold, S.: Breaking the memory wall in MonetDB. Commun. ACM 51(12), 77–85 (2008)CrossRefGoogle Scholar
  6. 6.
    Chen, Z., Gehrke, J., Korn, F.: Query optimization in compressed database systems. SIGMOD Rec. 30(2), 271–282 (2001)CrossRefGoogle Scholar
  7. 7.
    Copeland, G.P., Khoshafian, S.N.: A decomposition storage model. SIGMOD Rec. 14(4), 268–279 (1985)CrossRefGoogle Scholar
  8. 8.
    Damme, P., Habich, D., Lehner, W.: A benchmark framework for data compression techniques. In: Nambiar, R., Poess, M. (eds.) TPCTC 2015. LNCS, vol. 9508, pp. 77–93. Springer, Cham (2016). doi: 10.1007/978-3-319-31409-9_6 CrossRefGoogle Scholar
  9. 9.
    Damme, P., Habich, D., Lehner, W.: Direct transformation techniques for compressed data: general approach and application scenarios. In: Morzy, T., Valduriez, P., Bellatreche, L. (eds.) ADBIS 2015. LNCS, vol. 9282, pp. 151–165. Springer, Cham (2015). doi: 10.1007/978-3-319-23135-8_11 CrossRefGoogle Scholar
  10. 10.
    Delbru, R., Campinas, S., Samp, K., Tummarello, G., Dangan, L., Delbru, R., Campinas, S., Samp, K., Tummarello, G.: Adaptive frame of reference for compressing inverted lists (2010)Google Scholar
  11. 11.
    Goldstein, J., Ramakrishnan, R., Shaft, U.: Compressing relations and indexes. In: ICDE, pp. 370–379 (1998)Google Scholar
  12. 12.
    Habich, D., Richly, S., Lehner, W.: GignoMDA - exploiting cross-layer optimization for complex database applications. In: VLDB (2006)Google Scholar
  13. 13.
    Iyer, B.R., Wilhite, D.: Data compression support in databases. In: VLDB Conference, pp. 695–704 (1994)Google Scholar
  14. 14.
    Kissinger, T., Schlegel, B., Habich, D., Lehner, W.: KISS-Tree: smart latch-free in-memory indexing on modern architectures. In: DaMoN, pp. 16–23 (2012)Google Scholar
  15. 15.
    Kissinger, T., Schlegel, B., Habich, D., Lehner, W.: QPPT: query processing on prefix trees. In: CIDR 2013 (2013)Google Scholar
  16. 16.
    Kleppe, A., Warmer, J., Bast, W.: MDA Explained. The Model Driven Architecture: Practice and Promise. Addison-Wesley, Massachusetts (2003)Google Scholar
  17. 17.
    Leis, V., Kemper, A., Neumann, T.: The adaptive radix tree: artful indexing for main-memory databases. In: ICDE, pp. 38–49 (2013)Google Scholar
  18. 18.
    Lemire, D., Boytsov, L.: Decoding billions of integers per second through vectorization. Softw. Pract. Exper. 45(1), 1–29 (2015)CrossRefGoogle Scholar
  19. 19.
    Neumann, T.: Efficiently compiling efficient query plans for modern hardware. PVLDB 4(9), 539–550 (2011)Google Scholar
  20. 20.
    Qiao, L., Raman, V., Reiss, F., Haas, P.J., Lohman, G.M.: Main-memory scan sharing for multi-core cpus. PVLDB 1, 610–621 (2008)Google Scholar
  21. 21.
    Roth, M.A., Van Horn, S.J.: Database compression. SIGMOD Rec. 22(3), 31–39 (1993)CrossRefGoogle Scholar
  22. 22.
    Schlegel, B., Gemulla, R., Lehner, W.: Fast integer compression using SIMD instructions. In: DaMoN (2010)Google Scholar
  23. 23.
    Silvestri, F., Venturini, R.: Vsencoding: efficient coding and fast decoding of integer lists via dynamic programming. In: CIKM, pp. 1219–1228 (2010)Google Scholar
  24. 24.
    Stepanov, A.A., Gangolli, A.R., Rose, D.E., Ernst, R.J., Oberoi, P.S.: SIMD-based decoding of posting lists. In: CIKM, pp. 317–326 (2011)Google Scholar
  25. 25.
    Willhalm, T., Popovici, N., Boshmaf, Y., Plattner, H., Zeier, A., Schaffner, J.: SIMD-scan: ultra fast in-memory table scan using on-chip vector processing units. PVLDB 2(1), 385–394 (2009)Google Scholar
  26. 26.
    Williams, R.: Adaptive Data Compression. Kluwer International Series in Engineering and Computer Science: Communications and Information Theory. Springer, US (1991)CrossRefzbMATHGoogle Scholar
  27. 27.
    Zukowski, M., Heman, S., Nes, N., Boncz, P.: Super-scalar RAM-CPU cache compression. In: ICDE, p. 59 (2006)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Juliana Hildebrandt
    • 1
  • Dirk Habich
    • 1
    Email author
  • Patrick Damme
    • 1
  • Wolfgang Lehner
    • 1
  1. 1.Database Systems GroupTechnische Universität DresdenDresdenGermany

Personalised recommendations