To Copy or Not to Copy: Making In-Memory Databases Fast on Modern NICs

  • Aniraj KesavanEmail author
  • Robert Ricci
  • Ryan Stutsman
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10195)


When databases resided primarily on disks, the problem of data layout was focused on structures that enabled efficient reads and writes from that medium, as well the as effective use of main memory as a cache. With in-memory databases, this part of the I/O problem is largely gone, but that does not mean that I/O considerations can be completely ignored. We argue for the importance of considering the “other” side of the I/O equation—network communication with clients—when designing in-memory databases.

Modern NICs include a number of optimizations intended to improve I/O performance, including kernel bypass, zero-copy and scatter-gather DMA. Applied carefully, these features can reduce the involvement of the CPU in network transfers and can save memory bandwidth. However, our experiments show that some optimizations do not always provide a benefit in a database context, and using them can be tricky, as they affect strategies for processing updates and managing data lifetimes. In this paper, we explore the application of zero-copy NIC DMA to in-memory databases, and we explore how the NIC can influence and explicitly leverage data layout and concurrency control. We apply these results to the Bw-Tree structure by proposing a client-assisted design for transmitting large range scan results. Overall, the approach boosts throughput by 1.7\(\times \) and reduces CPU overhead by 75% compared to simple zero-copy DMA.


Memory Bandwidth Direct Memory Access Data Layout Remote Direct Memory Access Secondary Index 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This material is based upon work supported by the National Science Foundation under Grant No. CNS-1338155.


  1. 1.
  2. 2.
    Mellanox ConnectX-4 product brief. Accessed 04 Aug 2016
  3. 3.
    Atikoglu, B., Xu, Y., Frachtenberg, E., Jiang, S., Paleczny, M.: Workload analysis of a large-scale key-value store. In: Proceedings of the 12th ACM SIGMETRICS/PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems, SIGMETRICS 2012. ACM, New York (2012)Google Scholar
  4. 4.
    Bronson, N.: Personal Communication (2016)Google Scholar
  5. 5.
    Diaconu, C., Freedman, C., Ismert, E., Larson, P., Mittal, P., Stonecipher, R., Verma, N., Zwilling, M.: Hekaton: SQL server’s memory-optimized OLTP engine. In: SIGMOD (2013)Google Scholar
  6. 6.
    Dragojević, A., Narayanan, D., Castro, M., Hodson, O.: FaRM: fast remote memory. In: USENIX NSDI, Seattle, WA, April 2014. USENIX Association (2014)Google Scholar
  7. 7.
    Dragojević, A., Narayanan, D., Nightingale, E.B., Renzelmann, M., Shamis, A., Badam, A., Castro, M.: No compromises: distributed transactions with consistency, availability, and performance. In: SOSP (2015)Google Scholar
  8. 8.
    Kalia, A., Kaminsky, M., Andersen, D.G.: Using RDMA efficiently for key-value services. In: Proceedings of the 2014 ACM SIGCOMM Conference, SIGCOMM 2014. ACM, New York (2014)Google Scholar
  9. 9.
    Kalia, A., Kaminsky, M., Andersen, D.G.: Design guidelines for high performance RDMA systems. In: 2016 USENIX Annual Technical Conference (USENIX ATC 2016), Denver, CO, June 2016. USENIX Association (2016)Google Scholar
  10. 10.
    Kejriwal, A., Gopalan, A., Gupta, A., Jia, Z., Yang, S., Ousterhout, J.: SLIK: scalable low-latency indexes for a key-value store. In: USENIX Annual Technical Conference, Denver, CO, June 2016Google Scholar
  11. 11.
    Levandoski, J., Lomet, D., Sengupta, S., Stutsman, R., Wang, R.: High performance transactions in Deuteronomy. In: Proceeding of CIDR (2015)Google Scholar
  12. 12.
    Levandoski, J., Lomet, D., Sengupta, S., Stutsman, R., Wang, R.: Multi-version range concurrency control in Deuteronomy. In: Proceeding VLDB Endowment (2016)Google Scholar
  13. 13.
    Levandoski, J.J., Lomet, D.B., Sengupta, S., Bw-Tree, T.: A B-tree for new hardware platforms. In: International Conference on Data Engineering (ICDE). IEEE (2013)Google Scholar
  14. 14.
    Lim, H., Han, D., Andersen, D.G., Kaminsky, M.: MICA: a holistic approach to fast in-memory key-value storage. In: 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 2014), Seattle, WA, April 2014Google Scholar
  15. 15.
    Loesing, S., Pilman, M., Etter, T., Kossmann, D.: On the design and scalability of distributed shared-data databases. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, SIGMOD 2015. ACM, New York (2015)Google Scholar
  16. 16.
    Nishtala, R., Fugal, H., Grimm, S., Kwiatkowski, M., Lee, H., Li, H.C., McElroy, R., Paleczny, M., Peek, D., Saab, P., Stafford, D., Tung, T., Venkataramani, V.: Scaling memcache at Facebook. In: Symposium on Networked Systems Design and Implementation (NSDI). USENIX, Lombard (2013)Google Scholar
  17. 17.
    Ousterhout, J., Gopalan, A., Gupta, A., Kejriwal, A., Lee, C., Montazeri, B., Ongaro, D., Park, S.J., Qin, H., Rosenblum, M., Rumble, S., Stutsman, R., Yang, S.: The RAMCloud storage system. ACM Trans. Comput. Syst. 33(3), 7:1–7:55 (2015)CrossRefGoogle Scholar
  18. 18.
    Ricci, R., Wong, G., Stoller, L., Webb, K., Duerig, J., Downie, K., Hibler, M.: Apt: a platform for repeatable research in computer science. ACM SIGOPS Operating Syst. Rev. 49(1), 100–107 (2015). CrossRefGoogle Scholar
  19. 19.
    Rödiger, W., Mühlbauer, T., Kemper, A., Neumann, T.: High-speed query processing over high-speed networks. In: Proceeding VLDB Endowment, December 2015Google Scholar
  20. 20.
    Wei, X., Shi, J., Chen, Y., Chen, R., Chen, H.: Fast in-memory transaction processing using RDMA and HTM. In: Proceedings of SOSP, SOSP 2015. ACM, New York (2015)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.University of Utah School of ComputingSalt Lake CityUSA

Personalised recommendations