, Volume 18, Issue 3, pp 203–209 | Cite as

Scalable Data Management on Modern Networks

  • Carsten BinnigEmail author
Kurz erklärt


As data processing evolves towards large scale, distributed platforms, the network will necessarily play a substantial role in achieving efficiency and performance. Modern high-speed networks such as InfiniBand, RoCE, or Omni-Path provide advanced features such as Remote-Direct-Memory-Access (RDMA) that have shown to improve the performance and scalability of distributed data processing systems. Furthermore, switches and network cards are becoming more flexible while programmability at all levels (aka, software-defined networks) opens up many possibilities to tailor the network to data processing applications and to push processing down to the network elements. In this paper, we discuss opportunities and present our recent research results to redesign scalable data management systems for the capabilities of modern networks.



This work was funded by the German Research Foundation through a grant of the DFG Priority Program 2037 “Scalable Data Management for Future Hardware” and the Collaborative Research Center bin “Multi-Mechanisms Adaptation for the Future Internet”.

The work presented in this paper was mainly done by my fantastic PhD students and they truly deserve all the credit. Most notably, I would like to thank Andrew Crotty, Alex Galakatos, Abdallah Salama, Erfan Zamanian, and Tobias Ziegler. Furthermore, I was very lucky to have many fantastic collaborators, most notably Tim Kraska but also Ugur Cetintemel, Patrick Eugster, Rodrigo Fonseca, and Stan Zdonik.


  1. 1.
    Babu S et al (2013) Massively parallel databases and mapreduce systems. Found Trend Datab 5(1):1–104Google Scholar
  2. 2.
    Barthels C et al (2015) Rack-scale in-memory join processing using RDMA. In: ACM SIGMOD, pp 1463–1475Google Scholar
  3. 3.
    Barthels C et al (2017) Distributed join algorithms on thousands of cores. Proceedings VLDB Endowment 10(5):517–528CrossRefGoogle Scholar
  4. 4.
    Binnig C et al (2014) Distributed snapshot isolation: global transactions pay globally, local transactions pay locally. VLDB J 23(6):987–1011CrossRefGoogle Scholar
  5. 5.
    Binnig C et al (2016) The end of slow networks: It’s time for a redesign. Proceedings VLDB Endowment 9(7):528–539CrossRefGoogle Scholar
  6. 6.
    Blöcher M et al (2018) Boosting scalable data analytics with modern programmable networks. In: ACM DaMoN@SIGMOD ACM, pp 1:1–1:3Google Scholar
  7. 7.
    Brantner M et al (2008) Building a database on S3. In: Proc. of ACM SIGMOD, pp 251–264Google Scholar
  8. 8.
    Chen H et al (2017) Fast in-memory transaction processing using RDMA and HTM. ACM Trans Comput Syst 35(1):3:1–3:37CrossRefGoogle Scholar
  9. 9.
    Curino C et al (2010) Schism: a workload-driven approach to database replication and partitioning. Proceedings VLDB Endowment 3(1-2):48–57CrossRefGoogle Scholar
  10. 10.
    DDR3 SDRAM Standard. Accessed 19 Oct 2016
  11. 11.
    Devulapalli A et al (2005) Distributed queue-based locking using advanced network features. In: ICPP, pp 408–415Google Scholar
  12. 12.
    Dragojević A et al (2014) FaRM: Fast remote memory. In: Proc. of NSDI, pp 401–414Google Scholar
  13. 13.
    Dragojević A et al (2015) No compromises: distributed transactions with consistency, availability, and performance. In: Proc. of OSDI, pp 54–70Google Scholar
  14. 14.
    Dragojevic A et al (2017) RDMA reads: to use or not to use? IEEE Data Eng Bull 40(1):3–14Google Scholar
  15. 15.
    Elnikety S et al (2005) Database replication using generalized snapshot isolation. In: IEEE SRDS 2005, pp 73–84Google Scholar
  16. 16.
    Feldman M (2010) RoCE: an Ethernet-Infiniband love story. HPC Wire.
  17. 17.
    Firestone D et al (2018) Azure accelerated networking: SmartNICs in the public cloud. NSDI, 51–66. Accessed: 13 Apr 2018
  18. 18.
    Gropp W et al (2014) Using advanced MPI: modern features of the message-passing interface. MIT Press, CambridgeGoogle Scholar
  19. 19.
    Infiband Trade Association (2010) Infiniband architecture specification release 1.2.1. Accessed 19 Oct 2016Google Scholar
  20. 20.
    Infiband Trade Association (2013) InfiniBand Roadmap. Accessed 19 Oct 2016Google Scholar
  21. 21.
    Kalia A et al (2014) Using rdma efficiently for key-value services. In: Proc. of ACM SIGCOMM, pp 295–306Google Scholar
  22. 22.
    Kalia A et al (2016) FaSST: fast, scalable and simple distributed transactions with two-sided (RDMA) datagram RPCs. In: Proc. of OSDI, pp 185–201Google Scholar
  23. 23.
    Kraska T et al (2009) Consistency rationing in the cloud: pay only when it matters. Proceedings VLDB Endowment 2(1):253–264CrossRefGoogle Scholar
  24. 24.
    Krishnaswamy V et al (1997) Relative serializability: an approach for relaxing the atomicity of transactions. J Comput Syst Sci 55(2):344–354MathSciNetCrossRefGoogle Scholar
  25. 25.
    Levandoski JJ et al (2015) High performance transactions in deuteronomy. In: CIDR 2015, Online ProceedingsGoogle Scholar
  26. 26.
    Loesing S et al (2015) On the design and scalability of distributed shared-data databases. In: ACM SIGMOD, pp 663–676Google Scholar
  27. 27.
    Mitchell C et al (2013) Using one-sided RDMA reads to build a fast, CPU-efficient key-value store. In: Proc. of USENIX ATC, pp 103–114Google Scholar
  28. 28.
    Narravula S et al (2007) High performance distributed lock management services using network-based remote atomic operations. In: IEEE CCGrid, pp 583–590Google Scholar
  29. 29.
    Ousterhout J et al (2011) The case for RAMCloud. Commun ACM 54(7):121–130CrossRefGoogle Scholar
  30. 30.
    Pavlo A et al (2011) On predictive modeling for optimizing transaction execution in parallel oltp systems. Proceedings VLDB Endowment 5(2):85–96CrossRefGoogle Scholar
  31. 31.
    Pavlo A et al (2012) Skew-aware automatic database partitioning in shared-nothing, parallel oltp systems. In: Proc. of ACM SIGMOD, pp 61–72Google Scholar
  32. 32.
    Polychroniou O et al (2014) Track join: distributed joins with minimal network traffic. In: ACM SIGMODGoogle Scholar
  33. 33.
    Quamar A et al (2013) SWORD: scalable workload-aware data placement for transactional workloads. In: Proc. of EBDT, pp 430–441Google Scholar
  34. 34.
    Ramesh S et al (2008) Optimizing distributed joins with bloom filters. In: ICDCITGoogle Scholar
  35. 35.
    Rödiger W et al (2014) Locality-sensitive operators for parallel main-memory database clusters. In: ICDEGoogle Scholar
  36. 36.
    Salama A et al (2017) Rethinking distributed query execution on high-speed networks. IEEE Data Eng Bull 40(1):27–37Google Scholar
  37. 37.
    Sapio A et al (2017) DAIET: a system for data aggregation inside the network. In: SoCC ACM, p 626CrossRefGoogle Scholar
  38. 38.
    Subramoni H et al (2009) RDMA over ethernet – a preliminary study. In: CLUSTERGoogle Scholar
  39. 39.
    Yoon DY et al (2018) Distributed lock management with RDMA: decentralization without starvation. In: ACM SIGMOD, pp 1571–1586Google Scholar
  40. 40.
    Zamanian E et al (2017) The end of a myth: distributed transaction can scale. Proceedings VLDB Endowment 10(6):685–696CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Deutschland, ein Teil von Springer Nature 2018

Authors and Affiliations

  1. 1.TU DarmstadtDarmstadtGermany

Personalised recommendations