uNVMe-TCP: A User Space Approach to Optimizing NVMe over Fabrics TCP Transport

  • Ziye YangEmail author
  • Qun Wan
  • Gang Cao
  • Karol Latecki
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11894)


Recently, NVM Express® has released the new TCP transport specification for NVMe over fabrics (NVMe-oF). And there are two kinds of implementations, i.e., one in kernel space and the other in user space. The implementation in the kernel (e.g., Linux kernel) is feasible, but there are several drawbacks such as performance, flexibility, and stability. In this paper, we would like to introduce uNVMe-TCP, which follows the specification and provides the NVMe/TCP transport in user space with improved performance and usage experience. We choose the optimization in user space since it is very difficult to optimize the whole NVMe I/O stack in kernel space through different kernel modules, and the optimization may affect other applications in user space. The idea of uNVMe-TCP is to optimize the whole NVMe I/O stack on TCP transport, i.e., leveraging the lock-free user space NVMe I/O stack and configurable network I/O stack (both kernel and user space TCP stack can be supported). Currently uNVMe-TCP provides the solution on both target and initiator side, and it can be tested against Linux kernel solution with good interoperability. Besides, some experiments are conducted to demonstrate the performance of uNVMe-TCP. Compared with the kernel solution, uNVMe-TCP shows 15% to 30% latency improvement on average with FIO benchmark. And the per CPU core performance of uNVMe-TCP is promising, i.e., it is 2.2 times of the kernel on average with the increasing number of connections. Furthermore, uNVMe-TCP is also scalable in CPU aspect.


User space NVMe over Fabrics TCP transport solution User space NVMe I/O stack Performance improvement 


  1. 1.
  2. 2.
    IETF. Internet Small Computer Systems Interface (iSCSI).
  3. 3.
  4. 4.
    Yang, Z., et al.: SPDK: a development kit to build high performance storage applications. In: 2017 IEEE International Conference on Cloud Computing Technology and Science (CloudCom), pp. 154–161, December 2017Google Scholar
  5. 5.
  6. 6.
  7. 7.
    RDMA Consortium. iSCSI Extensions for RDMA (iSER) and Datamover Architecture for iSCSI (DA) SpecificationsGoogle Scholar
  8. 8.
    VMware Virtual NVMe support.
  9. 9.
    Yang, Z., Liu, C., Zhou, Y., Liu, X., Cao, G.: SPDK vhost-NVMe: accelerating I/Os in virtual machines on NVMe SSDs via user space vhost target. In: 2018 IEEE 8th International Symposium on Cloud and Service Computing (SC2), pp. 67–76, November 2018Google Scholar
  10. 10.
    Yang, J., Minturn, D.B., Hady, F.: When poll is better than interrupt. In: Proceedings of the 10th USENIX Conference on File and Storage Technologies (2012)Google Scholar
  11. 11.
    Data Plane Development Kit.
  12. 12.
    Vector Packet Processing.
  13. 13.
    Flexible I/O Tester.
  14. 14.
  15. 15.
    SPDK NVMe-oF (Target & Initiator) Performance Report Release 19.01.1.
  16. 16.
  17. 17.
    Shivam, P., Wyckoff, P., Panda, D.: EMP: Zero-copy OS-bypass NIC-driven gigabit ethernet message passing. In: SC 2001: Proceedings of the 2001 ACM/IEEE Conference on Supercomputing, p. 49, November 2001Google Scholar
  18. 18.
    Goldenberg, D., Kagan, M., Ravid, R., Tsirkin, M.S.: Zero copy sockets direct protocol over infiniband – preliminary implementation and performance analysis. In: Proceedings of the 13th Symposium on High Performance Interconnects, HOTI 2005, pp. 128–137. IEEE Computer Society, Washington, DC (2005)Google Scholar
  19. 19.
    Li, Y.-C., Chiang, M.-L.: LyraNET: a zero-copy TCP/IP protocol stack for embedded operating systems. In: 11th IEEE International Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA 2005), pp. 123–128, August 2005Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.IntelShanghaiChina
  2. 2.IntelGdanskPoland

Personalised recommendations