Abstract
Recently, NVM Express® has released the new TCP transport specification for NVMe over fabrics (NVMe-oF). And there are two kinds of implementations, i.e., one in kernel space and the other in user space. The implementation in the kernel (e.g., Linux kernel) is feasible, but there are several drawbacks such as performance, flexibility, and stability. In this paper, we would like to introduce uNVMe-TCP, which follows the specification and provides the NVMe/TCP transport in user space with improved performance and usage experience. We choose the optimization in user space since it is very difficult to optimize the whole NVMe I/O stack in kernel space through different kernel modules, and the optimization may affect other applications in user space. The idea of uNVMe-TCP is to optimize the whole NVMe I/O stack on TCP transport, i.e., leveraging the lock-free user space NVMe I/O stack and configurable network I/O stack (both kernel and user space TCP stack can be supported). Currently uNVMe-TCP provides the solution on both target and initiator side, and it can be tested against Linux kernel solution with good interoperability. Besides, some experiments are conducted to demonstrate the performance of uNVMe-TCP. Compared with the kernel solution, uNVMe-TCP shows 15% to 30% latency improvement on average with FIO benchmark. And the per CPU core performance of uNVMe-TCP is promising, i.e., it is 2.2 times of the kernel on average with the increasing number of connections. Furthermore, uNVMe-TCP is also scalable in CPU aspect.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
NVM Express Spec 1.3. https://nvmexpress.org/wp-content/uploads/NVM-Express-1_3d-2019.03.20-Ratified.pdf
IETF. Internet Small Computer Systems Interface (iSCSI). https://tools.ietf.org/html/rfc3720
NVM Express over Fabrics 1.0. http://nvmexpress.org/wp-content/uploads/NVMe_over_Fabrics_1_0_Gold_20160605-1.pdf
Yang, Z., et al.: SPDK: a development kit to build high performance storage applications. In: 2017 IEEE International Conference on Cloud Computing Technology and Science (CloudCom), pp. 154–161, December 2017
SPDK github. https://github.com/spdk/spdk
NVMe-oF - TP 8000 TCP Transport. https://nvmexpress.org/wp-content/uploads/NVM-Express-over-Fabrics-1.0-Ratified-TPs.zip
RDMA Consortium. iSCSI Extensions for RDMA (iSER) and Datamover Architecture for iSCSI (DA) Specifications
VMware Virtual NVMe support. https://kb.vmware.com/s/article/2147714
Yang, Z., Liu, C., Zhou, Y., Liu, X., Cao, G.: SPDK vhost-NVMe: accelerating I/Os in virtual machines on NVMe SSDs via user space vhost target. In: 2018 IEEE 8th International Symposium on Cloud and Service Computing (SC2), pp. 67–76, November 2018
Yang, J., Minturn, D.B., Hady, F.: When poll is better than interrupt. In: Proceedings of the 10th USENIX Conference on File and Storage Technologies (2012)
Data Plane Development Kit. http://dpdk.org/
Vector Packet Processing. https://fd.io/technology/
Flexible I/O Tester. https://github.com/axboe/fio
Intel DC series P4600 2TB SSD. https://www.intel.com/content/dam/www/public/us/en/documents/product-briefs/ssd-dc-p4600-brief.pdf
SPDK NVMe-oF (Target & Initiator) Performance Report Release 19.01.1. https://dqtibwqq6s6ux.cloudfront.net/download/performance-reports/SPDK_nvmeof_perf_report_19.01.1.pdf
Mellanox Messaging Accelerator. http://www.mellanox.com/related-docs/prod_acceleration_software/VMA_EN.pdf
Shivam, P., Wyckoff, P., Panda, D.: EMP: Zero-copy OS-bypass NIC-driven gigabit ethernet message passing. In: SC 2001: Proceedings of the 2001 ACM/IEEE Conference on Supercomputing, p. 49, November 2001
Goldenberg, D., Kagan, M., Ravid, R., Tsirkin, M.S.: Zero copy sockets direct protocol over infiniband – preliminary implementation and performance analysis. In: Proceedings of the 13th Symposium on High Performance Interconnects, HOTI 2005, pp. 128–137. IEEE Computer Society, Washington, DC (2005)
Li, Y.-C., Chiang, M.-L.: LyraNET: a zero-copy TCP/IP protocol stack for embedded operating systems. In: 11th IEEE International Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA 2005), pp. 123–128, August 2005
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Yang, Z., Wan, Q., Cao, G., Latecki, K. (2020). uNVMe-TCP: A User Space Approach to Optimizing NVMe over Fabrics TCP Transport. In: Hsu, CH., Kallel, S., Lan, KC., Zheng, Z. (eds) Internet of Vehicles. Technologies and Services Toward Smart Cities. IOV 2019. Lecture Notes in Computer Science(), vol 11894. Springer, Cham. https://doi.org/10.1007/978-3-030-38651-1_13
Download citation
DOI: https://doi.org/10.1007/978-3-030-38651-1_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-38650-4
Online ISBN: 978-3-030-38651-1
eBook Packages: Computer ScienceComputer Science (R0)