Abstract
The LHCb experiment [1] will be upgraded in 2018–2019 to change its operation to a triggerless full-software readout scheme from Run3. This results in increasing the load of the event building and filtering farm by a factor of 40. The farm will need to be able to handle all the 40 MHz rate of the particle collisions. The network of the data acquisition system is facing with a target speed of 40 Tb/s, aggregated by 500 nodes. It requires the links to be capable of delivering the data with at least 100 Gbps speeds per direction.
Three solutions are being evaluated: Intel® Omni-Path Architecture, 100G Ethernet and EDR InfiniBand. Intel® OPA and EDR IB runs by Remote Direct Memory Access. Ethernet uses TCP/IP or UDP/IP by default, which involves significant CPU load. However, there are solutions to implement RDMA-enabled data transfer via Ethernet as well. These technologies are called RoCE (RDMA over Converged Ethernet) and iWARP. We present first measurements with such technologies on 100 Gbps equipment in respect of the data acquisition use-case.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Cámpora Pérez, D.H., Schwemmer, R., Neufeld, N.: Protocol-independent event building evaluator for the LHCb DAQ system. IEEE Trans. Nucl. Sci. 62(3), 1110–1114 (2015)
The LHCb Collaboration et al.: LHCb Trigger and Online Upgrade Technical Design Report, CERN-LHCC-2014-016; LHCB-TDR-016 (2014)
Otto, A., Cámpora Pérez, D.H., Neufeld, N., Schwemmer, R., Pisani, F.: A first look at 100 Gbps LAN technologies, with an emphasis on future DAQ applications. In: 21st International Conference on Computing in High Energy and Nuclear Physics (2015)
Mellanox Homepage: Network Considerations for Global Pause, PFC and QoS with Mellanox Switches and Adapters. https://community.mellanox.com/docs/DOC-2022. Accessed 16 June 2017
Wikipedia Article Homepage: RDMA over Converged Ethernet. https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet. Accessed 16 June 2017
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Vőneki, B., Valat, S., Neufeld, N. (2018). RDMA Optimizations on Top of 100 Gbps Ethernet for the Upgraded Data Acquisition System of LHCb. In: Liu, ZA. (eds) Proceedings of International Conference on Technology and Instrumentation in Particle Physics 2017. TIPP 2017. Springer Proceedings in Physics, vol 212. Springer, Singapore. https://doi.org/10.1007/978-981-13-1313-4_72
Download citation
DOI: https://doi.org/10.1007/978-981-13-1313-4_72
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-13-1312-7
Online ISBN: 978-981-13-1313-4
eBook Packages: Physics and AstronomyPhysics and Astronomy (R0)