Abstract
Network library APIs have historically been developed with the emphasis on data movement, placement, and communication semantics. Many communication semantics are available across a large variety of network libraries, such as send-receive, data streaming, put/get/atomic, RPC, active messages, collective communication, etc. In this work we introduce new compute and data movement APIs that overcome the constraints of the single-program, multiple-data (SPMD) programming model by allowing users to send binary executable code between processing elements. Our proof-of-concept implementation of the API is based on the UCX communication framework and leverages the RDMA network for fast compute migration. We envision the API being used to dispatch user functions from a host CPU to a SmartNIC (DPU), computational storage drive (CSD), or remote servers. In addition, the API can be used by large-scale irregular applications (such as semantic graph analysis), composed of many coordinating tasks operating on a data set so big that it has to be stored on many physical devices. In such cases, it may be more efficient to dynamically choose where code runs as the applications progresses.
L. E. Peña and W. Lu—Contributed equally.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Two-Chains source code. https://github.com/openucx/ucx-two-chains
Acun, B., et al.: Parallel programming with migratable objects: charm++ in practice. In: SC 2014: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 647–658. IEEE (2014). http://charm.cs.illinois.edu/newPapers/14-07/paper.pdf
Agache, A., et al.: Firecracker: lightweight virtualization for serverless applications. In: 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 2020), pp. 419–434. USENIX Association, Santa Clara, CA (2020). https://www.usenix.org/conference/nsdi20/presentation/agache
Bonachea, D., Hargrove, P.H.: Gasnet-ex: a high-performance, portable communication library for exascale. In: International Workshop on Languages and Compilers for Parallel Computing, pp. 138–158. Springer (2018). https://bytebucket.org/berkeleylab/upcxx/wiki/pubs/gasnet-ex-lcpc18-6da6911-tech.pdf
Dragojević, A., Narayanan, D., Castro, M., Hodson, O.: Farm: fast remote memory. In: 11th \(\{\)USENIX\(\}\) Symposium on Networked Systems Design and Implementation (\(\{\)NSDI\(\}\) 14), pp. 401–414 (2014)
Fouladi, S., et al.: From laptop to lambda: outsourcing everyday jobs to thousands of transient functional containers. In: 2019 USENIX Annual Technical Conference (USENIX ATC 19), pp. 475–488. USENIX Association, Renton, WA (2019). https://www.usenix.org/conference/atc19/presentation/fouladi
Grodowitz, M., Peña, L.E., Dunham, C., Zhong, D., Shamis, P., Poole, S.: Two-chains: high performance framework for function injection and execution. In: (To appear in) 2021 IEEE International Conference on Cluster Computing (CLUSTER). IEEE (2021). https://arxiv.org/abs/2108.02253
Infiniband trade association specification. https://www.infinibandta.org
Klinkenberg, J., Samfass, P., Bader, M., Terboven, C., Müller, M.S.: Chameleon: Reactive load balancing for hybrid MPI+openmp task-parallel applications. J. Parallel Distrib. Comput. 138, 55–64 (2020). https://doi.org/10.1016/j.jpdc.2019.12.005, http://www.sciencedirect.com/science/article/pii/S0743731519305180, https://gauss-allianz.de/files/projects/Chameleon_Reactive_Task_Migration_for_Hybrid_MPI__OpenMP_Applications_9_HPCStatusKonferenz_191128083805.pdf
Marty, M., et al.: Snap: a microkernel approach to host networking. In: Proceedings of the 27th ACM Symposium on Operating Systems Principles, pp. 399–413. SOSP 2019, Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3341301.3359657, http://pages.cs.wisc.edu/~yxy/cs839-s20/papers/snap.pdf
Rothenberger, B., Taranov, K., Perrig, A., Hoefler, T.: ReDMArk: bypassing RDMA Security Mechanisms. In: Software for Exascale Computing - SPPEXA 2016–2019. USENIX (2021)
Shamis, P., et al.: Ucx: an open source framework for HPC network APIS and beyond. In: 2015 IEEE 23rd Annual Symposium on High-Performance Interconnects, pp. 40–43. IEEE (2015)
Von Eicken, T., Culler, D.E., Goldstein, S.C., Schauser, K.E.: Active messages: a mechanism for integrated communication and computation. ACM SIGARCH Comput. Arch. News 20(2), 256–266 (1992)
Acknowledgments
The authors would like to thank the Los Alamos National Laboratory for their continued support of this project. In addition, we would like thank Curtis Dunham, Megan Grodowitz, Jon Hermes, and Eric Van Hensbergen for their review of the paper and code.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Peña, L.E., Lu, W., Shamis, P., Poole, S. (2022). UCX Programming Interface for Remote Function Injection and Invocation. In: Poole, S., Hernandez, O., Baker, M., Curtis, T. (eds) OpenSHMEM and Related Technologies. OpenSHMEM in the Era of Exascale and Smart Networks. OpenSHMEM 2021. Lecture Notes in Computer Science, vol 13159. Springer, Cham. https://doi.org/10.1007/978-3-031-04888-3_9
Download citation
DOI: https://doi.org/10.1007/978-3-031-04888-3_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-04887-6
Online ISBN: 978-3-031-04888-3
eBook Packages: Computer ScienceComputer Science (R0)