A Smart HPC Interconnect for Clusters of Virtual Machines

  • Anastassios Nanos
  • Nikos Nikoleris
  • Stratos Psomadakis
  • Elisavet Kozyri
  • Nectarios Koziris
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7156)


In this paper, we present the design of a VM-aware, high-performance cluster interconnect architecture over 10Gbps Ethernet. Our framework provides a direct data path to the NIC for applications that run on VMs, leaving non-critical paths (such as control) to be handled by intermediate virtualization layers. As a result, we are able to multiplex and prioritize network access per VM. We evaluate our design via a prototype implementation that integrates RDMA semantics into the privileged guest of the Xen virtualization platform. Our framework allows VMs to communicate with the network using a simple user-level RDMA protocol. Preliminary results show that our prototype achieves 681MiB/sec over generic 10GbE hardware and relieves the guest from CPU overheads, while limiting the guest’s CPU utilisation to 34%.


Virtual Machine Virtual Machine Monitor Ethernet Frame USENIX Annual Technical Grant Mechanism 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Whitaker, A., Shaw, M., Gribble, S.D.: Denali: Lightweight virtual machines for distributed and networked applications. In: Proc. of the USENIX Annual Technical Conference (2002)Google Scholar
  2. 2.
  3. 3.
    Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T., Ho, A., Neugebauer, R., Pratt, I.A., Warfield, A.: Xen and the Art of Virtualization. In: SOSP 2003: Proc. of the 19th ACM Symposium on Operating Systems Principles, pp. 164–177. ACM, NY (2003)CrossRefGoogle Scholar
  4. 4.
    Recio, R., Culley, P., Garcia, D., Hilland, J.: An RDMA Protocol Specification (Version 1.0) This document is a Release Specification of the RDMA ConsortiumGoogle Scholar
  5. 5.
    Goglin, B.: Design and Implementation of Open-MX: High-Performance Message Passing over generic Ethernet hardware. In: CAC 2008: Workshop on Communication Architecture for Clusters, held in conjunction with IPDPS 2008. IEEE Computer Society Press, Miami (2008)Google Scholar
  6. 6.
    Shalev, L., Satran, J., Borovik, E., Ben-Yehuda, M.: IsoStack—Highly Efficient Network Processing on Dedicated Cores. In: USENIX ATC 2010: USENIX Annual Technical Conference (2010)Google Scholar
  7. 7.
    Youseff, L., Wolski, R., Gorda, B., Krintz, C.: Evaluating the Performance Impact of Xen on MPI and Process Execution For HPC Systems. In: 1st Intern. Workshop on Virtualization Techn. in Dstrb. Computing. VTDC 2006 (2006)Google Scholar
  8. 8.
    Nanos, A., Goumas, G., Koziris, N.: Exploring I/O Virtualization Data Paths for MPI Applications in a Cluster of VMs: A Networking Perspective. In: Guarracino, M.R., Vivien, F., Träff, J.L., Cannatoro, M., Danelutto, M., Hast, A., Perla, F., Knüpfer, A., Di Martino, B., Alexander, M. (eds.) Euro-Par-Workshop 2010. LNCS, vol. 6586, pp. 665–671. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  9. 9.
    Menon, A., Cox, A.L., Zwaenepoel, W.: Optimizing network virtualization in Xen. In: ATEC 2006: Proceedings of the Annual Conference on USENIX 2006 Annual Technical Conference, p. 2. USENIX, Berkeley (2006)Google Scholar
  10. 10.
    Ram, K.K., Santos, J.R., Turner, Y.: Redesigning xen’s memory sharing mechanism for safe and efficient I/O virtualization. In: WIOV 2010: Proceedings of the 2nd Conference on I/O Virtualization, p. 1. USENIX, Berkeley (2010)Google Scholar
  11. 11.
    Dong, Y., Dai, J., Huang, Z., Guan, H., Tian, K., Jiang, Y.: Towards high-quality I/O virtualization. In: SYSTOR 2009: Proceedings of SYSTOR 2009: The Israeli Experimental Systems Conference, pp. 1–8. ACM, NY (2009)CrossRefGoogle Scholar
  12. 12.
    Santos, J.R., Turner, Y., Janakiraman, G., Pratt, I.: Bridging the gap between software and hardware techniques for I/O virtualization. In: ATC 2008: USENIX 2008 Annual Technical Conference on Annual Technical Conference, pp. 29–42. USENIX, Berkeley (2008)Google Scholar
  13. 13.
    Ram, K.K., Santos, J.R., Turner, Y., Cox, A.L., Rixner, S.: Achieving 10 Gb/s using safe and transparent network interface virtualization. In: VEE 2009: Proceedings of the 2009 ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, pp. 61–70. ACM, NY (2009)CrossRefGoogle Scholar
  14. 14.
    Liu, J., Huang, W., Abali, B., Panda, D.K.: High performance VMM-bypass I/O in virtual machines. In: ATEC 2006: Proceedings of the Annual Conference on USENIX 2006 Annual Technical Conference, p. 3. USENIX, Berkeley (2006)Google Scholar
  15. 15.
    Nanos, A., Koziris, N.: MyriXen: Message Passing in Xen Virtual Machines over Myrinet and Ethernet. In: 4th Workshop on Virtualization in High-Performance Cloud Computing, The Netherlands (2009)Google Scholar
  16. 16.
    Jones, R.: Netperf,

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Anastassios Nanos
    • 1
  • Nikos Nikoleris
    • 2
  • Stratos Psomadakis
    • 1
  • Elisavet Kozyri
    • 3
  • Nectarios Koziris
    • 1
  1. 1.Computing Systems LaboratoryNational Technical University of AthensGreece
  2. 2.Uppsala Architecture Research TeamUppsala UniversitySweden
  3. 3.Cornell UniversityUSA

Personalised recommendations