Performance Enhancement of SMP Clusters with Multiple Network Interfaces Using Virtualization

  • Peter Strazdins
  • Richard Alexander
  • David Barr
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4331)


Clusters of small-scale SMP/CMP nodes are becoming increasingly popular due to their cost-effectiveness. As these nodes are typically capable of supporting a number of network interfaces similar to the number of CPUs, the issue arises how to optimally configure the cluster for optimum communication performance. This paper evaluates a number of configurations on a 4-CPU Opteron cluster with multiple Gigabit Ethernet interfaces. Techniques include channel bonding and using independent communication pathways. With the latter, the use of virtualization via the Xen Virtual Machine Monitor offers the best potential to parallelize all stages of message transmission, for the case when multiple CPUs on a node are communicating simultaneously. Network-level microbenchmarks indicate the best performance is achieved with a configuration where guest virtual machines running on each CPU communicate directly with a dedicated interface, bypassing the virtual machine monitor. Channel bonding also proved to be more effective over multiple communication streams than over single.


Virtual Machine Network Interface Virtual Machine Monitor Virtual Interface Channel Bonding 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aberdeen, D., Baxter, J., Edwards, R.: A 98c/MFLOP Ultra-Large Scale Neural Network Training on a PIII Cluster. In: Proceedings of Supercomputing 2000 (2000)Google Scholar
  2. 2.
    Capello, F., Richard, O., Etiemble, D.: Understanding performance of SMP clusters running MPI programs. Future Generation Computer Systems 17, 711–720 (2001)CrossRefGoogle Scholar
  3. 3.
    Pukayastha, A., Guiang, C.S., Schulz, K., Minyard, T., Milfeld, K., Barth, W., Hurley, P., Boisseau, J.R.: Performance Characteristics of Dual-processor HPC Cluster Nodes based on 64-bit Commodity Processors. In: Proceedings of the Linux Clusters Institute (LCI) International Conference: the HPC Revolution (2004)Google Scholar
  4. 4.
    Advanced Microelectronic Devices: AMD Microprocessor Solutions,
  5. 5.
    Barham, P., Dragovic, B., Fraser, K., Harris, S.H.T., Ho, A., Neugebauer, R., Pratt, I., Warfield, A.: Xen and the Art of Virtualization. In: Proceedings of SOSP 2003: the Nineteenth ACM Symposium on Operating Systems Principles, pp. 164–177. ACM, New York (2003)CrossRefGoogle Scholar
  6. 6.
    Huang, W., Liu, J., Abali, B., Panda, D.: A Case fopr High Performance Computing with Virtual Machines. In: Proceedings of ICS 2006: International Conference of Supercomputing, Cairns (2006)Google Scholar
  7. 7.
    Turner, D., Oline, A., Chen, X., Benjegerdes, T.: Integrating New Capabilities into NetPIPE. In: 10th European PVM/MPI User’s Group Meeting, Venice, pp. 37–44. Springer, Heidelberg (2003)Google Scholar
  8. 8.
    Liu, J., Chandrasekaran, B., Wu, J., Jiang, W., Kini, S., Yu, W., Buntinas, D., Wyckoff, P., Panda, D.: Performance Comparison of MPI Implementations over Infiniband, Myrinet and Quadrics. In: Proceedings of the SuperComputing 2003 Conference, Phoenix (2003)Google Scholar
  9. 9.
    Menon, A., Jose Renato Santos, A.Y.T., Janakiraman, G., Zwaenepoel, W.: Diagnosing Performance Overheads in the Xen Virtual Machine Environment. In: First ACM/USENIX Conference on Virtual Execution Environments (VEE 2005), pp. 13–25 (2005)Google Scholar
  10. 10.
    Menon, A., Cox, A.L., Zwaenepoel, W.: Optimizing Network Virtualization in Xen. In: Proceedings of the 2006 USENIX Annual Technical Conference, Boston, pp. 15–28 (2006)Google Scholar
  11. 11.
    Liu, J., Huang, W., Abali, B., Panda, D.: High Performance VMM-Bypass I/O in Virtual Machines. In: Proceedings of the 2006 USENIX Annual Technical Conference, Boston (2006)Google Scholar
  12. 12.
    University of Cambridge Computing Laboratory: The Xen virtual machine monitor,
  13. 13.
    Willmann, P., Rixner, S., Cox, A.L.: An Evaluation of Network Stack Parallelization Strategies in Modern Operating Systems. Technical Report TR06-872, Rice University Computer Science (2006)Google Scholar
  14. 14.
    Bhattacharya, S.P., Apte, V.: A Measurement Study of the Linux TCP/IP Stack Performance and Scalability on SMP systems. In: Proceedings of the 1st International Conference on COMmunication Systems softWAre and middlewaRE (COMSWARE), New Delhi (2006)Google Scholar
  15. 15.
    Nowlabs, Ohio State University: MVAPICH2 Toolset,

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Peter Strazdins
    • 1
  • Richard Alexander
    • 2
  • David Barr
    • 2
  1. 1.Department of Computer ScienceAustralian National University 
  2. 2.Alexander TechnologyCanberra

Personalised recommendations