Paravirtualization for HPC Systems

  • Lamia Youseff
  • Rich Wolski
  • Brent Gorda
  • Chandra Krintz
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4331)


In this work, we investigate the efficacy of using paravirtualizing software for performance-critical HPC kernels and applications. We present a comprehensive performance evaluation of Xen, a low-overhead, Linux-based, virtual machine monitor, for paravirtualization of HPC cluster systems at LLNL. We investigate subsystem and overall performance using a wide range of benchmarks and applications. We employ statistically sound methods to compare the performance of a paravirtualized kernel against three Linux operating systems: RedHat Enterprise 4 for build versions 2.6.9 and 2.6.12 and the LLNL CHAOS kernel. Our results indicate that Xen is very efficient and practical for HPC systems.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Whitaker, A., Shaw, M., Gribble, S.: Scale and Performance in the Denali Isolation Kernel. In: Symposium on Operating Systems Design and Implementation (OSDI) (2002),
  2. 2.
    Adcroft, A., Campin, J., Heimbach, P., Hill, C., Marshall, J.: MIT-GCM User Manual. In: Earth, Atmospheric and Planetary Sciences, Massachusetts Institute of Technology (2002)Google Scholar
  3. 3.
    LLNL ASC Purple Benchmark Suite,
  4. 4.
    Bagley, J.D., Floto, E.R., Hsieh, S.C., Watson, V.: Sharing data and services in a virtual machine system. In: SOSP 1975: Proceedings of the fifth ACM symposium on Operating systems principles, pp. 82–88. ACM Press, New York (1975)CrossRefGoogle Scholar
  5. 5.
    Bailey, D., Harris, T., Saphir, W., van der Wijngaart, R., Woo, A., Yarrow, M.: The nas parallel benchmarks 2.0. The International Journal of Supercomputer Applications (1995)Google Scholar
  6. 6.
    Bailey, D.H., Barszcz, E., Barton, J.T., Browning, D.S., Carter, R.L., Dagum, D., Fatoohi, R.A., Frederickson, P.O., Lasinski, T.A., Schreiber, R.S., Simon, H.D., Venkatakrishnan, V., Weeratunga, S.K.: The nas parallel benchmarks. The International Journal of Supercomputer Applications 5(3), 63–73 (1991)CrossRefGoogle Scholar
  7. 7.
    Bjerke, H.: HPC Virtualization with Xen on Itanium. Master’s thesis, Norwegian University of Science and Technology (NTNU) (July 2005)Google Scholar
  8. 8.
    BMJ Publishing Group: Statistics at Square One: The t Tests (2006)
  9. 9.
    Bonnie Disk I/O Benchmark,
  10. 10.
    Braby, R., Garlick, J., Goldstone, R.: Achieving Order through CHAOS: the LLNL HPC Linux Cluster Experience (June 2003)Google Scholar
  11. 11.
    Clark, B., Deshane, T., Dow, E., Evanchik, S., Finlayson, M., Herne, J., Matthews, J.N.: Xen and the art of repeated research. In: USENIX Annual Technical Conference, FREENIX Track, pp. 135–144 (2004)Google Scholar
  12. 12.
    Clark, C., Fraser, K., Hand, S., Hansen, J.G., Jul, E., Limpach, C., Pratt, I., Warfield, A.: Live Migration of Virtual Machines. In: USENIX Symposium on Networked Systems Design and Implementation (NSDI 2005), Boston, MA, USA (May 2005)Google Scholar
  13. 13.
    Clustered High Availability Operating System (CHAOS) Overview,
  14. 14.
    Duda, K.J., Cheriton, D.R.: Borrowed-virtual-time (BVT) scheduling: supporting latency-sensitive threads in a general-purpose schedular. In: Symposium on Operating Systems Principles, pp. 261–276 (1999)Google Scholar
  15. 15.
    Van Hensbergen, E.: The Effect of Virtualization on OS Interference. In: Workshop on Operating System Interference in High Performance Applications, held in cooperation with The Fourteenth International Conference on Parallel Architectures and Compilation Techniques: PACT 2005 (September 2005),
  16. 16.
    Galley, S.W.: PDP-10 virtual machines. In: Proceedings of the workshop on virtual computer systems, pp. 30–34. ACM Press, New York (1973)CrossRefGoogle Scholar
  17. 17.
    Hansen, J., Jul, E.: Self-migration of Operating Systems. In: ACM SIGOPS European Workshop (EW 2004), pp. 126–130 (2004)Google Scholar
  18. 18.
    Bjerke, H., Andresen, R.: Virtualization in clusters (2004),
  19. 19.
    Hensbergen, E.V.: PROSE: Partitioned Reliable Operating System Environment. In: IBM Research Technical Report RC23694 (2005)Google Scholar
  20. 20.
    Sugerman, J., Venkitachalam, G., Lim, B.: Virtualizing I/O devices on VMware workstations hosted virtual machine monitor. In: USENIX Annual Technical Conference (2001)Google Scholar
  21. 21.
    Kozuch, M., Satyanarayanan, M.: Internet suspend/resume. In: WMCSA 2002: Proceedings of the Fourth IEEE Workshop on Mobile Computing Systems and Applications, Washington, DC, USA, p. 40. IEEE Computer Society, Los Alamitos (2002)CrossRefGoogle Scholar
  22. 22.
    Krintz, C., Wolski, R.: Using phase behavior in scientific application to guide linux operating system customization. In: Workshop on Next Generation Software at IEEE International Parallel and Distributed Processing Symposium (IPDPS) (April 2005)Google Scholar
  23. 23.
    Larsen, R.J., Marx, M.L.: An Introduction to Mathematical Statistics and Its Applications, 3rd edn. Prentice Hall, Englewood Cliffs (2001)Google Scholar
  24. 24.
    Luszczek, P., Dongarra, J., Koester, D., Rabenseifner, R., Lucas, B., Kepner, J., McCalpin, J., Bailey, D., Takahashi, D.: Introduction to the hpc challenge benchmark suite (March 2005),
  25. 25.
    Madnick, S.E., Donovan, J.J.: Application and analysis of the virtual machine approach to information system security and isolation. In: Proceedings of the workshop on virtual computer systems, pp. 210–224. ACM Press, New York (1973)CrossRefGoogle Scholar
  26. 26.
    Marotzke, J., Giering, R., et al.: Construction of the adjoint MIT ocean general circulation model and application to Atlantic heat transport sensitivity. Journal of Geophysical Research 104(C12) (1999)Google Scholar
  27. 27.
    MIT’s Climate Modeling Initiative,
  28. 28.
    Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T., Ho, A., Neugebauer, R.: Virtual machine monitors: Xen and the art of virtualization. In: Symposium on Operating System Principles (2003),
  29. 29.
    AMD Virtualization Codenamed ”Pacifica” Technology, Secure Virtual Machine Architecture Reference Manual (May 2005)Google Scholar
  30. 30.
    Popek, G.J., Goldberg, R.P.: Formal requirements for virtualizable third generation architectures. Commun. ACM 17(7), 412–421 (1974)zbMATHCrossRefMathSciNetGoogle Scholar
  31. 31.
    Popek, G.J., Kline, C.S.: PDP-11 virtual machine architecture: A case study. In: SOSP 1975: Proceedings of the fifth ACM symposium on Operating systems principles, pp. 97–105. ACM Press, New York (1975)CrossRefGoogle Scholar
  32. 32.
    Meyer, R.A., Seawright, L.H.: A Virtual Machine Time Sharing System. IBM Systems Journal, 199–218 (1970)Google Scholar
  33. 33.
    Rosenblum, M., Garfinkel, T.: Virtual machine monitors: Current technology and future trends. Computer 38(5), 39–47 (2005)CrossRefGoogle Scholar
  34. 34.
    Smith, J.E., Nair, R.: Virtual Machines: Versatile Platforms for Systems and Processes. Morgan Kaufmann/Elsevier (2005)Google Scholar
  35. 35.
    Sokolowski, P.J., Grosu, D.: Performance considerations for network switch fabrics on linux clusters. In: Proceedings of the 16th IASTED International Conference on Parallel and Distributed Computing and Systems (November 2004)Google Scholar
  36. 36.
    The memory stress benchmark codes: stream,
  37. 37.
    Enhanced Virtualization on Intel Architecture-based Servers (March 2005)Google Scholar
  38. 38.
    Waldspurger, C.A.: Memory resource management in vmware esx server. SIGOPS Oper. Syst. Rev. 36(SI), 181–194 (2002)CrossRefGoogle Scholar
  39. 39.
    Whitaker, A., Cox, R., Shaw, M., Gribble, S.: Constructing services with interposable virtual hardware (2004)Google Scholar
  40. 40.
    Xen Virtual Machine Monitor Performance,
  41. 41.
    Xenidis, J.: rHype: IBM Research Hypervisor. In: IBM Research (March 2005),
  42. 42.
  43. 43.
    Youseff, L., Wolski, R., Gorda, B., Krintz, C.: Paravirtualization for HPC Systems. Technical Report Technical Report Numer 2006-10, Computer Science Department University of California, Santa Barbara (August 2006)Google Scholar
  44. 44.
    Youseff, L., Wolski, R., Krintz, C.: Linux kernel specialization for scientific application performance. Technical Report UCSB Technical Report 2005-29, Univ. of California, Santa Barbara (November 2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Lamia Youseff
    • 1
  • Rich Wolski
    • 1
  • Brent Gorda
    • 2
  • Chandra Krintz
    • 1
  1. 1.Department of Computer ScienceUniversity of CaliforniaSanta Barbara
  2. 2.Lawrence Livermore National Lab (LLNL) 

Personalised recommendations