Migration Techniques in HPC Environments

  • Simon Pickartz
  • Ramy Gad
  • Stefan Lankes
  • Lars Nagel
  • Tim Süß
  • André Brinkmann
  • Stephan Krempel
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8806)


Process migration is an important feature in modern computing centers as it allows for a more efficient use and maintenance of hardware. Especially in virtualized infrastructures it is successfully exploited by schemes for load balancing and energy efficiency. One can divide the tools and techniques into three groups: Process-level migration, virtual machine migration, and container-based migration.

This paper presents a qualitative and quantitative investigation of the different migration types for their application in (HPC). In addition to an overhead analysis of the various migration frameworks, our performance indicators include the migration time. The overall analysis suggests that VM migration has the most advantages and can even compete performance-wise.

The results are applied in the research project FaST addressing the problem of process scheduling in exascale environments. It is assumed that a shift in hardware architectures will result in a growing gap between the performance of CPUs and that of other resources like I/O. To avoid that these resources become bottlenecks, we suggest to monitor key performance indicators and, if conducive, trigger local amendments to the schedule requiring the efficient migration of jobs so that the downtime is reduced to a minimum.


Virtual Machine Migration Time Virtual Machine Migration Live Migration Virtualization Layer 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Dongarra, J.: Impact of Architecture and Technology for Extreme Scale on Software and Algorithm Design. Presented at the Department of Energy Workshop on Cross-cutting Technologies for Computing at the Exascale (February 2010)Google Scholar
  2. 2.
    Roman, E.: A Survey of Checkpoint/Restart Implementations. Technical report, Lawrence Berkeley National Laboratory, Tech (2002)Google Scholar
  3. 3.
    Wang, C., Mueller, F., Engelmann, C., Scott, S.: Proactive process-level live migration in hpc environments. In: International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2008, pp. 1–12 (November 2008)Google Scholar
  4. 4.
    Litzkow, M., Tannenbaum, T., Basney, J., Livny, M.: Checkpoint and Migration of UNIX Processes in the Condor Distributed Processing System. Technical Report UW-CS-TR-1346, University of Wisconsin – Madison Computer Sciences Department (April 1997)Google Scholar
  5. 5.
    Plank, J.S., Beck, M., Kingsley, G., Li, K.: Libckpt: Transparent Checkpointing under Unix (1995)Google Scholar
  6. 6.
    Duell, J.: The Design and Implementation of Berkeley Lab’s Linux Checkpoint/Restart. Technical report, Lawrence Berkeley National Laboratory (2003)Google Scholar
  7. 7.
    Hargrove, P.H., Duell, J.C.: Berkeley Lab Checkpoint/Restart (BLCR) for Linux Clusters. Journal of Physics: Conference Series 46(1), 494 (2006)Google Scholar
  8. 8.
    Sankaran, S., Squyres, J.M., Barrett, B., Lumsdaine, A.: The LAM/MPI Checkpoint/Restart Framework: System-Initiated Checkpointing. In: Proceedings of LACSI Symposium, Sante Fe, pp. 479–493 (2003)Google Scholar
  9. 9.
    Hursey, J., Squyres, J.M., Lumsdaine, A.: A Checkpoint and Restart Service Specification for Open MPI. Technical Report TR635, Indiana University, Bloomington, Indiana, USA (July 2006)Google Scholar
  10. 10.
    Milojičić, D.S., Douglis, F., Paindaveine, Y., Wheeler, R., Zhou, S.: Process migration. ACM Computing Surveys (CSUR) 32(3), 241–299 (2000)CrossRefGoogle Scholar
  11. 11.
    Clark, C., Fraser, K., Hand, S., Hansen, J.G., Jul, E., Limpach, C., Pratt, I., Warfield, A.: Live Migration of Virtual Machines. In: Proceedings of the 2nd Conference on Symposium on Networked Systems Design & Implementation, NSDI 2005, vol. 2 (2005)Google Scholar
  12. 12.
    Ranadive, A., Kesavan, M., Gavrilovska, A., Schwan, K.: Performance implications of virtualizing multicore cluster machines. In: Proceedings of the 2nd Workshop on System-level Virtualization for High Performance Computing, HPCVirt 2008, pp. 1–8. ACM, New York (2008)Google Scholar
  13. 13.
    Birkenheuer, G., Brinkmann, A., Kaiser, J., Keller, A., Keller, M., Kleineweber, C., Konersmann, C., Niehörster, O., Schäfer, T., Simon, J., Wilhelm, M.: Virtualized HPC: a contradiction in terms?. Softw., Pract. Exper. 42(4), 485–500 (2012)CrossRefGoogle Scholar
  14. 14.
    Younge, A.J., Henschel, R., Brown, J.T., von Laszewski, G., Qiu, J., Fox, G.C.: Analysis of Virtualization Technologies for High Performance Computing Environments. In: Proceedings of 2011 IEEE International Conference on Cloud Computing (CLOUD), pp. 9–16. IEEE (2011)Google Scholar
  15. 15.
    Intel Virtualization Technology for Directed I/O. Technical report, Intel Corporation (2013)Google Scholar
  16. 16.
    Intel LAN Access Division: PCI-SIG SR-IOV Primer. Technical Report 2.5, Intel Corporation (January 2011)Google Scholar
  17. 17.
    Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T., Ho, A., Neugebauer, R., Pratt, I., Warfield, A.: Xen and the Art of Virtualization. SIGOPS Oper. Syst. Rev. 37(5), 164–177 (2003)CrossRefGoogle Scholar
  18. 18.
    Kivity, A., Kamay, Y., Laor, D., Lublin, U.: kvm: the Linux Virtual Machine Monitor. In: Proceedings of the Linux Symposium, vol. 1, Ottawa, Ontario, Canada, pp. 225–230 (June 2007)Google Scholar
  19. 19.
    Nussbaum, L., Anhalt, F., Mornard, O., Gelas, J.P.: Linux-based virtualization for HPC clusters. In: Proceedings of the Linux Symposium (July 2009)Google Scholar
  20. 20.
    Uhlig, R., Neiger, G., Rodgers, D., Santoni, A.L., Martins, F.C.M., Anderson, A.V., Bennett, S.M., Kagi, A., Leung, F.H., Smith, L.: Intel Virtualization Technology. Computer 38(5), 48–56 (2005)CrossRefGoogle Scholar
  21. 21.
    Virtualization, A.: Secure Virtual Machine Architecture Reference Manual. AMD Publication (2005)Google Scholar
  22. 22.
    Regola, N., Ducom, J.C.: Recommendations for Virtualization Technologies in High Performance Computing. In: Proceedings of 2nd IEEE International Conference on Cloud Computing Technology and Science (CloudCom), pp. 409–416 (November 2010)Google Scholar
  23. 23.
    Darling, A., Carey, L., Feng, W.: The design, implementation, and evaluation of mpiBLAST. In: Proceedings of ClusterWorld (2003)Google Scholar
  24. 24.
    Bailey, D.H., Barszcz, E., Barton, J.T., Browning, D.S., Carter, R.L., Dagum, L., Fatoohi, R.A., Frederickson, P.O., Lasinski, T.A., Schreiber, R.S., Simon, H.D., Venkatakrishnan, V., Weeratunga, S.K.: The NAS Parallel Benchmarks. International Journal of High Performance Computing Applications 5(3), 63–73 (1991)CrossRefGoogle Scholar
  25. 25.
    Hu, W., Hicks, A., Zhang, L., Dow, E.M., Soni, V., Jiang, H., Bull, R., Matthews, J.N.: A quantitative study of virtual machine live migration (August 2013)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Simon Pickartz
    • 1
  • Ramy Gad
    • 2
  • Stefan Lankes
    • 1
  • Lars Nagel
    • 2
  • Tim Süß
    • 2
  • André Brinkmann
    • 2
  • Stephan Krempel
    • 3
  1. 1.Institute for Automation of Complex Power Systems, E.ON Energy Research CenterRWTH Aachen UniversityAachenGermany
  2. 2.Zentrum für DatenverarbeitungJohannes Gutenberg UniversitätMainzGermany
  3. 3.ParTec Cluster Competence Center GmbHMunichGermany

Personalised recommendations