Delivering LHC Software to HPC Compute Elements with CernVM-FS

  • Jakob BlomerEmail author
  • Gerardo Ganis
  • Nikola Hardi
  • Radu Popescu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10524)


In recent years, there was a growing interest in improving the utilization of supercomputers by running applications of experiments at the Large Hadron Collider (LHC) at CERN when idle cores cannot be assigned to traditional HPC jobs. At the same time, the upcoming LHC machine and detector upgrades will produce some 60 times higher data rates and challenge LHC experiments to use so far untapped compute resources. LHC experiment applications are tailored to run on high-throughput computing resources and they have a different anatomy than HPC applications. LHC applications comprise a core framework that allows hundreds of researchers to plug in their specific algorithms. The software stacks easily accumulate to many gigabytes for a single release. New releases are often produced on a daily basis. To facilitate the distribution of these software stacks to world-wide distributed computing resources, LHC experiments use a purpose-built, global, POSIX file system, the CernVM File System. CernVM-FS pre-processes data into content-addressed, digitally signed Merkle trees and it uses web caches and proxies for data distribution. Fuse-mounted files system clients on the compute nodes load and cache on demand only the small fraction of files needed at any given moment. In this paper, we report on problems and lessons learned in the deployment of CernVM-FS on supercomputers such as the supercomputers at NERSC in Berkeley, at LRZ in Munich, and at CSCS in Lugano. We compare CernVM-FS to a shared software area on a traditional HPC storage system and to container-based systems.


  1. 1.
    Bird, I., et al.: LHC computing grid: Technical design report. Technical report LCG-TDR-001, CERN (2005)Google Scholar
  2. 2.
    Blomer, J., Aguado-Sanchez, C., Buncic, P., Harutyunyan, A.: Distributing LHC application software and conditions databases using the CernVM file system. J. Phys. Conf. Ser. 331, 042003 (2011). CrossRefGoogle Scholar
  3. 3.
    Blomer, J., Buncic, P., Meusel, R., Ganis, G., Sfiligoi, I., Thain, D.: The evolution of global scale filesystems for scientific software distribution. Comput. Sci. Eng. 17(6), 61–71 (2015)CrossRefGoogle Scholar
  4. 4.
    Blomer, J., Fuhrmann, T.: A fully decentralized file system cache for the CernVMFS. In: Proceedings 10th International Conference on Computer and Communications Networks (ICCCN), August 2010Google Scholar
  5. 5.
    Canon, S., Jacobsen, D.: Shifter: Containers for hpc. In: Proceedings of the Cray User Group (2016)Google Scholar
  6. 6.
    Childersa, J.T., Gerhardt, L.: Developments in architectures and services for using high performance computing in energy frontier experiments. In: Proceedings 38th International Conference on High Energy Physics (ICHEP 2016) (2016)Google Scholar
  7. 7.
    Fasel, M.: Using nersc high-performance computing (hpc) systems for high-energy nuclear physics applications with alice. J. Phys. Conf. Ser. 762, 012031 (2016). IOP PublishingCrossRefGoogle Scholar
  8. 8.
    Filipcic, A., Haug, S., Hostettler, M., Walker, R., Weber, M.: Atlas computing on cscs hpc. J. Phys. Conf. Ser. 664, 092011 (2015). IOP PublishingCrossRefGoogle Scholar
  9. 9.
    Gardner, R.: Xsede integration. In: US ATLAS Physics Support, Software and Computing Technical Planning Meeting (2016)Google Scholar
  10. 10.
    Henk, C., Szeredi, M.: Filesystem in Userspace (FUSE).,
  11. 11.
    Mashinistov, R.: Panda @ nrc ki. Talk at the PanDA Workshop (2016)Google Scholar
  12. 12.
    Meier, K., Fleig, G., Hauth, T., Janczyk, M., Quast, G., von Suchodoletz, D., Wiebelt, B.: Dynamic provisioning of a hep computing infrastructure on a shared hybrid hpc system. J. Phys. Conf. Ser. 762, 012012 (2016). IOP PublishingCrossRefGoogle Scholar
  13. 13.
    Nilsson, P., Panitkin, S., Oleynik, D., Maeno, T., De, K., Wu, W., Filipcic, A., Wenaus, T., Klimentov, A.: Extending atlas computing to commercial clouds and supercomputers. PoS, p. 034 (2014)Google Scholar
  14. 14.
    Sugiyama, S., Wallace, D.: Cray dvs: Data virtualization service. In: Cray User Group Annual Technical Conference (2008)Google Scholar
  15. 15.
    Thain, D., Livny, M.: Parrot: an application environment for data-intensive computing. Scalable Comput. Pract. Experience 6(3), 9–18 (2005)Google Scholar
  16. 16.
    Vangoor, B.K.R., Tarasov, V., Zadok, E.: To FUSE or Not to FUSE: performance of user-space file systems. In: Proceedings of the 15th USENIX Conference on File and Storage Technologies (FAST 2017) (2017)Google Scholar
  17. 17.
    Walker, R.: Hep software on supercomputers. In: Talk at the CernVM Users Workshop (2016)Google Scholar
  18. 18.
    Washbrook, A.: Processing lhc workloads on archer. In: Talk at GridPP35 Conference (2015)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Jakob Blomer
    • 1
    Email author
  • Gerardo Ganis
    • 1
  • Nikola Hardi
    • 1
  • Radu Popescu
    • 1
  1. 1.European Organization for Particle Research (CERN)GenevaSwitzerland

Personalised recommendations