Skip to main content

Interactive Supercomputing for Experimental Data-Driven Workflows

Part of the Communications in Computer and Information Science book series (CCIS,volume 1190)


Large scale experimental facilities such as the Swiss Light Source and the free-electron X-ray laser SwissFEL at the Paul Scherrer Institute, and the particle accelerators and detectors at CERN are experiencing unprecedented data generation growth rates. Consequently, management, processing and storage requirements of data are increasing rapidly. Historically, online and on-demand processing of data generated by the instruments used to be tightly-coupled with a dedicated, domains-specific, site-local IT infrastructure. Cost and performance scaling of these facilities not only pose technical but also planning and scheduling challenges. Supercomputing ecosystems optimize cost and scaling for computing and storage resources but typically exploit a shared batch access model, which is optimized for high utilization of compute resources. In comparison, in public clouds, on-demand service delivery models address the concept of elasticity while maintaining isolation with performance trade-offs. Furthermore, these on-demand access models allow for different degrees of privileges to users for managing IT infrastructure services, in contrast with shared, bare-metal supercomputing ecosystems. This paper outlines an approach for enabling interactive, on-demand supercomputing for experimental data-driven workflows, which are characterised by a managed but bursty data and computing requirements. We present a delegated batch reservation model, controlled by the customer and provisioned by the supercomputing site, that allows scientists at the experimental facility to couple generation of data to the allocation of compute, data and network resources at the supercomputing centre. Scientists are then able to manage resources both at the experimental and supercomputing facilities interactively for managing their scientific workflows. Prototype implementation demonstrates that this rather simple co-designed extension to a supercomputing classic batch scheduling system with a controlled degree of privilege can be easily incorporated to the experimental facilities existing IT resource management and scheduling pipelines.

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-44728-1_10
  • Chapter length: 15 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
USD   64.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-44728-1
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   84.99
Price excludes VAT (USA)
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.


  1. 1.


  1. Alam, S.R., Gilly, L., McMurtrie, C., Schulthess, T.C.: CSCS and the Piz Daint System, pp. 149–174, May 2019

    Google Scholar 

  2. Alam, S.R., Martinasso, M., Schulthess, T.C.: Hybrid cloud and HPC services for extreme data workflows. In: Extreme Data: Demands, Technologies, and Services - A Community Workshop (2018)

    Google Scholar 

  3. Benedicic, L., Cruz, F.A., Madonna, A., Mariotti, K.: Portable, high-performance containers for HPC. CoRR, abs/1704.03383 (2017)

    Google Scholar 

  4. Cameron, D., et al.: The advanced resource connector for distributed LHC computing. PoS (2008)

    Google Scholar 

  5. Martinasso, M., Gila, M., Bianco, M., Alam, S.R., McMurtrie, C., Schulthess, T.C.: RM-replay: a high-fidelity tuning, optimization and exploration tool for resource management. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018 (2018)

    Google Scholar 

  6. Milne, C., et al.: SwissFEL: the Swiss X-ray free electron laser. Appl. Sci. 7(7), 720 (2017)

    CrossRef  Google Scholar 

  7. Paul Scherrer Institut: cSAXS X12SA: Coherent Small-Angle X-ray Scattering. Accessed 20 Sept 2019

  8. SchedMD: Slurm workload manager - scontrol. Accessed 20 Sept 2019

  9. SchedMD: Slurm workload manager - user permissions. Accessed 20 Sept 2019

Download references


We would like to thank our colleagues at PSI for their insightful remarks and their input for co-designing the early prototype. The work presented in this paper is partly funded by a swissuniversities P-5 grant called SELVEDAS (Services for Large Volume Experiment-Data Analysis utilising Supercomputing and Cloud technologies at CSCS).

Author information

Authors and Affiliations


Corresponding author

Correspondence to Siew Hoon Leong .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Klein, M., Martinasso, M., Leong, S.H., Alam, S.R. (2020). Interactive Supercomputing for Experimental Data-Driven Workflows. In: Juckeland, G., Chandrasekaran, S. (eds) Tools and Techniques for High Performance Computing. HUST SE-HER WIHPC 2019 2019 2019. Communications in Computer and Information Science, vol 1190. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-44727-4

  • Online ISBN: 978-3-030-44728-1

  • eBook Packages: Computer ScienceComputer Science (R0)