Lobachevskii Journal of Mathematics

, Volume 39, Issue 9, pp 1170–1178 | Cite as

Software Platform for European XFEL: Towards Online Experimental Data Analysis

  • S. A. BobkovEmail author
  • A. B. Teslyuk
  • S. I. Zolotarev
  • M. Rose
  • K. A. Ikonnikova
  • V. E. Velikhov
  • I. A. Vartanyants
  • V. A. Ilyin
Part 1. Special issue “High Performance Data Intensive Computing” Editors: V. V. Voevodin, A. S. Simonov, and A. V. Lapin


Large amount of data being generated at large scale facilities like European X-ray Free- Electron Laser (XFEL) requires new approaches for data processing and analysis. One of the most computationally challenging experiments at an XFEL is single-particle structure determination. In this paper we propose a new design for an integrated software platform which combines well-established techniques for XFEL data analysis with High Performance Data Analysis (HPDA) methods. In our software platform we use streaming data analysis algorithms with high performance computing solutions. This approach should allow analysis of the experimental dataflow in quasi-online regime.

Keywords and phrases

single particle imaging Expectation-Maximization X-ray Free- Electron Laser High Performance Data Analysis X-ray Cross-Correlation Analysis 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    A. Osseyran and M. Giles, Industrial Applications of High-Performance Computing: Best Global Practices (Chapman and Hall/CRC, Boca Raton, 2015).CrossRefGoogle Scholar
  2. 2.
    K. J. Gaffney and H. N. Chapman, “Imaging atomic structure and dynamics with ultrafast X-ray scattering,” Science (Washington, DC, U. S.) 316 (5830), 1444–1448 (2007).CrossRefGoogle Scholar
  3. 3.
    R. Neutze, R. Wouts, D. van der Spoel, E. Weckert, and J. Hajdu, “Potential for biomolecular imaging with femtosecond X-ray pulses,” Nature (London, U. K.) 406 (6797), 752 (2000).CrossRefGoogle Scholar
  4. 4.
    M. Altarelli, R. Brinkmann, M. Chergui, W. Decking, B. Dobson, S. Düsterer, G. Grübel, W. Graeff, H. Graafsma, J. Hajdu, et al., in The European X-Ray Free-Electron Laser, Technical Design Report, No. 97, 1–26 (2006).Google Scholar
  5. 5.
    A. Aquila, A. Barty, C. Bostedt, S. Boutet, G. Carini, D. DePonte, P. Drell, et al., “The linac coherent light source single particle imaging road map,” Struct. Dyn. 2, 041701 (2015).CrossRefGoogle Scholar
  6. 6.
    H. N. Chapman, A. Barty, M. J. Bogan, S. Boutet, M. Frank, S. P. Hau-Riege, S. Marchesini, B. W. Woods, S. Bajt, W. H. Benner, et al., “Femtosecond diffractive imaging with a soft-X-ray free-electron laser,” Nat. Phys. 2, 839 (2006).CrossRefGoogle Scholar
  7. 7.
    M. M. Seibert, T. Ekeberg, F. Maia, M. Svenda, J. Andreasson, O. Jönsson, D. Odić, B. Iwan, A. Rocker, D. Westphal, et al., “Single mimivirus particles intercepted and imaged with an X-ray laser,” Nature (London, U. K.) 470 (7332), 78 (2011).CrossRefGoogle Scholar
  8. 8.
    M. F. Hantke, D. Hasse, F. R. N. C. Maia, T. Ekeberg, K. John, M. Svenda, N. Duane Loh, A. V. Martin, N. Timneanu, D. S. D. Larsson, et al., “High-throughput imaging of heterogeneous cell organelles with an X-ray laser,” Nat. Photon. 8, 943 (2014).CrossRefGoogle Scholar
  9. 9.
    T. Ekeberg, M. Svenda, C. Abergel, F. Maia, V. Seltzer, J. Claverie, M. Hantke, O. Jönsson, C. Nettelblad, G. Van Der Schot, et al., “Three-dimensional reconstruction of the giant mimivirus particle with an x-ray free-electron laser,” Phys. Rev. Lett. 114, 098102 (2015).CrossRefGoogle Scholar
  10. 10.
    R. P. Kurta, J. J. Donatelli, C. H. Yoon, P. Berntsen, J. Bielecki, B. J. Daurer, H. DeMirci, P. Fromme, M. Hantke, F. Maia, et al., “Correlations in scattered x-ray laser pulses reveal nanoscale structural features of viruses,” Phys. Rev. Lett. 119, 158102 (2017).CrossRefGoogle Scholar
  11. 11.
    A. Barty, R. A. Kirian, F. R. N. C. Maia,M. Hantke, C. H. Yoon, T. A. White, and H. Chapman, “Cheetah: software for high-throughput reduction and analysis of serial femtosecond X-ray diffraction data,” J. Appl. Cryst. 37, 1118 (2014).CrossRefGoogle Scholar
  12. 12.
    K. Ayyer, T. Lan, V. Elser, and N. D. Loh, “Dragonfly: an implementation of the expand–maximize–compress algorithm for single-particle imaging,” Int. J. Appl. Cryptogr. 49, 1320–1335 (2016).Google Scholar
  13. 13.
    F. R. N. C. Maia, T. Ekeberg, D. van der Spoel, and J. Hajdu, “Hawk: the image reconstruction package for coherent x-ray diffractive imaging,” J. Appl. Cryst. 43, 1535 (2010).CrossRefGoogle Scholar
  14. 14.
    B. Henrich, J. Becker, R. Dinapoli, P. Goettlicher, H. Graafsma, H. Hirsemann, R. Klanner, H. Krueger, R. Mazzocco, A. Mozzanica, et al., “The adaptive gain integrating pixel detector AGIPD a detector for the European XFEL,” Nucl. Instrum. Methods Phys. Res., Sect. A 633, S11–S14 (2011).Google Scholar
  15. 15.
    S. A. Bobkov, “Comparison study of different approaches to classification of diffraction images of biological particles obtained in coherent X-ray diffractive imaging experiments,” Math. Biol. Bioinform. 12, 411–434 (2017).CrossRefGoogle Scholar
  16. 16.
    F. Maia, “The coherent X-ray imaging data bank,” Nat. Methods 9, 854–855 (2012).CrossRefGoogle Scholar
  17. 17.
    B. J. Daurer, K. Okamoto, J. Bielecki, F. Maia, K. Mühlig, M. Seibert, M. Hantke, C. Nettelblad, W. H. Benner, M. Svenda, et al., “Experimental strategies for imaging bioparticles with femtosecond hard X-ray pulses,” Int. Union Cryst. J. 4, 854–855 (2017).CrossRefGoogle Scholar
  18. 18.
    J. Andreasson, A. V. Martin, M. Liang, N. Timneanu, A. Aquila, F. Wang, B. Iwan, M. Svenda, T. Ekeberg, M. Hantke, et al., “Automated identification and classification of single particle serial femtosecond X-ray diffraction data,” Opt. Express 22, 2497–2510 (2014).CrossRefGoogle Scholar
  19. 19.
    S. A. Bobkov, A. B. Teslyuk, R. P. Kurta, O. Yu. Gorobtsov, O. M. Yefanov, V. A. Ilyin, R. A. Senin, and I. A. Vartanyants, “Sorting algorithms for single-particle imaging experiments at X-ray free-electron lasers,” J. Synchrotr. Rad. 22, 1345–1352 (2015).CrossRefGoogle Scholar
  20. 20.
    R. P. Kurta, M. Altarelli, and I. A. Vartanyants, “Structural analysis by x-ray intensity angular cross correlations,” Adv. Chem. Phys. 161, 1–39 (2016).Google Scholar
  21. 21.
    N. D. Loh and V. Elser, “Reconstruction algorithm for single-particle diffraction imaging experiments,” Phys. Rev.. 80, 026705 (2009).Google Scholar
  22. 22.
    A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum likelihood from incomplete data via the EM algorithm,” J. R. Stat. Soc., Ser. B, 1–38 (1977).Google Scholar
  23. 23.
    M. Kevin, Machine Learning: A Probabilistic Perspective (MIT Press, Cambridge,MA, 2012).zbMATHGoogle Scholar
  24. 24.
    R. M. Neal and G. E. Hinton, “A view of the EM algorithm that justifies incremental, sparse, and other variants,” in Learning in GraphicalModels (MIT Press, Cambridge,MA, 1998).Google Scholar
  25. 25.
    M. A. Sato and S. Ishii, “On-line EMalgorithm for the normalized Gaussian network,” Neural Comput. 12, 407–432 (2000).CrossRefGoogle Scholar
  26. 26.
    O. Cappeánd E. Moulines, “On-line expectation–maximization algorithmfor latent datamodels,” J. R. Stat. Soc., Ser. B 71, 593–613 (2009).MathSciNetCrossRefGoogle Scholar
  27. 27.
    G. Celeux, D. Chauveau, and J. Diebolt, “On stochastic versions of the EM algorithm,” Research Report RR-2514 (INRIA, 1995).Google Scholar
  28. 28.
    S. F. Nielsen et al., “The stochastic EMalgorithm: estimation and asymptotic results,” Bernoulli 6, 457–489 (2000).MathSciNetCrossRefGoogle Scholar
  29. 29.
    G. Wei and M. A. Tanner, “A Monte Carlo implementation of the EM algorithm and the poor man’s data augmentation algorithms,” J. Am. Statist. Assoc. 85 (411), 699–704 (1990).CrossRefGoogle Scholar
  30. 30.
    J. R. Fienup, “Phase retrieval algorithms: a comparison,” Appl. Opt. 21, 2758 (1982).CrossRefGoogle Scholar
  31. 31.
    S. Marchesini, “A unified evaluation of iterative projection algorithms for phase retrieval,” Rev. Sci. Instrum. 78, 011301 (2007).CrossRefGoogle Scholar
  32. 32.
    G. Behrmann et al., “A distributed storage system with dCache,” J. Phys.: Conf. Ser. 119, 062014 (2008).Google Scholar
  33. 33.
    F. Maia et al., “Hawk: the image reconstruction package for coherent X-ray,” J. Appl. Crystallogr. 43, 1535–1539 (2010).CrossRefGoogle Scholar

Copyright information

© Pleiades Publishing, Ltd. 2018

Authors and Affiliations

  • S. A. Bobkov
    • 1
    Email author
  • A. B. Teslyuk
    • 1
    • 2
  • S. I. Zolotarev
    • 1
  • M. Rose
    • 3
  • K. A. Ikonnikova
    • 1
  • V. E. Velikhov
    • 1
  • I. A. Vartanyants
    • 3
    • 4
  • V. A. Ilyin
    • 1
    • 2
  1. 1.National Research Centre “Kurchatov Institute,”MoscowRussia
  2. 2.Moscow Institute of Physics and Technology (State University)Dolgoprudny, Moscow oblastRussia
  3. 3.Deutsches Elektronen-Synchrotron DESYHamburgGermany
  4. 4.National Research Nuclear University MEPhIMoscowRussia

Personalised recommendations