Climbing the Data Mountain: Processing of SFX Data

  • Chun Hong Yoon
  • Thomas A. White


Serial femtosecond crystallography experiments produce mountains of data that require FEL facilities to provide many petabytes of storage space and large compute clusters for timely processing of user data. The route to reach the summit of the data mountain requires peak finding, indexing, integration, refinement, and phasing. Those who reach the summit get a crystal clear view of the “radiation damage-free” structure of a protein that is most consistent with the observed measurements. Data processing plays a critical role in the ability to measure accurate structure factor intensities from individual diffraction snapshots and combine them in three-dimensional space. Current developments in SFX aim to take into account the huge complexity of SFX experiments, modeling variations in the beam and crystals, uncertainties in geometry, partiality, mosaicity, and figures of merit that are unique to SFX.



TAW acknowledges the Helmholtz Association via Programme-Oriented Funds. Portions of this research were carried out at the Linac Coherent Light Source (LCLS) at the SLAC National Accelerator Laboratory. LCLS is an Office of Science User Facility operated for the US Department of Energy Office of Science by Stanford University. Use of the Linac Coherent Light Source (LCLS), SLAC National Accelerator Laboratory, is supported by the US Department of Energy, Office of Science, Office of Basic Energy Sciences under Contract No. DE-AC02-76SF00515.


  1. 1.
    Allahgholi, A., Becker, J., Bianco, L., Delfs, A., Dinapoli, R., Goettlicher, P., et al. (2015). AGIPD, a high dynamic range fast detector for the European XFEL. Journal of Instrumentation, 10, C01023.CrossRefGoogle Scholar
  2. 2.
    Bajt, S., Chapman, H. N., Spiller, E. A., Alameda, J. B., Woods, B. W., Frank, M., et al. (2008). Camera for coherent diffractive imaging and holography with a soft-x-ray free-electron laser. Applied Optics, 47, 1673–1683.CrossRefGoogle Scholar
  3. 3.
    Barends, T. R. M., Foucar, L., Ardevol, A., Nass, K., Aquila, A., Botha, S., et al. (2015). Direct observation of ultrafast collective motions in CO myoglobin upon ligand dissociation. Science, 350, 445–450.CrossRefGoogle Scholar
  4. 4.
    Barends, T. R. M., Foucar. L., Botha, S., Doak, R. B., Shoeman, R. L., Nass, K., (2014). De novo protein crystal structure determination from X-ray free-electron laser data. Nature, 505, 244–247.CrossRefGoogle Scholar
  5. 5.
    Barends, T. R. M., Foucar, L., Shoeman, R. L., Bari, S., Epp, S. W., Hartmann, R., et al. (2013). Anomalous signal from S atoms in protein crystallographic data from an X-ray free-electron laser. Acta Crystallographica D, 69, 838–842.CrossRefGoogle Scholar
  6. 6.
    Barty, A., Boutet, S., Bogan, M. J., Hau-Riege. S., Marchesini, S., Sokolowski-Tinten, K., et al. (2008). Ultrafast single-shot diffraction imaging of nanoscale dynamics. Nature Photonics, 2, 415–419. CrossRefGoogle Scholar
  7. 7.
    Barty, A., Kirian, R. A., Maia, F. R. N. C., Hantke, M., Yoon, C. H., White, T. A., et al. (2014). Cheetah: Software for high-throughput reduction and analysis of serial femtosecond X-ray diffraction data. Journal of Applied Crystallography, 47(3), 1118–1131.CrossRefGoogle Scholar
  8. 8.
    Batyuk, A., Galli, L., Ishchenko, A., Han, G. W., Gati, C., Popov, P. A., et al. (2016). Native phasing of x-ray free-electron laser data for a G protein-coupled receptor. Science Advances, 2, e1600292.CrossRefGoogle Scholar
  9. 9.
    Beyerlein, K., White, T. A., Yefanov, O., Gati, C., Kazantsev, I. G., Fog-Gade, N., et al. (2017). FELIX: An algorithm for indexing multiple crystallites in X-ray free-electron laser snapshot diffraction images. Journal of Applied Crystallography, 50, 1075–1083.CrossRefGoogle Scholar
  10. 10.
    Blaj, G., Caragiulo, P., Carini, G., Carron, S., Dragone, A., Freytag, D., et al. (2015). X-ray detectors at the Linac Coherent Light Source. Journal of Synchrotron Radiation, 22(3), 577–583. CrossRefGoogle Scholar
  11. 11.
    Boutet, S., Foucar, L., Barends, T. R. M., Botha, S., Doak, R. B., Koglin, J. E., et al. (2015). Characterization and use of the spent beam for serial operation of LCLS. Journal of Synchrotron Radiation, 22, 634–643. CrossRefGoogle Scholar
  12. 12.
    Brehm, W., & Diederichs, K. (2014). Breaking the indexing ambiguity in serial crystallography. Acta Crystallographica Section D, 70, 101–109.CrossRefGoogle Scholar
  13. 13.
    Carini, G. A., Boutet, S., Chollet, M., Dragone, A., Haller, G., Hart, P. A., et al. (2014). Experience with the CSPAD during dedicated detector runs at LCLS. Journal of Physics Conference Series, 493, 012011.CrossRefGoogle Scholar
  14. 14.
    Casanas, A., Warshamanage, R., Finke, A. D., Panepucci, E., Olieric, V., Nöll, A., et al. (2016). EIGER detector: Application in macromolecular crystallography. Acta Crystallographica. Section D, Structural Biology, 72(9), 1036–1048. CrossRefGoogle Scholar
  15. 15.
    Chapman, H. N., Barty, A., Bogan, M. J., Boutet, S., Frank, M., Hau-Riege, S. P., et al. (2006). Femtosecond diffractive imaging with a soft-X-ray free-electron laser. Nature Physics, 2, 839. CrossRefGoogle Scholar
  16. 16.
    Chapman, H. N., Fromme, P., Barty, A., White, T. A., Kirian, R. A., Aquila, A., et al. (2011). Femtosecond x-ray protein nanocrystallography. Nature, 470, 73–77.CrossRefGoogle Scholar
  17. 17.
    Conrad, C. E., Basu, S., James, D., Wang, D., Schaffer, A., Roy-Chowdhury, S., et al. (2015). A novel inert crystal delivery medium for serial femtosecond crystallography. IUCrJ, 2, 421–430.CrossRefGoogle Scholar
  18. 18.
    Damiani, D., Dubrovin, M., Gaponenko, I., Kroeger, W., Lane, T. J., Mitra, A., et al. (2016). Linac Coherent Light Source data analysis using psana. Journal of Applied Crystallography, 49, 672–679.CrossRefGoogle Scholar
  19. 19.
    Dauter, Z. (2006). Estimation of anomalous signal in diffraction data. Acta Crystallographica Section D, 62, 867–876.CrossRefGoogle Scholar
  20. 20.
    Duisenberg, A. J. M. (1992). Indexing in single-crystal diffractometry with an obstinate list of reflections. Journal of Applied Crystallography, 25, 92–96.CrossRefGoogle Scholar
  21. 21.
    Foucar, L. (2016). CFEL-ASG Software Suite (CASS): usage for free-electron laser experiments with biological focus. Journal of Applied Crystallography, 49(4), 1336–1346.CrossRefGoogle Scholar
  22. 22.
    Galli, L., Son, S. K., Barends, T. R. M., White, T. A., Barty, A., Botha, S., et al. (2015). Towards phasing using high X-ray intensity. IUCrJ, 2, 627–634.CrossRefGoogle Scholar
  23. 23.
    Galli, L., Son, S. K., Klinge, M., Bajt, S., Barty, A., Bean, R., et al. (2015). Electronic damage in S atoms in a native protein crystal induced by an intense X-ray free-electron laser pulse. Structural Dynamics, 2, 041703.CrossRefGoogle Scholar
  24. 24.
    Gildea, R. J., Waterman, D. G., Parkhurst, J. M., Axford, D., Sutton, G., Stuart, D. I., et al. (2014). New methods for indexing multi-lattice diffraction data. Acta Crystallographica Section D, 70, 2652–2666.CrossRefGoogle Scholar
  25. 25.
    Ginn, H. M., Brewster, A. S., Hattne, J., Evans, G., Wagner, A., Grimes, J. M., et al. (2015). A revised partiality model and post-refinement algorithm for X-ray free-electron laser data. Acta Crystallographica Section D, 71, 1400–1410.CrossRefGoogle Scholar
  26. 26.
    Ginn, H. M., Evans, G., Sauter, N. K., & Stuart, D. I. (2016). On the release of cppxfel for processing X-ray free-electron laser images. Journal of Applied Crystallography, 49, 1065–1072.CrossRefGoogle Scholar
  27. 27.
    Ginn, H. M., Messerschmidt, M., Ji, X., Zhang, H., Axford, D., Gildea, R. J., (2015) Structure of CPV17 polyhedrin determined by the improved analysis of serial femtosecond crystallographic data. Nature Communications 6, 6435.CrossRefGoogle Scholar
  28. 28.
    Ginn, H. M., Roedig, P., Kuo, A., Evans, G., Sauter, N. K., Ernst, O., et al. (2016). TakeTwo: An indexing algorithm suited to still images with known crystal parameters. Acta Crystallographica Section D, 72, 956–965.CrossRefGoogle Scholar
  29. 29.
    Hattne, J., Echols, N., Tran, R., Kern, J., Gildea, R. J., Brewster, A. S., et al. (2014). Accurate macromolecular structures using minimal measurements from X-ray free-electron lasers. Nature Methods, 11, 545–548.CrossRefGoogle Scholar
  30. 30.
    Heisen, B. C., Boukhelef, D., Esenov, S., Hauf, S., Kozlova, I., Maia, L., et al. (2013). Karabo: An integrated software framework combining control, data management, and scientific computing tasks. In Proceedings of ICALEPCS, San Francisco.Google Scholar
  31. 31.
    Hunter, M. S., Segelke, B., Messerschmidt, M., Williams, G. J., Zatsepin, N. A., Barty, A., et al. (2014). Fixed-target protein serial microcrystallography with an x-ray free electron laser. Scientific Reports, 4, 6026. CrossRefGoogle Scholar
  32. 32.
    Hunter, M. S., Yoon, C. H., DeMirci, H., Sierra, R. G., Dao, E. H., Ahmadi, R., et al. (2016). Selenium single-wavelength anomalous diffraction de novo phasing using an X-ray-free electron laser. Nature Communications, 7, 13388.CrossRefGoogle Scholar
  33. 33.
    Hutchison, C. D. M., Cordon-Preciado, V., Morgan, R. M. L., Nakane, T., Ferreira, J., Dorlhiac, G., et al. (2017). X-ray free electron laser determination of crystal structures of dark and light states of a reversibly photoswitching fluorescent protein at room temperature. International Journal of Molecular Sciences, 18 (1918). CrossRefGoogle Scholar
  34. 34.
    Kabsch, W. (1988). Evaluation of single-crystal x-ray diffraction data from a position-sensitive detector. Journal of Applied Crystallography, 21, 916–924.CrossRefGoogle Scholar
  35. 35.
    Kabsch, W. (2014). Processing of X-ray snapshots from crystals in random orientations. Acta Crystallographica Section D, 70, 2204–2216.CrossRefGoogle Scholar
  36. 36.
    Karplus, P. A., & Diederichs, K. (2012). Linking crystallographic model and data quality. Science, 336, 1030–1033.CrossRefGoogle Scholar
  37. 37.
    Kirian, R. A., Wang, X., Weierstall, U., Schmidt, K. E., Spence, J. C. H., et al. (2010). Femtosecond x-ray protein nanocrystallography — data analysis methods. Optics Express, 18, 5713–5723.CrossRefGoogle Scholar
  38. 38.
    Könnecke, M., Akeroyd, F. A., Bernstein, H. J., Brewster, A. S., Campbell, S. I., Clausen, B., et al. (2015). The NeXus data format. Journal of Applied Crystallography 48, 301–305. CrossRefGoogle Scholar
  39. 39.
    Kraft, P., Bergamaschi, A., Broennimann, C., Dinapoli, R., Eikenberry, E. F., Henrich, B., et al. (2009). Performance of single-photon-counting PILATUS detector modules. Journal of Synchrotron Radiation 16(3), 368–375. CrossRefGoogle Scholar
  40. 40.
    Kroon-Batenburg, L. M. J., Schreurs, A. M. M., Ravelli, R. B. G., & Gros, P. (2015). Accounting for partiality in serial crystallography using ray-tracing principles. Acta Crystallographica Section D, 71, 1799–1811.CrossRefGoogle Scholar
  41. 41.
    Lyubimov, A. Y., Uervirojnangkoorn, M., Zeldin, O. B., Brewster, A. S., Murray, T. D., Sauter, N. K., et al. (2016). IOTA: Integration optimization, triage and analysis tool for the processing of XFEL diffraction images. Journal of Applied Crystallography, 49, 1057–1064.CrossRefGoogle Scholar
  42. 42.
    Maia, F. R. N. C. (2012). The coherent x-ray imaging data bank. Nature Methods, 9(9), 854–855. CrossRefGoogle Scholar
  43. 43.
    Mancuso, A. P., Aquila, A., Borchers, G., Giewekemeyer, K., & Reimers, N. (2013). Technical design report: scientific instrument single particles, clusters, and biomolecules (SPB). Google Scholar
  44. 44.
    Mariani, V., Morgan, A., Yoon, C. H., Lane, T. J., White, T. A., O’Grady, C. P., et al. (2016) OnDA: Online data analysis and feedback for serial X-ray imaging. Journal of Applied Crystallography, 49(3), 1073–1080.CrossRefGoogle Scholar
  45. 45.
    Mozzanica, A., Bergamaschi, A., Cartier, S., Dinapoli, R., Greiffenberg, D., Johnson, I., et al. (2014) Prototype characterization of the JUNGFRAU pixel detector for SwissFEL. Journal of Instrumentation, 9, C05010.CrossRefGoogle Scholar
  46. 46.
    Nakane, T., Joti, Y., Tono, K., Yabashi, M., Nango, E., Iwata, S., et al. (2016). Data processing pipeline for serial femtosecond crystallography at SACLA. Journal of Applied Crystallography, 49, 1035–1041.CrossRefGoogle Scholar
  47. 47.
    Nakane, T., Song, C., Suzuki, M., Nango, E., Kobayashi, J., Masuda, T., et al. (2015). Native sulfur/chlorine SAD phasing for serial femtosecond crystallography. Acta Crystallographica Section D, 71, 2519–2525.CrossRefGoogle Scholar
  48. 48.
    Nass, K., Meinhart, A., Barends, T. R. M., Fourcar, L., Gorel, A., Aquila, A., et al. (2016). Protein structure determination by single-wavelength anomalous diffraction phasing of X-ray free-electron laser data. IUCrJ, 3, 180–191.CrossRefGoogle Scholar
  49. 49.
    Pande, K., Hutchison, C. D. M., Groenhof, G., Aquila, A., Robinson, J. S., Tenboer, J., et al. (2016). Femtosecond structural dynamics drives the trans/cis isomerization in photoactive yellow protein. Science, 352, 725–729.CrossRefGoogle Scholar
  50. 50.
    Pixel array detectors. Accessed 20.11.2017.
  51. 51.
    Powell, H. R. (1999). The Rossmann Fourier autoindexing algorithm in MOSFLM. Acta Crystallographica Section D, 55(10), 1690–1695. CrossRefGoogle Scholar
  52. 52.
    Rossmann, M. G. (1979). Processing oscillation diffraction data for very large unit cells with an automatic convolution technique and profile fitting. Journal of Applied Crystallography, 12, 225–238.CrossRefGoogle Scholar
  53. 53.
    Rossmann, M. G., Leslie, A. G. W., Abdel-Meguid, S. S., & Tsukihara, T. (1979). Processing and post-refinement of oscillation camera data. Journal of Applied Crystallography, 12, 570–581.CrossRefGoogle Scholar
  54. 54.
    Sauter, N. K. (2015). XFEL diffraction: Developing processing methods to optimize data quality. Journal of Synchrotron Radiation, 22, 239–248.CrossRefGoogle Scholar
  55. 55.
    Sauter, N. K., Hattne, J., Brewster, A. S., Echols, N., Zwart, P. H., & Adams, P. D. (2014). Improved crystal orientation and physical properties from single-shot XFEL stills. Acta Crystallographica Section D, 70, 3299–3309.CrossRefGoogle Scholar
  56. 56.
    Schreurs, A. M. M., Xian, X., & Kroon-Batenburg, L. M. J. (2010). EVAL15: A diffraction data integration method based on ab initio predicted profiles. Journal of Applied Crystallography, 43, 70–82.CrossRefGoogle Scholar
  57. 57.
    Thayer, J., Damiani, D., Ford, C., Dubrovin, M., Gaponenko, I., O’Grady, C. P., et al. (2017). Data systems for the Linac coherent light source. Advances in Structural Chemical Imaging, 3(1), 3. CrossRefGoogle Scholar
  58. 58.
    Uervirojnangkoorn, M., Zeldin, O. B., Lyubimov, A. Y., Hattne, J., Brewster, A. S., Sauter, N. K., et al. (2015). Enabling X-ray free electron laser crystallography for challenging biological systems from a limited number of crystals. eLife, 4, e05421.CrossRefGoogle Scholar
  59. 59.
    White, T. A. (2014). Post-refinement method for snapshot serial crystallography. Philosophical Transactions of the Royal Society B 369, 20130330.CrossRefGoogle Scholar
  60. 60.
    White, T. A., Barty, A., Stellato, F., Holton, J. M., Kirian, R. A., Zatsepin, N. A., et al. (2013). Crystallographic data processing for free-electron laser sources. Acta Crystallographica D, 69, 1231–1240.CrossRefGoogle Scholar
  61. 61.
    White, T. A., Mariani, V., Brehm, W., Yefanov, O., Barty, A., Beyerlein, K. R., et al. (2016). Recent developments in CrystFEL. Journal of Applied Crystallography, 49, 680–689.CrossRefGoogle Scholar
  62. 62.
    Yamashita, K., Kuwabara, N., Nakane, T., Murai, T., Mizohata, E., Sugahara, M., et al. (2017). Experimental phase determination with selenome-thionine or mercury-derivatization in serial femtosecond crystallography. IUCrJ, 4, 639–647.CrossRefGoogle Scholar
  63. 63.
    Yamashita, K., Pan, D., Okuda, T., Sugahara, M., Kodan, A., Yamaguchi, T., et al. (2015). An isomorphous replacement method for efficient de novo phasing for serial femtosecond crystallography. Scientific Reports, 5, 14017.CrossRefGoogle Scholar
  64. 64.
    Yefanov, O., Mariani, V., Gati, C., White, T. A., Chapman, H. N., Barty, A. (2015). Accurate determination of segmented X-ray detector geometry. Optics Express, 23, 28459.CrossRefGoogle Scholar
  65. 65.
    Yoon, C. H., DeMirci, H., Sierra, R. G., Dao, H. E., Ahmadi, R., Aksit, F., et al. (2017). Se-SAD serial femtosecond crystallography datasets from selenobiotinyl-streptavidin. Scientific Data, 4, 170055. CrossRefGoogle Scholar
  66. 66.
    Zeldin, O. B., Brewster, A. S., Hattne, J., Uervirojnangkoorn, M., Lyubimov, A. Y., Zhou, Q., et al. (2015). Data exploration toolkit for serial diffraction experiments. Acta Crystallographica Section D, 71, 352–356.CrossRefGoogle Scholar
  67. 67.
    Zhu, D., Feng, Y., Stoupin, S., Terentyev, S. A., Lemke, H. T., Fritz, D. M., et al. (2014). Performance of a beam-multiplexing diamond crystal monochromator at the Linac Coherent Light Source. Review of Scientific Instruments 85(6), 063106.CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Linac Coherent Light Source, SLAC National Accelerator LaboratoryMenlo ParkUSA
  2. 2.Center for Free-Electron Laser ScienceGerman Electron Synchrotron DESYHamburgGermany

Personalised recommendations