Rapid Tomographic Image Reconstruction via Large-Scale Parallelization

  • Tekin BicerEmail author
  • Doga Gursoy
  • Rajkumar Kettimuthu
  • Francesco De Carlo
  • Gagan Agrawal
  • Ian T. Foster
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9233)


Synchrotron (x-ray) light sources permit investigation of the structure of matter at extremely small length and time scales. Advances in detector technologies enable increasingly complex experiments and more rapid data acquisition. However, analysis of the resulting data then becomes a bottleneck—preventing near-real-time error detection or experiment steering. We present here methods that leverage highly parallel computers to improve the performance of iterative tomographic image reconstruction applications. We apply these methods to the conventional per-slice parallelization approach and use them to implement a novel in-slice approach that can use many more processors. To address programmability, we implement the introduced methods in high-performance MapReduce-like computing middleware, which is further optimized for reconstruction operations. Experiments with four reconstruction algorithms and two large datasets show that our methods can scale up to 8 K cores on an IBM BG/Q supercomputer with almost perfect speedup and can reduce total reconstruction times for large datasets by more than 95.4 % on 32 K cores relative to 1 K cores. Moreover, the average reconstruction times are improved from \(\sim \)2 h (256 cores) to \(\sim \)1 min (32 K cores), thus enabling near-real-time use.


Execution Time Reconstruction Algorithm Reconstruction Time Race Condition Data Chunk 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This work was supported by the Office of Advanced Scientific Computing Research, Office of Science, U.S. Department of Energy, under the contract DE-AC02-06CH11357 and the RAMSES project under the Next Generation Networking for Science Program.


  1. 1.
    Agulleiro, J., Fernandez, J.-J.: Fast tomographic reconstruction on multicore computers. Bioinformatics 27(4), 582–583 (2011)CrossRefGoogle Scholar
  2. 2.
    Apache software foundation. Apache Hadoop (2014). [Online Accessed January 2015]
  3. 3.
    Beister, M., Kolditz, D., Kalender, W.A.: Iterative reconstruction methods in X-ray CT. Physica Medica 28(2), 94–108 (2012)CrossRefGoogle Scholar
  4. 4.
    Bicer, T.: Supporting data-intensive scientific computing on bandwidth and space constrained environments. Ph.D. thesis, The Ohio State University (2014)Google Scholar
  5. 5.
    Chou, C.-Y., Chuo, Y.-Y., Hung, Y., Wang, W.: A fast forward projection using multithreads for multirays on GPUs in medical image reconstruction. Med. Phys. 38(7), 4052–4065 (2011)CrossRefGoogle Scholar
  6. 6.
    De Carlo, F., Gürsoy, D., Marone, F., Rivers, M., Parkinson, D.Y., Khan, F., Schwarz, N., Vine, D.J., Vogt, S., Gleber, S.-C., Narayanan, S., Newville, M., Lanzirotti, T., Sun, Y., Hong, Y.P., Jacobsen, C.: Scientific data exchange: a schema for HDF5-based storage of raw and analyzed data. J. Synchrotron Radiat. 21(6), 1224–1230 (2014)CrossRefGoogle Scholar
  7. 7.
    Deslippe, J., Essiari, A., Patton, S.J., Samak, T., Tull, C.E., Hexemer, A., Kumar, D., Parkinson, D., Stewart, P.: Workflow management for real-time analysis of lightsource experiments. In: Proceedings of the 9th Workshop on Workflows in Support of Large-Scale Science, pp. 31–40. IEEE Press (2014)Google Scholar
  8. 8.
    Gürsoy, D., Biçer, T., Almer, J.D., Kettimuthu, R., Stock, S.R., De Carlo, F.: Maximum a posteriori estimation of crystallographic phases in X-ray diffraction tomography. Philos. Trans. Royal Soc. Lond. A: Math. Phys. Eng. Sci. 373(2043), 20140392 (2015)CrossRefGoogle Scholar
  9. 9.
    Gürsoy, D., Biçer, T., Lanzirotti, A., Newville, M.G., De Carlo, F.: Hyperspectral image reconstruction for X-ray fluorescence tomography. Opt. Express 23(7), 9014–9023 (2015)CrossRefGoogle Scholar
  10. 10.
    Gürsoy, D., De Carlo, F., Xiao, X., Jacobsen, C.: TomoPy: a framework for the analysis of synchrotron tomographic data. J. Synchrotron Radiat. 21(5), 1188–1193 (2014)CrossRefGoogle Scholar
  11. 11.
    Hsieh, J., Nett, B., Yu, Z., Sauer, K., Thibault, J.-B., Bouman, C.A.: Recent advances in CT image reconstruction. Curr. Radiol. Rep. 1(1), 39–51 (2013)CrossRefGoogle Scholar
  12. 12.
    Jang, B., Kaeli, D., Do, S., Pien, H.: Multi GPU implementation of iterative tomographic reconstruction algorithms. In: IEEE International Symposium on Biomedical Imaging: From Nano to Macro, ISBI 2009, pp. 185–188. IEEE (2009)Google Scholar
  13. 13.
    Jiang, W., Ravi, V.T., Agrawal, G.: A map-reduce system with an alternate API for multi-core environments. In: Proceedings of the 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing, CCGRID 2010, Washington, DC, USA, pp. 84–93. IEEE Computer Society (2010)Google Scholar
  14. 14.
    Jones, M., Yao, R., Bhole, C.: Hybrid MPI-OpenMP programming for parallel OSEM PET reconstruction. IEEE Trans. Nucl. Sci. 53(5), 2752–2758 (2006)CrossRefGoogle Scholar
  15. 15.
    Kanitpanyacharoen, W., Parkinson, D.Y., De Carlo, F., Marone, F., Stampanoni, M., Mokso, R., MacDowell, A., Wenk, H.-R.: A comparative study of X-ray tomographic microscopy on shales at different synchrotron facilities: ALS, APS and SLS. J. Synchrotron Radiat. 20(1), 172–180 (2013)CrossRefGoogle Scholar
  16. 16.
    Lee, D., Dinov, I., Dong, B., Gutman, B., Yanovsky, I., Toga, A.W.: CUDA optimization strategies for compute-and memory-bound neuroimaging algorithms. Comput. Methods Programs Biomed. 106(3), 175–187 (2012)CrossRefGoogle Scholar
  17. 17.
    Mohan, K., Venkatakrishnan, S., Gibbs, J., Gulsoy, E., Xiao, X., De Graef, M., Voorhees, P., Bouman, C.: TIMBIR: a method for time-space reconstruction from interlaced views. IEEE Trans. Comput. Imaging PP(99), 1 (2015)CrossRefGoogle Scholar
  18. 18.
    Phatak, C., Gürsoy, D.: Iterative reconstruction of magnetic induction using lorentz transmission electron tomography. Ultramicroscopy 150, 54–64 (2015)CrossRefGoogle Scholar
  19. 19.
    Pratx, G., Chinn, G., Olcott, P., Levin, C.: Fast, accurate and shift-varying line projections for iterative reconstruction using the GPU. IEEE Trans. Med. Imaging 28(3), 435–445 (2009)CrossRefGoogle Scholar
  20. 20.
    Sidky, E.Y., Kao, C.-M., Pan, X.: Accurate image reconstruction from few-views and limited-angle data in divergent-beam CT. J. X-ray Sci. Technol. 14(2), 119–139 (2006)Google Scholar
  21. 21.
    Stone, S.S., Haldar, J.P., Tsao, S.C., Hwu, W.-M., Sutton, B.P., Liang, Z.-P., et al.: Accelerating advanced MRI reconstructions on GPUs. J. Parallel Distrib. Comput. 68(10), 1307–1318 (2008)CrossRefGoogle Scholar
  22. 22.
    Thielemans, K., Tsoumpas, C., Mustafovic, S., Beisel, T., Aguiar, P., Dikaios, N., Jacobson, M.W.: Stir: software for tomographic image reconstruction release 2. Phys. Med. Biol. 57(4), 867 (2012)CrossRefGoogle Scholar
  23. 23.
    Treibig, J., Hager, G., Hofmann, H.G., Hornegger, J., Wellein, G.: Pushing the limits for medical image reconstruction on recent standard multicore processors. Int. J. High Perform. Comput. Appl. 27(2), 162–177 (2012)CrossRefGoogle Scholar
  24. 24.
    Wang, Y., De Carlo, F., Mancini, D.C., McNulty, I., Tieman, B., Bresnahan, J., Foster, I., Insley, J., Lane, P., von Laszewski, G., et al.: A high-throughput X-ray microtomography system at the advanced photon source. Rev. Sci. Instrum. 72(4), 2062–2068 (2001)CrossRefGoogle Scholar
  25. 25.
    Zaharia, M., Chowdhury, M., Franklin, M.J., Shenker, S., Stoica, I.: Spark: cluster computing with working sets. In: Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing, HotCloud 2010, Berkeley, CA, USA, p. 10. USENIX Association (2010)Google Scholar
  26. 26.
    Zeng, K., Bai, E., Wang, G.: A fast CT reconstruction scheme for a general multi-core PC. Int. J. Biomed. Imaging 2007, 1 (2007)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  • Tekin Bicer
    • 1
    Email author
  • Doga Gursoy
    • 2
  • Rajkumar Kettimuthu
    • 1
  • Francesco De Carlo
    • 2
  • Gagan Agrawal
    • 3
  • Ian T. Foster
    • 1
    • 4
  1. 1.Mathematics and Computer Science DivisionArgonne National LaboratoryLemontUSA
  2. 2.X-Ray Science DivisionArgonne National LaboratoryLemontUSA
  3. 3.Department of Computer Science and EngineeringOhio State UniversityColumbusUSA
  4. 4.Department of Computer ScienceUniversity of ChicagoChicagoUSA

Personalised recommendations