Skip to main content
SpringerLink
Log in

Performance Evaluation of Pseudospectral Ultrasound Simulations on a Cluster of Xeon Phi Accelerators

  • Conference paper
  • First Online:
High Performance Computing in Science and Engineering (HPCSE 2019)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 12456))

  • 319 Accesses

Abstract

The rapid development of novel procedures in medical ultrasonics, including treatment planning in therapeutic ultrasound and image reconstruction in photoacoustic tomography, leads to increasing demand for large-scale ultrasound simulations. However, routine execution of such simulations using traditional methods, e.g., finite difference time domain, is expensive and often considered intractable due to the computational and memory requirements. The k-space corrected pseudospectral time domain method used by the k-Wave toolbox allows for significant reductions in spatial and temporal grid resolution. These improvements are achieved at the cost of all-to-all communication, which are inherent to the multi-dimensional fast Fourier transforms. To improve data locality, reduce communication and allow efficient use of accelerators, we recently implemented a domain decomposition technique based on a local Fourier basis.

In this paper, we investigate whether it is feasible to run the distributed k-Wave implementation on the Salomon cluster equipped with 864 Intel Xeon Phi (Knight’s Corner) accelerators. The results show the immaturity of the KNC platform with issues ranging from limited support of Infiniband and LustreFS in Intel MPI on this platform to poor performance of 3D FFTs achieved by Intel MKL on the KNC architecture. Yet, we show that it is possible to achieve strong and weak scaling comparable to CPU-only platforms albeit with the runtime \(1.8\times \) to \(4.3\times \) longer. However, the accounting policy for Salomon’s accelerators is far more favorable and thus their employment reduces the computational cost significantly.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://docs.it4i.cz/salomon/hardware-overview/.

  2. 2.

    http://mvapich.cse.ohio-state.edu/benchmarks/.

References

  1. Beard, P.: Biomedical photoacoustic imaging. Interf. Focus 1(4), 602–631 (2011)

    Article  Google Scholar 

  2. Boyd, J.P.: A comparison of numerical algorithms for Fourier extension of the first, second, and third kinds. J. Comput. Phys. 178(1), 118–160 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  3. Boyd, J.P.: Asymptotic Fourier coefficients for a C\(\infty \) bell (Smoothed-“Top-Hat”) & the Fourier extension problem. J. Sci. Comput. 29(1), 1–24 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  4. Coloma, K., et al.: A new flexible MPI collective I/O implementation. In: 2006 IEEE International Conference on Cluster Computing, pp. 1–10. IEEE (2006)

    Google Scholar 

  5. Dubinsky, T.J., Cuevas, C., Dighe, M.K., Kolokythas, O., Joo, H.H.: High-intensity focused ultrasound: current potential and oncologic applications. Am. J. Roentgenol. 190(1), 191–199 (2008)

    Article  Google Scholar 

  6. Folk, M., Heber, G., Koziol, Q., Pourmal, E., Robinson, D.: An overview of the HDF5 technology suite and its applications. In: Proceedings of the EDBT/ICDT 2011 Workshop on Array Databases - AD 2011 (2011)

    Google Scholar 

  7. Frigo, M., Johnson, S.G.: The design and implementation of FFTW3. Proc. IEEE 93(2), 216–231 (2005)

    Article  Google Scholar 

  8. Gholami, A., Hill, J., Malhotra, D., Biros, G.: AccFFT: a library for distributed-memory FFT on CPU and GPU architectures (2016)

    Google Scholar 

  9. Gu, J., Jing, Y.: Modeling of wave propagation for medical ultrasound: a review. IEEE Trans. Ultrason. Ferroelectr. Freq. Control 62(11), 1979–1992 (2015)

    Article  MathSciNet  Google Scholar 

  10. Howison, M., Koziol, Q., Knaak, D., Mainzer, J., Shalf, J.: Tuning HDF5 for Lustre file systems. In: Proceedings of the Workshop on Interfaces and Abstractions for Scientific Data Storage 5, IASDS 2010 (2012)

    Google Scholar 

  11. Intel Corporation: Math Kernel Library 11.3 Developer Reference. Intel Corporation (2015)

    Google Scholar 

  12. Israeli, M., Vozovoi, L., Averbuch, A.: Spectral multidomain technique with local Fourier basis. J. Sci. Comput. 8(2), 135–149 (1993)

    Article  MathSciNet  MATH  Google Scholar 

  13. Jaros, J., Rendell, A.P., Treeby, B.E.: Full-wave nonlinear ultrasound simulation on distributed clusters with applications in high-intensity focused ultrasound. Int. J. High Perform. Comput. Appl. 30(2), 137–155 (2016)

    Article  Google Scholar 

  14. Jaros, J., Vaverka, F., Treeby, B.E.: Spectral domain decomposition using local Fourier basis: application to ultrasound simulation on a cluster of GPUs. Supercomput. Front. Innov. 3(3), 40–55 (2016)

    Google Scholar 

  15. Jeffers, J., Reinders, J.: Intel Xeon Phi Coprocessor High Performance Programming. Elsevier Inc., Waltham (2013). No. 1

    Google Scholar 

  16. Klepárník, P., Bařina, D., Zemčík, P., Jaroš, J.: Efficient low-resource compression of HIFU data. Information 9(7), 1–14 (2018). https://doi.org/10.3390/info9070155. https://www.fit.vut.cz/research/publication/11764

    Article  Google Scholar 

  17. Mast, T., Souriau, L., Liu, D.L., Tabei, M., Nachman, A., Waag, R.: A k-space method for large-scale models of wave propagation in tissue. IEEE Trans. Ultrason. Ferroelectr. Freq. Control 48(2), 341–354 (2001)

    Article  Google Scholar 

  18. Meairs, S., Alonso, A.: Ultrasound, microbubbles and the blood-brain barrier. Prog. Biophys. Mol. Biol. 93(1–3), 354–362 (2007)

    Article  Google Scholar 

  19. Nandapalan, N., Jaros, J., Treeby, B.E., Rendell, A.P.: Implementation of 3D FFTs across multiple GPUs in shared memory environments. In: Proceedings of the Thirteenth International Conference on Parallel and Distributed Computing, Applications and Technologies, pp. 167–172 (2012)

    Google Scholar 

  20. Nikl, V., Jaros, J.: Parallelisation of the 3D fast Fourier transform using the hybrid OpenMP/MPI decomposition. In: Hliněný, P., et al. (eds.) MEMICS 2014. LNCS, vol. 8934, pp. 100–112. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-14896-0_9

    Chapter  Google Scholar 

  21. Pekurovsky, D.: P3DFFT: a framework for parallel computations of Fourier transforms in three dimensions (2012)

    Google Scholar 

  22. Pinton, G.F., Dahl, J., Rosenzweig, S., Trahey, G.E.: A heterogeneous nonlinear attenuating full-wave model of ultrasound. IEEE Trans. Ultrason. Ferroelectr. Freq. Control 56(3), 474–488 (2009)

    Article  Google Scholar 

  23. Pippig, M.: PFFT-an extension of FFTW to massively parallel architectures. SIAM J. Sci. Comput. 35(3), 213–236 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  24. Sorensen, H., Jones, D., Heideman, M., Burrus, C.: Real-valued fast Fourier transform algorithms. IEEE Trans. Acoust. Speech Signal Process. 35(6), 849–863 (1987)

    Article  Google Scholar 

  25. Tabei, M., Mast, T.D., Waag, R.C.: A k-space method for coupled first-order acoustic propagation equations. J. Acoust. Soc. Am. 111(1 Pt 1), 53–63 (2002)

    Article  Google Scholar 

  26. Tomov, S., Haidar, A., Ayala, A., Schultz, D., Dongarra, J.: FFT-ECP fast Fourier transform, 01 2019 (2019)

    Google Scholar 

  27. Treeby, B.E., Jaros, J., Rendell, A.P., Cox, B.T.: Modeling nonlinear ultrasound propagation in heterogeneous media with power law absorption using a k-space pseudospectral method. J. Acoust. Soc. Am. 131(6), 4324–4336 (2012)

    Article  Google Scholar 

  28. Tufail, Y., Yoshihiro, A., Pati, S., Li, M.M., Tyler, W.J.: Ultrasonic neuromodulation by brain stimulation with transcranial ultrasound. Nat. Protoc. 6(9), 1453–1470 (2011)

    Article  Google Scholar 

  29. Vaverka, F., Treeby, B.E., Jaros, J.: Evaluation of the suitability of Intel Xeon Phi clusters for the simulation of ultrasound wave propagation using pseudospectral methods. In: Rodrigues, J.M.F., et al. (eds.) ICCS 2019. LNCS, vol. 11538, pp. 577–590. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22744-9_45

    Chapter  Google Scholar 

  30. Wang, E., et al.: High-Performance Computing on the Intel® Xeon Phi™. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-06486-4

    Book  Google Scholar 

  31. Yu, W., Mittra, R., Su, T., Liu, Y., Yang, X.: Parallel Finite-Difference Time-Domain Method. Artech House, Inc., Norwood (2006)

    MATH  Google Scholar 

Download references

Acknowledgement

This work was supported by The Ministry of Education, Youth and Sports from the National Programme of Sustainability (NPU II) project “IT4Innovations excellence in science - LQ1602” and by the IT4Innovations infrastructure which is supported from the Large Infrastructures for Research, Experimental Development and Innovations project “IT4Innovations National Supercomputing Center - LM2015070”. This project has received funding from the European Union’s Horizon 2020 research and innovation programme H2020 ICT 2016–2017 under grant agreement No. 732411 and is an initiative of the Photonics Public Private Partnership. This work was further supported in part by the Engineering and Physical Sciences Research Council (EPRSC), UK, grant numbers EP/L020262/1 and EP/S026371/1.

Author information

Authors and Affiliations

  1. Faculty of Information Technology, Centre of Excellence IT4Innovations, Brno University of Technology, Bozetechova 2, 612 00, Brno, Czech Republic

    Filip Vaverka & Jiri Jaros

  2. Medical Physics and Biomedical Engineering, University College London, London, WC1E 6BT, UK

    Bradley E. Treeby

Authors
  1. Filip Vaverka
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bradley E. Treeby
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiri Jaros
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Filip Vaverka .

Editor information

Editors and Affiliations

  1. VSB - Technical University of Ostrava, Ostrava-Poruba, Czech Republic

    Tomáš Kozubek

  2. ETH Zurich, Zurich, Switzerland

    Peter Arbenz

  3. Brno University of Technology, Brno, Czech Republic

    Jiří Jaroš

  4. Technical University of Ostrava, Ostrava-Poruba, Czech Republic

    Lubomír Říha

  5. Institute of Mathematics of the CAS, Prague, Czech Republic

    Jakub Šístek

  6. Charles University in Prague, Prague, Czech Republic

    Petr Tichý

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Vaverka, F., Treeby, B.E., Jaros, J. (2021). Performance Evaluation of Pseudospectral Ultrasound Simulations on a Cluster of Xeon Phi Accelerators. In: Kozubek, T., Arbenz, P., Jaroš, J., Říha, L., Šístek, J., Tichý, P. (eds) High Performance Computing in Science and Engineering. HPCSE 2019. Lecture Notes in Computer Science(), vol 12456. Springer, Cham. https://doi.org/10.1007/978-3-030-67077-1_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-67077-1_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-67076-4

  • Online ISBN: 978-3-030-67077-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation