Skip to main content
SpringerLink
Log in

Cardiac point-of-care to cart-based ultrasound translation using constrained CycleGAN

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

 The emerging market of cardiac handheld ultrasound (US) is on the rise. Despite the advantages in ease of access and the lower cost, a gap in image quality can still be observed between the echocardiography (echo) data captured by point-of-care ultrasound (POCUS) compared to conventional cart-based US, which limits the further adaptation of POCUS. In this work, we aim to present a machine learning solution based on recent advances in adversarial training to investigate the feasibility of translating POCUS echo images to the quality level of high-end cart-based US systems.

Methods

 We propose a constrained cycle-consistent generative adversarial architecture for unpaired translation of cardiac POCUS to cart-based US data. We impose a structured shape-wise regularization via a critic segmentation network to preserve the underlying shape of the heart during quality translation. The proposed deep transfer model is constrained to the anatomy of the left ventricle (LV) in apical two-chamber (AP2) echo views.

Results

 A total of 1089 echo studies from 841 patients are used in this study. The AP2 frames are captured by POCUS (Philips Lumify and Clarius) and cart-based (Philips iE33 and Vivid E9) US machines. The dataset of quality translation comprises a total of 441 echo studies from 395 patients. Data from both POCUS and cart-based systems of the same patient were available in 122 cases. The deep-quality transfer model is integrated into a pipeline for an automated cardiac evaluation task, namely segmentation of LV in AP2 view. By transferring the low-quality POCUS data to the cart-based US, a significant average improvement of 30% and 34 mm is obtained in the LV segmentation Dice score and Hausdorff distance metrics, respectively.

Conclusion

 This paper presents the feasibility of a machine learning solution to transform the image quality of POCUS data to that of high-quality high-end cart-based systems. The experiments show that by leveraging the quality translation through the proposed constrained adversarial training, the accuracy of automatic segmentation with POCUS data could be improved.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Achim A (2001) Novel bayesian multiscale method for speckle removal in medical ultrasound images. IEEE TMI 20(8):772–783

    CAS  Google Scholar 

  2. Alsharqi M, Woodward W, Mumith J, Markham D, Upton R, Leeson P (2018) Artificial intelligence and echocardiography. Echo Res Pract 5(4):R115–R125

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  3. Armanious K, Jiang C, Fischer M, Küstner T, Nikolaou K, Gatidis S, Yang B (2018) Medgan: medical image translation using GANs. arXiv preprint arXiv:1806.06397

  4. Behnami D. Liao Z, Girgis H, Luong C, Rohling R, Gin K, Tsang T, Abolmaesumi P (2019) Dual-view joint estimation of left ventricular ejection fraction with uncertainty modelling in echocardiograms. In: International conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 696–704

    Google Scholar 

  5. Carneiro G, Nascimento JC, Freitas A (2012) The segmentation of the left ventricle of the heart from ultrasound data using deep learning architectures and derivative-based search methods. IEEE Trans Image Process 21(3):968–982

    Article  PubMed  Google Scholar 

  6. Chen H, Zhang Y, Zhang W, Liao P, Li K, Zhou J, Wang G (2017) Low-dose CT via convolutional neural network. Biomed Opt Express 8(2):679–694

    Article  PubMed  PubMed Central  Google Scholar 

  7. Chen H, Zheng Y, Park JH, Heng PA, Zhou SK (2016) Iterative multi-domain regularized deep learning for anatomical structure detection and segmentation from ultrasound images. In: International conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 487–495

    Chapter  Google Scholar 

  8. Cherian A, Sullivan A (2019) Sem-GAN: Semantically-consistent image-to-image translation. In: 2019 IEEE winter conference on applications of computer vision (WACV). IEEE, pp 1797–1806

  9. Coupé P (2009) Nonlocal means-based speckle filtering for ultrasound images. IEEE TIP 18(10):2221–2229

    Google Scholar 

  10. Degel MA, Navab N, Albarqouni S (2018) Domain and geometry agnostic CNNs for left atrium segmentation in 3D ultrasound. In: MICCAI, pp 630–637

    Chapter  Google Scholar 

  11. Dietrichson F, Smistad E, Ostvik A, Lovstakken L (2018) Ultrasound speckle reduction using generative adversial networks. In: 2018 IEEE international ultrasonics symposium (IUS). IEEE, pp 1–4

  12. Dong S, Luo G, Wang K, Cao S, Mercado A, Shmuilovich O, Zhang H, Li S (2018) Voxelatlasgan: 3D left ventricle segmentation on echocardiography with atlas guided generation and voxel-to-voxel discrimination. In: International conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 622–629

    Chapter  Google Scholar 

  13. Dykes JC, Kipps AK, Chen A, Nourse S, Rosenthal DN, Tierney ESS (2019) Parental acquisition of echocardiographic images in pediatric heart transplant patients using a handheld device: a pilot telehealth study. J Am Soc Echocardiogr 32(3):404–411

    Article  PubMed  Google Scholar 

  14. Esteva A, Robicquet A, Ramsundar B, Kuleshov V, DePristo M, Chou K, Cui C, Corrado G, Thrun S, Dean J (2019) A guide to deep learning in healthcare. Nat Med 25(1):24–29

    Article  CAS  PubMed  Google Scholar 

  15. Gaudet J, Waechter J, McLaughlin K, Ferland A, Godinez T, Bands C, Boucher P, Lockyer J (2016) Focused critical care echocardiography: development and evaluation of an image acquisition assessment tool. Crit Care Med 44(6):e329–e335

    Article  PubMed  Google Scholar 

  16. Goudarzi S, Asif A, Rivaz H (2019) Multi-focus ultrasound imaging using generative adversarial networks. In: 2019 IEEE 16th international symposium on biomedical imaging (ISBI 2019), pp 1118–1121

  17. Huang O, Long W, Bottenus N, Lerendegui M, Trahey GE, Farsiu S, Palmeri ML (2020) Mimicknet, mimicking clinical image post-processing under black-box constraints. IEEE Trans Med Imaging

  18. Huo Y, Xu Z, Bao S, Assad A, Abramson RG, Landman BA (2018) Adversarial synthesis learning enables segmentation without target modality ground truth. In: IEEE ISBI, pp 1217–1220

  19. Jafari MH, Girgis H, Abdi AH, Liao Z, Pesteie M, Rohling R, Gin K, Tsang T, Abolmaesumi P (2019) Semi-supervised learning for cardiac left ventricle segmentation using conditional deep generative models as prior. In: 2019 IEEE 16th international symposium on biomedical imaging (ISBI 2019). IEEE, pp 649–652

  20. Jafari MH, Girgis H, Liao Z, Behnami D, Abdi A, Vaseli H, Luong C, Rohling R, Gin K, Tsang T (2018) A unified framework integrating recurrent fully-convolutional networks and optical flow for segmentation of the left ventricle in echocardiography data. In: Deep learning in medical image analysis and multimodal learning for clinical decision support. Springer, Berlin, pp 29–37

    Chapter  Google Scholar 

  21. Jafari MH, Girgis H, Van Woudenberg N, Liao Z, Rohling R, Gin K, Abolmaesumi P, Tsang T (2019) Automatic biplane left ventricular ejection fraction estimation with mobile point-of-care ultrasound using multi-task learning and adversarial training. Int J Comput Assist Radiol Surg 14(6):1027–1037

    Article  PubMed  Google Scholar 

  22. Jafari MH, Liao Z, Girgis H, Pesteie M, Rohling R, Gin K, Tsang T, Abolmaesumi P (2019) Echocardiography segmentation by quality translation using anatomically constrained cyclegan. In: International conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 655–663

    Google Scholar 

  23. Johri AM, Durbin J, Newbigging J, Tanzola R, Chow R, De S, Tam J (2018) Cardiac point-of-care ultrasound: state-of-the-art in medical school education. J Am Soc Echocardiogr 31(7):749–760

    Article  PubMed  Google Scholar 

  24. Lang RM, Badano LP, Mor-Avi V, Afilalo J, Armstrong A, Ernande L, Flachskampf FA, Foster E, Goldstein SA, Kuznetsova T (2015) Recommendations for cardiac chamber quantification by echocardiography in adults: an update from the american society of echocardiography and the european association of cardiovascular imaging. Eur Heart J-Cardiovasc Imaging 16(3):233–271

    Article  PubMed  Google Scholar 

  25. Leclerc S, Smistad E, Pedrosa J, Østvik A, Cervenansky F, Espinosa F, Espeland T, Berg EAR, Jodoin P, Grenier T, Lartizien C, D’hooge J, Lovstakken L, Bernard O (2019) Deep learning for segmentation using an open large-scale dataset in 2D echocardiography. IEEE Trans Med Imaging 38(9):2198–2210

    Article  PubMed  Google Scholar 

  26. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444

    Article  CAS  PubMed  Google Scholar 

  27. Liao Z, Jafari MH, Girgis H, Gin K, Rohling R, Abolmaesumi P, Tsang T (2019) Echocardiography view classification using quality transfer star generative adversarial networks. In: International conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 687–695

    Google Scholar 

  28. Litjens G, Ciompi F, Wolterink JM, de Vos BD, Leiner T, Teuwen J, Išgum I (2019) State-of-the-art deep learning in cardiovascular image analysis. JACC: Cardiovasc Imaging 12(8):1549–1565

    Google Scholar 

  29. Liu S, Wang Y, Yang X, Lei B, Liu L, Li SX, Ni D, Wang T (2019) Deep learning in medical ultrasound analysis: a review. Engineering

  30. Lyu Q, You C, Shan H, Wang G (2018) Super-resolution MRI through deep learning. arXiv preprint arXiv:1810.06776

  31. Madani A, Ong JR, Tibrewal A, Mofrad MR (2018) Deep echocardiography: data-efficient supervised and semi-supervised deep learning towards automated diagnosis of cardiac disease. NPJ Digit Med 1(1):59

    Article  PubMed  PubMed Central  Google Scholar 

  32. McCormick TJ, Miller EC, Chen R, Naik VN (2018) Acquiring and maintaining point-of-care ultrasound (POCUS) competence for anesthesiologists. Can J Anesth/J Can d’anesthésie 65(4):427–436

    Article  Google Scholar 

  33. Noble JA, Boukerroui D (2006) Ultrasound image segmentation: a survey. IEEE Trans Med Imaging 25(8):987–1010. https://doi.org/10.1109/TMI.2006.877092

    Article  PubMed  Google Scholar 

  34. Oktay O, Ferrante E, Kamnitsas K, Heinrich M, Bai W, Caballero J, Cook SA, de Marvao A, Dawes T, O’Regan DP (2018) Anatomically constrained neural networks (ACNNs): application to cardiac image enhancement and segmentation. IEEE Trans Med Imaging 37(2):384–395

    Article  PubMed  Google Scholar 

  35. Østvik A, Smistad E, Aase SA, Haugen BO, Lovstakken L (2019) Real-time standard view classification in transthoracic echocardiography using convolutional neural networks. Ultrasound Med Biol 45(2):374–384

    Article  PubMed  Google Scholar 

  36. Perdios D, Vonlanthen M, Besson A, Martinez F, Arditi M, Thiran JP (2018) Deep convolutional neural network for ultrasound image enhancement. In: 2018 IEEE international ultrasonics symposium (IUS). IEEE, pp 1–4

  37. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 234–241

    Google Scholar 

  38. Rykkje A, Carlsen JF, Nielsen MB (2019) Hand-held ultrasound devices compared with high-end ultrasound systems: a systematic review. Diagnostics 9(2):61

    Article  PubMed Central  Google Scholar 

  39. Schmidhuber J (2015) Deep learning in neural networks: an overview. Neural Netw 61:85–117

    Article  PubMed  Google Scholar 

  40. Silva JF, Silva JM, Guerra A, Matos S, Costa C (2018) Ejection fraction classification in transthoracic echocardiography using a deep learning approach. In: 2018 IEEE 31st international symposium on computer-based medical systems (CBMS). IEEE, pp 123–128

  41. Smistad E, Østvik A (2017) 2D left ventricle segmentation using deep learning. In: 2017 IEEE international ultrasonics symposium (IUS), IEEE, pp 1–4

  42. Tsantis S (2014) Multiresolution edge detection using enhanced fuzzy c-means clustering for ultrasound image speckle reduction. Med Phys 41(7):72903

    Article  Google Scholar 

  43. Vedula S, Senouf O, Bronstein AM, Michailovich OV, Zibulevsky M (2017) Towards ct-quality ultrasound imaging using deep learning. arXiv preprint arXiv:1710.06304

  44. Veni G, Moradi M, Bulu H, Narayan G, Syeda-Mahmood T (2018) Echocardiography segmentation based on a shape-guided deformable model driven by a fully convolutional network prior. In: 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), pp 898–902

  45. Wejner-Mik P, Kasprzak JD, Filipiak-Strzecka D, Miśkowiec D, Lorens A, Lipiec P (2019) Personal mobile device-based pocket echocardiograph: the diagnostic value and clinical utility. Adv Med Sci 64(1):157–161

    Article  PubMed  Google Scholar 

  46. Wejner-Mik P, Teneta A, Jankowski M, Czyszpak L, Wdowiak-Okrojek K, Szymczyk E, Kasprzak JD, Lipiec P (2019) Feasibility and clinical utility of real-time tele-echocardiography using personal mobile device-based pocket echocardiograph. Arch Med Sci. https://doi.org/10.5114/aoms.2019.83136

  47. Wolterink JM (2019) Left ventricle segmentation in the era of deep learning. J Nucl Cardiol. https://doi.org/10.1007/s12350-019-01674-3

  48. Yang H, Sun J, Carass A, Zhao C, Lee J, Xu Z, Prince J (2018) Unpaired brain mr-to-ct synthesis using a structure-constrained cyclegan. In: Deep learning in medical image analysis and multimodal learning for clinical decision support. Springer, Berlin, pp 174–182

    Chapter  Google Scholar 

  49. Zhang J, Gajjala S, Agrawal P, Tison GH, Hallock LA, Beussink-Nelson L, Lassen MH, Fan E, Aras MA, Jordan C (2018) Fully automated echocardiogram interpretation in clinical practice: feasibility and diagnostic accuracy. Circulation 138(16):1623–1635

    Article  PubMed  PubMed Central  Google Scholar 

  50. Zhang Z, Yang L, Zheng Y (2018) Translating and segmenting multimodal medical volumes with cycle- and shape-consistency generative adversarial network. In: IEEE CVPR

  51. Zhu JY, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: IEEE CVPR, pp 2223–2232

Download references

Author information

Authors and Affiliations

  1. The University of British Columbia, Vancouver, Canada

    Mohammad H. Jafari, Hany Girgis, Nathan Van Woudenberg, Nathaniel Moulson, Christina Luong, Andrea Fung, Shane Balthazaar, John Jue, Micheal Tsang, Parvathy Nair, Ken Gin, Robert Rohling, Purang Abolmaesumi & Teresa Tsang

  2. Vancouver General Hospital, Vancouver, Canada

    Hany Girgis, Nathaniel Moulson, Christina Luong, Andrea Fung, Shane Balthazaar, John Jue, Micheal Tsang, Parvathy Nair, Ken Gin & Teresa Tsang

Authors
  1. Mohammad H. Jafari
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hany Girgis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nathan Van Woudenberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nathaniel Moulson
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Christina Luong
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Andrea Fung
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Shane Balthazaar
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. John Jue
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Micheal Tsang
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Parvathy Nair
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Ken Gin
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Robert Rohling
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Purang Abolmaesumi
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Teresa Tsang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mohammad H. Jafari.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Hany Girgis: Joint first author and Purang Abolmaesumi, Teresa Tsang: Joint senior authors.

This work is funded in part by the Natural Sciences and Engineering Research Council of Canada (NSERC) and in part by the Canadian Institutes of Health Research (CIHR)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jafari, M.H., Girgis, H., Van Woudenberg, N. et al. Cardiac point-of-care to cart-based ultrasound translation using constrained CycleGAN. Int J CARS 15, 877–886 (2020). https://doi.org/10.1007/s11548-020-02141-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-020-02141-y

Keywords

Search

Navigation