Skip to main content

Frame Rate Up-Conversion in Echocardiography Using a Conditioned Variational Autoencoder and Generative Adversarial Model

  • Conference paper
  • First Online:
Medical Image Computing and Computer Assisted Intervention – MICCAI 2019 (MICCAI 2019)


Accurate detection of heart-related diseases in echocardiography (echo) often requires determining the performance of cardiac valves or contractile events such as strain at a high temporal resolution. In high-end cart-based imaging systems, this is achieved by increasing the frame rate using specialized beamforming and imaging hardware, or by limiting the imaging field of view (FOV). In point-of-care imaging, such a high frame rate imaging technology is currently unavailable. In this paper, we propose a new frame rate up-conversion technique, as a post-processing step during or after the echo acquisition. The proposed technique takes advantage of both variational autoencoders (VAE) and generative adversarial networks (GAN), and produces realistic frames at a high frame rate that can be used to augment conventional imaging. The proposed technique is robust to variations in heart rate since its latent space not only uses immediate previous frames, but it also takes into account the appearance of end-diastolic and end-systolic frames in its estimation. Our results show that the proposed technique can increase the frame rate by at least 5 times without any requirement for limiting the imaging FOV.

F. T. Dezaki and H. Girgis—Joint first authors.

P. Abolmaesumi and T. Tsang—Joint senior authors.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions


  1. Cikes, M., Tong, L., et al.: Ultrafast cardiac ultrasound imaging: technical principles, applications, and clinical benefits. JACC Cardiovasc. Imaging 7(8), 812–823 (2014)

    Article  Google Scholar 

  2. Gifani, P., Behnam, H., et al.: Temporal super resolution enhancement of echocardiographic images based on sparse representation. IEEE Trans. Ultrason. Ferroelectr. Freq. Control 63(1), 6–19 (2016)

    Article  Google Scholar 

  3. Provost, J., Lee, W.N., et al.: Electromechanical wave imaging of normal and ischemic hearts in vivo. IEEE Trans. Med. Imaging 29(3), 625–635 (2010)

    Article  Google Scholar 

  4. Papadacci, C., Pernot, M., et al.: High-contrast ultrafast imaging of the heart. IEEE Trans. Ultrason. Ferroelectr. Freq. Control. 61(2), 288–301 (2014)

    Article  Google Scholar 

  5. Tong, L., Ramalli, A., et al.: Multi-transmit beam forming for fast cardiac imaging–experimental validation and in vivo application. IEEE Trans. Med. Imaging 33(6), 1205–1219 (2014)

    Article  Google Scholar 

  6. Contijoch, F., Fernandez-de Manuel, L., et al.: Increasing temporal resolution of 3D transesophageal ultrasound by rigid body registration of sequential, temporally offset sequences. In: 2010 IEEE International Symposium on Biomedical Imaging: From Nano to Macro, pp. 328–331. IEEE (2010)

    Google Scholar 

  7. Perrin, D.P., Vasilyev, N.V., et al.: Temporal enhancement of 3D echocardiography by frame reordering. JACC Cardiovasc. Imaging 5(3), 300–304 (2012)

    Article  Google Scholar 

  8. Srivastava, N., Mansimov, E., Salakhudinov, R.: Unsupervised learning of video representations using LSTMS. In: International Conference on Machine Learning, pp. 843–852 (2015)

    Google Scholar 

  9. Finn, C., Goodfellow, I., Levine, S.: Unsupervised learning for physical interaction through video prediction. In: NIPS, pp. 64–72 (2016)

    Google Scholar 

  10. Babaeizadeh, M., Finn, C., et al.: Stochastic variational video prediction. arXiv preprint. arXiv:1710.11252 (2017)

  11. Lee, A.X., Zhang, R., Ebert, F., Abbeel, P., Finn, C., Levine, S.: Stochastic adversarial video prediction. arXiv preprint. arXiv:1804.01523 (2018)

  12. Xingjian, S., Chen, Z., et al.: Convolutional LSTM network: a machine learning approach for precipitation nowcasting. In: NIPS, pp. 802–810 (2015)

    Google Scholar 

  13. Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint. arXiv:1312.6114 (2013)

  14. Zhu, J.Y., Zhang, R., et al.: Toward multimodal image-to-image translation. In: NIPS, pp. 465–476 (2017)

    Google Scholar 

  15. Ulyanov, D., Vedaldi, A., Lempitsky, V.: Instance normalization: the missing ingredient for fast stylization. arXiv preprint. arXiv:1607.08022 (2016)

  16. Nair, V., Hinton, G.E.: Rectified linear units improve restricted Boltzmann machines. In: ICML, pp. 807–814 (2010)

    Google Scholar 

  17. Zhang, R., Isola, P., et al.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR, pp. 586–595 (2018)

    Google Scholar 

Download references


This work is supported in part by the Canadian Institutes of Health Research (CIHR) and in part by the Natural Sciences and Engineering Research Council of Canada (NSERC). The authors would like to acknowledge the support provided by Dale Hawley and the Vancouver Coastal Health in providing us with the anonymized, deidentified data.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Fatemeh Taheri Dezaki .

Editor information

Editors and Affiliations

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (zip 313 KB)

Supplementary material 2 (pdf 826 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Taheri Dezaki, F., Girgis, H., Rohling, R., Gin, K., Abolmaesumi, P., Tsang, T. (2019). Frame Rate Up-Conversion in Echocardiography Using a Conditioned Variational Autoencoder and Generative Adversarial Model. In: Shen, D., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2019. MICCAI 2019. Lecture Notes in Computer Science(), vol 11765. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-32244-1

  • Online ISBN: 978-3-030-32245-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics