Skip to main content

Imaging Biomarker Knowledge Transfer for Attention-Based Diagnosis of COVID-19 in Lung Ultrasound Videos

  • Conference paper
  • First Online:
Simplifying Medical Ultrasound (ASMUS 2021)

Abstract

The use of lung ultrasound imaging has recently emerged as a quick, cost-effective, and safe method for diagnosis of patients with COVID-19. Challenges with training deep networks to identify COVID-19 signatures in lung ultrasound data are that large datasets do not yet exist; disease signatures are sparse, but are spatially and temporally correlated; and signatures may appear sporadically in ultrasound video sequences. We propose an attention-based video model that is specifically designed to detect these disease signatures, and leverage a knowledge transfer approach to overcome existing limitations in data availability. In our design, a convolutional neural network extracts spatially encoded features, which are fed to a transformer encoder to capture temporal information across the frames and focus on the most important frames. We guide the network to learn clinically relevant features by training it on a pulmonary biomarker detection task, and then transferring the model’s knowledge learned from this problem to achieve 80% precision and 87% recall for COVID-19. Our results outperform the state-of-the-art model on a public lung ultrasound dataset. We perform ablation studies to highlight the efficacy of our design over previous state-of-the-art frame-based approaches. To demonstrate that our approach learns clinically relevant imaging biomarkers, we introduce a novel method for generating attention-based video classification explanations called Biomarker Attention-scaled Class Activation Mapping (Bio-AttCAM). Our analysis of the activation map shows high correlation with the key frames selected by clinicians.

T. Lum and M. Mahdavi contributed equally to this work.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 54.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 69.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Abrams, E.R., Rose, G., Fields, J.M., Esener, D.: Point-of-care ultrasound in the evaluation of COVID-19. J. Emerg. Med. 2020, 403–408 (2020)

    Article  Google Scholar 

  2. Amann, J., Blasimme, A., Vayena, E., Frey, D., Madai, V.: Explainability for artificial intelligence in healthcare: a multidisciplinary perspective. BMC Med. Inform. Decision Making 20, 310 (2020)

    Google Scholar 

  3. Bach, S., et al.: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE 10, 1–46 (2015)

    Google Scholar 

  4. Born, J., et al.: Accelerating detection of lung pathologies with explainable ultrasound image analysis. Appl. Sci. 11(2), 672 (2021)

    Article  Google Scholar 

  5. Brattain, L.J., Telfer, B.A., Dhyani, M., Grajo, J.R., Samir, A.E.: Machine learning for medical ultrasound: status, methods, and future opportunities. Abdom Radiol. (NY) 2018, 786–799 (2018)

    Article  Google Scholar 

  6. Buda, N., Segura-Grau, E., Cylwik, J., Wełnickid, M.: Lung ultrasound in the diagnosis of COVID-19 infection - a case series and review of the literature. Adv. Med. Sci. 2020, 378–385 (2020)

    Article  Google Scholar 

  7. Dong, E., Du, H., Gardner, L.: An interactive web-based dashboard to track Covid-19 in real time. Lancet Infect. Dis. 20(5), 533–534 (2020)

    Article  Google Scholar 

  8. Hiley, L., Preece, A., Hicks, Y.: Explainable deep learning for video recognition tasks: a framework & recommendations. arXiv preprint arXiv:1909.05667 (2019)

  9. Li, J., Qiu, H.: Comparing attention-based neural architectures for video captioning (2019)

    Google Scholar 

  10. Liu, T., Shao, Q.: BERT for large-scale video segment classification with test-time augmentation. arXiv preprint arXiv:1912.01127 (2019)

  11. Peng, Q., et al.: Findings of lung ultrasonography of novel corona virus pneumonia during the 2019–2020 epidemic. Intensive Care Med. 46, 849–850 (2020)

    Google Scholar 

  12. Roy, S., et al.: Deep learning for classification and localization of COVID-19 markers in point-of-care lung ultrasound. IEEE Trans. Med. Imag. 39(8), 2676–2687 (2020)

    Article  Google Scholar 

  13. Selvaraju, R.R., et al.: Grad-CAM: visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626 (2017)

    Google Scholar 

Download references

Acknowledgement

This work was supported by Canada’s Digital Technology Supercluster, the Natural Sciences and Engineering Research Council of Canada (NSERC), and the Canadian Institutes of Health Research (CIHR).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Tyler Lum .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Lum, T. et al. (2021). Imaging Biomarker Knowledge Transfer for Attention-Based Diagnosis of COVID-19 in Lung Ultrasound Videos. In: Noble, J.A., Aylward, S., Grimwood, A., Min, Z., Lee, SL., Hu, Y. (eds) Simplifying Medical Ultrasound. ASMUS 2021. Lecture Notes in Computer Science(), vol 12967. Springer, Cham. https://doi.org/10.1007/978-3-030-87583-1_16

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-87583-1_16

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-87582-4

  • Online ISBN: 978-3-030-87583-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics