Skip to main content
SpringerLink
Log in

VolPAM: Volumetric Phenotype-Activation-Map for data-driven discovery of 3D imaging phenotypes and interpretability

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Knowledge about the subtypes of a disease critically affects clinical decisions ranging from the choice of therapeutic options to patient management. If the understanding of a disease is partial and the subtypes of the disease are not yet known, a traditional supervised approach becomes untenable for disease subtype classification. In these contexts, unsupervised methods for subtype discovery are essential. There has been very little prior work on the discovery of phenotypic subtypes based on imaging. Moreover, within that limited body of work, the discovered phenotypes often lack interpretability. In this paper, we present a data-driven approach to discovering interpretable imaging phenotypes in 3D image volumes. In particular, the phenotypes are discovered in a latent space learned through a 3D autoencoder. To interpret the discovered phenotypes, we learn a convolutional neural network to classify the phenotype label and present VolPAM (Volumetric Phenotype-Activation-Map) to interpret the latent phenotypes in terms of their imaging footprints. We present results and visualizations on datasets of Computed Tomography images as an example 3D imaging modality. The proposed methods can become a useful aid to further the understanding of the condition in question through phenotype discovery and interpretability in terms of distinct aspects of the discovered phenotypes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data availability and Code

Both datasets used to produce the results in this manuscript are publicly available at the following URLs: LIDC Data https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=1966254 COVID-19 CT Data https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/6ACUZJ Code Project code is available at the following GitHub repository: https://github.com/MahboobehNorouzi95/VolPAM.git.

Notes

  1. Visualizations of \(VolPAM_{3D}\) map volumes are available as videos online at: https://tinyurl.com/VolPAM3DVisualizations.

References

  1. Rios Velazquez E, Parmar C, Liu Y, Coroller TP, Cruz G, Stringfield O, Ye Z, Makrigiorgos M, Fennessy F, Mak RH, Gillies R, Quackenbush J, Aerts HJWL (2017) Somatic mutations drive distinct imaging phenotypes in Lung cancer. Cancer Res 77(14):3922–3930. https://doi.org/10.1158/0008-5472.CAN-17-0122

    Article  Google Scholar 

  2. Cho N (2016) Molecular subtypes and imaging phenotypes of breast cancer. Ultrasonography 35(4):281–288. https://doi.org/10.14366/usg.16030

    Article  Google Scholar 

  3. Savitz JB, Drevets WC (2009) Imaging phenotypes of major depressive disorder: genetic correlates. Neuroscience 164(1):300–330. https://doi.org/10.1016/j.neuroscience.2009.03.082

    Article  Google Scholar 

  4. Vogel JW, Young AL, Oxtoby NP, Smith R, Ossenkoppele R, Aksman LM, Strandberg O, La Joie R, Grothe M, Medina YI, Rabinovici GD, Alexander DC, Evans AC, Hansson O (2020) Spatiotemporal imaging phenotypes of tau pathology in Alzheimer’s disease. Alzheimer’s Dement 16(S4):045612. https://doi.org/10.1002/alz.045612

    Article  Google Scholar 

  5. Zhou M, Leung A, Echegaray S, Gentles A, Shrager JB, Jensen KC, Berry GJ, Plevritis SK, Rubin DL, Napel SOG (2018) Non-small cell lung cancer radiogenomics map identifies relationships between molecular and imaging phenotypes with prognostic implications. Radiology 286(1):307–315. https://doi.org/10.1148/radiol.2017161845

    Article  Google Scholar 

  6. Ashraf A, Daye D, Gavenonis S, Mies C, Feldman M, Rosen M, Kontos D (2014) Identification of intrinsic imaging phenotypes for breast cancer tumors: preliminary associations with gene expression profiles. Radiology 272(2):374–384. https://doi.org/10.1148/radiol.14131375

    Article  Google Scholar 

  7. Lundervold AS, Lundervold A (2019) An overview of deep learning in medical imaging focusing on MRI. Z Med Phys 29(2):102–127. https://doi.org/10.1016/j.zemedi.2018.11.002

    Article  Google Scholar 

  8. Masci J, Meier U, Cireşan D, Schmidhuber J (2011) Stacked convolutional auto-encoders for hierarchical feature extraction. In: Honkela T, Duch W, Girolami M, Kaski S (eds) Artificial neural networks and machine learning—ICANN 2011. Springer, Berlin, Heidelberg, pp 52–59

    Chapter  Google Scholar 

  9. Hosseini-Asl E, Keynton R, El-Baz A (2016) Alzheimer’s disease diagnostics by adaptation of 3d convolutional network. In: 2016 IEEE international conference on image processing (ICIP), pp 126–130. https://doi.org/10.1109/ICIP.2016.7532332

  10. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2017) Grad-cam: visual explanations from deep networks via gradient-based localization. In: 2017 IEEE international conference on computer vision (ICCV), pp 618–626. https://doi.org/10.1109/ICCV.2017.74

  11. Zhou SK, Greenspan H, Davatzikos C, Duncan JS, Van Ginneken B, Madabhushi A, Prince JL, Rueckert D, Summers RM (2021) A review of deep learning in medical imaging: imaging traits, technology trends, case studies with progress highlights, and future promises. Proc IEEE 109(5):820–838. https://doi.org/10.1109/JPROC.2021.3054390

    Article  Google Scholar 

  12. Aggarwal R, Sounderajah V, Martin G, Ting DSW, Karthikesalingam A, King D, Ashrafian H, Darzi A (2021) Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis. NPJ Digit Med 4(1):65. https://doi.org/10.1038/s41746-021-00438-z

    Article  Google Scholar 

  13. Ashraf AB, Daye D, Gavenonis S, Mies C, Feldman M, Rosen M, Kontos D (2014) Identification of intrinsic imaging phenotypes for breast cancer tumors: preliminary associations with gene expression profiles. Radiology 272(2):374

    Article  Google Scholar 

  14. Sommer C, Hoefler R, Samwer M, Gerlich DW (2017) A deep learning and novelty detection framework for rapid phenotyping in high-content screening. Mol Biol Cell 28(23):3428–3436. https://doi.org/10.1091/mbc.e17-05-0333

    Article  Google Scholar 

  15. Dürr O, Murina E, Siegismund D, Tolkachev V, Steigele S, Sick B (2018) Know when you don’t know: a robust deep learning approach in the presence of unknown phenotypes. Assay Drug Dev Technol 16(6):343–349. https://doi.org/10.1089/adt.2018.859

    Article  Google Scholar 

  16. Kontos D, Winham SJ, Oustimov A, Pantalone L, Hsieh M-K, Gastounioti A, Whaley DH, Hruska CB, Kerlikowske K, Brandt K, Conant EF, Vachon CM (2019) Radiomic phenotypes of mammographic parenchymal complexity: toward augmenting breast density in breast cancer risk assessment. Radiology 290(1):41–49. https://doi.org/10.1148/radiol.2018180179

    Article  Google Scholar 

  17. Mei S, Ji J, Geng Y, Zhang Z, Li X, Du Q (2019) Unsupervised spatial-spectral feature learning by 3d convolutional autoencoder for hyperspectral classification. IEEE Trans Geosci Remote Sens 57(9):6808–6820. https://doi.org/10.1109/TGRS.2019.2908756

    Article  Google Scholar 

  18. Armato SG III, McLennan G, Bidaut L, McNitt-Gray MF, Meyer CR, Reeves AP, Zhao B, Aberle DR, Henschke CI, Hoffman EA (2011) The lung image database consortium (LIDC) and image database resource initiative (IDRI): a completed reference database of lung nodules on ct scans. Med Phys 38(2):915–931

    Article  Google Scholar 

  19. Shakouri S, Bakhshali MA, Layegh P, Kiani B, Masoumi F, Ataei Nakhaei S, Mostafavi SM (2021) Covid19-ct-dataset: an open-access chest ct image repository of 1000+ patients with confirmed covid-19 diagnosis. BMC Res Notes 14(1):1–3

    Article  Google Scholar 

  20. Hinton GE, Zemel R (1994) Autoencoders, minimum description length and Helmholtz free energy. In: Advances in neural information processing systems, vol 6.

  21. Japkowicz N, Hanson SJ, Gluck MA (2000) Nonlinear autoassociation is not equivalent to PCA. Neural Comput 12(3):531–545

    Article  Google Scholar 

  22. Vincent P, Larochelle H, Bengio Y, Manzagol P-A. Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th international conference on machine learning, pp 1096–1103

  23. Hinton GE, Krizhevsky A, Wang SD (2011) Transforming auto-encoders. In: International conference on artificial neural networks. Springer, pp 44–51

  24. Kingma DP, Welling M (2013) Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114

  25. Zhang L, Qi G-J, Wang L, Luo J (2019) Aet vs. aed: unsupervised representation learning by auto-encoding transformations rather than data. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 2547–2555

  26. Bourlard H, Kabil SH (2022) Autoencoders reloaded. Biol Cybern 116(4):389–406

    Article  Google Scholar 

  27. Chen Y-C, Huang S-F, Lee H-Y, Wang Y-H, Shen C-H (2019) Audio word2vec: sequence-to-sequence autoencoding for unsupervised learning of audio segmentation and representation. IEEE/ACM Trans Audio, Speech, Lang Process 27(9):1481–1493. https://doi.org/10.1109/TASLP.2019.2922832

    Article  Google Scholar 

  28. Larici AR, Farchione A, Franchi P, Ciliberto M, Cicchetti G, Calandriello L, del Ciello A, Bonomo L (2017) Lung nodules: size still matters. Eur Respir Rev. https://doi.org/10.1183/16000617.0025-2017

  29. Sasirekha K, Baby P (2013) Agglomerative hierarchical clustering algorithm-A. Int J Sci Res Publ 83(3):83

    Google Scholar 

  30. Yang M (1993) A survey of hierarchical clustering. Mathl. Comput. Model. 18(11):1–16

    Article  Google Scholar 

  31. Xu D, Tian Y (2015) A comprehensive survey of clustering algorithms. Ann. Data Sci. 2(2):165–193

    Article  MathSciNet  Google Scholar 

  32. Khan SS, Ahmad A (2004) Cluster center initialization algorithm for k-means clustering. Pattern Recogn Lett 25(11):1293–1302

    Article  Google Scholar 

  33. Mittal H, Pandey AC, Saraswat M, Kumar S, Pal R, Modwel G (2022) A comprehensive survey of image segmentation: clustering methods, performance parameters, and benchmark datasets. Multimedia Tools Appl 81:35001–35026

    Article  Google Scholar 

  34. Jain AK, Dubes RC (1988) Algorithms for clustering data. Prentice-Hall, Inc., Upper Saddle River. http://portal.acm.org/citation.cfm?id=46712

  35. Syakur M, Khotimah B, Rochman E, Satoto BD (2018) Integration k-means clustering method and elbow method for identification of the best customer profile cluster. In: IOP conference series: materials science and engineering, vol 336. IOP Publishing, p 012017

  36. Thorndike RL (1953) Who belongs in the family. Psychometrika 18(4):267–276

    Article  Google Scholar 

  37. Yamashita R, Nishio M, Do RKG, Togashi K (2018) Convolutional neural networks: an overview and application in radiology. Insights Imaging 9(4):611–629

    Article  Google Scholar 

  38. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2017) Grad-cam: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision, pp 618–626

  39. Çiçek Ö, Abdulkadir A, Lienkamp SS, Brox T, Ronneberger O (2016) 3d u-net: learning dense volumetric segmentation from sparse annotation. In: Ourselin S, Joskowicz L, Sabuncu MR, Unal G, Wells W (eds) Medical image computing and computer-assisted intervention—MICCAI 2016. Springer, pp 424–432

    Google Scholar 

  40. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  41. Chung J, Gulcehre C, Cho K, Bengio Y (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling. In: NIPS 2014 workshop on deep learning

  42. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Lu, Polosukhin I (2017) Attention is all you need. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, Garnett R (eds) Advances in neural information processing systems, vol 30. Curran Associates Inc

    Google Scholar 

  43. Gu Y, Tinn R, Cheng H, Lucas M, Usuyama N, Liu X, Naumann T, Gao J, Poon H (2021) Domain-specific language model pretraining for biomedical natural language processing. ACM Trans Comput Healthc 3(1):1–23

    Article  Google Scholar 

  44. Zhang Z, Zhang A, Li M, Zhao H, Karypis G, Smola A (2023) Multimodal chain-of-thought reasoning in language models

  45. Brown T, Mann B, Ryder N, Subbiah M, Kaplan JD, Dhariwal P, Neelakantan A, Shyam P, Sastry G, Askell A, Agarwal S, Herbert-Voss A, Krueger G, Henighan T, Child R, Ramesh A, Ziegler D, Wu J, Winter C, Hesse C, Chen M, Sigler E, Litwin M, Gray S, Chess B, Clark J, Berner C, McCandlish S, Radford A, Sutskever I, Amodei D (2020) Language models are few-shot learners. In: Larochelle H, Ranzato M, Hadsell R, Balcan MF, Lin H (eds) Advances in neural information processing systems, vol 33. Curran Associates, Inc., pp 1877–1901

Download references

Funding

Financial support from Natural Sciences and Engineering Research Council of Canada (NSERC) is acknowledged.

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, University of Manitoba, Winnipeg, MB, Canada

    Mahboobeh Norouzi, Shehroz S. Khan & Ahmed Ashraf

  2. KITE, University Health Network, Toronto, ON, Canada

    Shehroz S. Khan

  3. Institute of Biomedical Engineering, University of Toronto, Toronto, ON, Canada

    Shehroz S. Khan

Authors
  1. Mahboobeh Norouzi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shehroz S. Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ahmed Ashraf
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

MN, SK, and AA contributed to conceptualization, methodology, formal analysis, and reviewing and editing. MN was responsible for software and algorithm implementation, original draft preparation, and writing. All authors have read and agreed to the submitted version of the manuscript.

Corresponding author

Correspondence to Ahmed Ashraf.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Norouzi, M., Khan, S.S. & Ashraf, A. VolPAM: Volumetric Phenotype-Activation-Map for data-driven discovery of 3D imaging phenotypes and interpretability. Neural Comput & Applic 36, 2961–2972 (2024). https://doi.org/10.1007/s00521-023-09172-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-09172-x

Keywords

Search

Navigation