Subject2Vec: Generative-Discriminative Approach from a Set of Image Patches to a Vector

  • Sumedha Singla
  • Mingming Gong
  • Siamak Ravanbakhsh
  • Frank Sciurba
  • Barnabas Poczos
  • Kayhan N. BatmanghelichEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11070)


We propose an attention-based method that aggregates local image features to a subject-level representation for predicting disease severity. In contrast to classical deep learning that requires a fixed dimensional input, our method operates on a set of image patches; hence it can accommodate variable length input image without image resizing. The model learns a clinically interpretable subject-level representation that is reflective of the disease severity. Our model consists of three mutually dependent modules which regulate each other: (1) a discriminative network that learns a fixed-length representation from local features and maps them to disease severity; (2) an attention mechanism that provides interpretability by focusing on the areas of the anatomy that contribute the most to the prediction task; and (3) a generative network that encourages the diversity of the local latent features. The generative term ensures that the attention weights are non-degenerate while maintaining the relevance of the local regions to the disease severity. We train our model end-to-end in the context of a large-scale lung CT study of Chronic Obstructive Pulmonary Disease (COPD). Our model gives state-of-the art performance in predicting clinical measures of severity for COPD. The distribution of the attention provides the regional relevance of lung tissue to the clinical measurements.



This work is partially supported by NIH Award Number 1R01HL141813-01. We gratefully thank NVIDIA Corporation for their donation of the Titan X Pascal GPU. We thank Competitive Medical Research Fund (CMRF) grant for their funding.


  1. 1.
    Cheplygina, V., Peña, I.P., Pedersen, J.H., Lynch, D.A., Sørensen, L., de Bruijne, M.: Transfer learning for multi-center classification of chronic obstructive pulmonary disease, January 2017Google Scholar
  2. 2.
    Clevert, D.A., Unterthiner, T., Hochreiter, S.: Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs), November 2015Google Scholar
  3. 3.
    Dubost, F., et al.: GP-Unet: lesion detection from weak labels with a 3D regression network. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10435, pp. 214–221. Springer, Cham (2017). Scholar
  4. 4.
    Estépar, R.S.J., Kinney, G.L.: Computed tomographic measures of pulmonary vascular morphology in smokers and their clinical implications. AJRCCM 188(2), 231–239 (2013)Google Scholar
  5. 5.
    González, G., Ash, S.Y., Vegas-Sánchez-Ferrero, G.: Disease staging and prognosis in smokers using deep learning in chest computed tomography. AJRCCM 197(2), 193–203 (2017)Google Scholar
  6. 6.
    Hayhurst, M.D., MacNee, W., Flenley, D.C.: Diagnosis of pulmonary emphysema by computerised tomography. Lancet 2(8398), 320–322 (1984)CrossRefGoogle Scholar
  7. 7.
    Kingma, D.P., Ba, J.: Adam: A Method for Stochastic Optimization, December 2014Google Scholar
  8. 8.
    LeCun, Y., Cortes, C.: MNIST handwritten digit database. AT&T Labs (2010)Google Scholar
  9. 9.
    Luong, M.T., Pham, H., Manning, C.D.: Effective Approaches to Attention-based Neural Machine Translation, August 2015Google Scholar
  10. 10.
    van der Maaten, L., Hinton, G., Visualizing Data using t-SNE: Visualizing Data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008)zbMATHGoogle Scholar
  11. 11.
    Masci, J., Meier, U., Cireşan, D., Schmidhuber, J.: Stacked convolutional auto-encoders for hierarchical feature extraction. In: Honkela, T., Duch, W., Girolami, M., Kaski, S. (eds.) ICANN 2011. LNCS, vol. 6791, pp. 52–59. Springer, Heidelberg (2011). Scholar
  12. 12.
    Müller, N.L., Staples, C.A., Miller, R.R., Abboud, R.T.: Density mask: an objective method to quantitate emphysema using computed tomography. Chest 94, 782–787 (1988)CrossRefGoogle Scholar
  13. 13.
    Regan, E.A., et al.: Genetic epidemiology of COPD (COPDGene) study design. J. COPD 7(1), 32–43 (2010)CrossRefGoogle Scholar
  14. 14.
    San Jose Estepar, R., Ross, J.C., Harmouche, R., Onieva, J., Diaz, A.A., Washko, G.R.: CIP: an open-source library and workstation for quantitative chest imaging. Am. J. Respir. Crit. Care Med. 191, A4975 (2015)Google Scholar
  15. 15.
    Satoh, K., Kobayashi, T., Murota, M.: CT assessment of subtypes in pulmonary emphysema in smokers. JJCR 46(1), 98–102 (2001)Google Scholar
  16. 16.
    Schabdach, J., Wells, W.M., Cho, M., Batmanghelich, K.N.: A likelihood-free approach for characterizing heterogeneous diseases in large-scale studies. In: Niethammer, M., et al. (eds.) IPMI 2017. LNCS, vol. 10265, pp. 170–183. Springer, Cham (2017). Scholar
  17. 17.
    Shapiro, S.D.: Evolving concepts in the pathogenesis of chronic obstructive pulmonary disease. Clin. Chest Med. 21(4), 621–632 (2000)CrossRefGoogle Scholar
  18. 18.
    Sorensen, L., Nielsen, M., Lo, P., Ashraf, H., Pedersen, J.H., de Bruijne, M.: Texture-based analysis of COPD: a data-driven approach. IEEE Trans. Med. Imaging 31(1), 70–78 (2012)CrossRefGoogle Scholar
  19. 19.
    Xu, K., et al.: Show, attend and tell: neural image caption generation with visual attention. In: International Conference on Machine Learning, February 2015Google Scholar
  20. 20.
    Yang, J., et al.: Unsupervised discovery of spatially-informed lung texture patterns for pulmonary emphysema: the MESA COPD study. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10433, pp. 116–124. Springer, Cham (2017). Scholar
  21. 21.
    Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R., Smola, A.: Deep sets. In: Advances in Neural Information Processing Systems, pp. 3391–3401, March 2017Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Sumedha Singla
    • 1
  • Mingming Gong
    • 2
  • Siamak Ravanbakhsh
    • 3
  • Frank Sciurba
    • 4
  • Barnabas Poczos
    • 5
  • Kayhan N. Batmanghelich
    • 1
    • 2
    • 5
    Email author
  1. 1.Computer Science DepartmentUniversity of PittsburghPittsburghUSA
  2. 2.Department of Biomedical InformaticsUniversity of PittsburghPittsburghUSA
  3. 3.Computer Science DepartmentUniversity of British ColumbiaVancouverCanada
  4. 4.University of Pittsburgh School of MedicineUniversity of PittsburghPittsburghUSA
  5. 5.Machine Learning DepartmentCarnegie Mellon UniversityPittsburghUSA

Personalised recommendations