Multi-label Deep Regression and Unordered Pooling for Holistic Interstitial Lung Disease Pattern Detection

  • Mingchen GaoEmail author
  • Ziyue Xu
  • Le Lu
  • Adam P. Harrison
  • Ronald M. Summers
  • Daniel J. Mollura
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10019)


Holistically detecting interstitial lung disease (ILD) patterns from CT images is challenging yet clinically important. Unfortunately, most existing solutions rely on manually provided regions of interest, limiting their clinical usefulness. In addition, no work has yet focused on predicting more than one ILD from the same CT slice, despite the frequency of such occurrences. To address these limitations, we propose two variations of multi-label deep convolutional neural networks (CNNs). The first uses a deep CNN to detect the presence of multiple ILDs using a regression-based loss function. Our second variant further improves performance, using spatially invariant Fisher Vector encoding of the CNN feature activations. We test our algorithms on a dataset of 533 patients using five-fold cross-validation, achieving high area-under-curve (AUC) scores of 0.982, 0.972, 0.893 and 0.993 for Ground Glass, Reticular, Honeycomb and Emphysema, respectively. As such, our work represents an important step forward in providing clinically effective ILD detection.


Interstitial lung disease detection Convolutional neural network Multi-label deep regression Unordered pooling Fisher vector encoding 


  1. 1.
    Depeursinge, A., Vargas, A., Platon, A., Geissbuhler, A., Poletti, P.A., Müller, H.: Building a reference multimedia database for interstitial lung diseases. CMIG 36(3), 227–238 (2012)Google Scholar
  2. 2.
    Bartholmai, B., Karwoski, R., Zavaletta, V., Robb, R., Holmes, D.: The lung tissue research consortium: an extensive open database containing histological, clinical, and radiological data to study chronic lung disease (2006).
  3. 3.
    van Tulder, G., de Bruijne, M.: Combining generative and discriminative representation learning for lung CT analysis with convolutional restricted boltzmann machines. IEEE Trans. Med. Imag. 35(5), 1262–1272 (2016)CrossRefGoogle Scholar
  4. 4.
    Anthimopoulos, M., Christodoulidis, S., Ebner, L., Christe, A., Mougiakakou, S.: Lung pattern classification for interstitial lung diseases using a deep convolutional neural network. IEEE Trans. Med. Imag. 35(5), 1207–1216 (2016)CrossRefGoogle Scholar
  5. 5.
    Song, Y., Cai, W., Huang, H., Zhou, Y., Feng, D.D., Wang, Y., Fulham, M.J., Chen, M.: Large margin local estimate with applications to medical image classification. IEEE Trans. Med. Imag. 34(6), 1362–1377 (2015)CrossRefGoogle Scholar
  6. 6.
    Song, Y., Cai, W., Zhou, Y., Feng, D.D.: Feature-based image patch approximation for lung tissue classification. IEEE Trans. Med. Imag. 32(4), 797–808 (2013)CrossRefGoogle Scholar
  7. 7.
    Hofmanninger, J., Langs, G.: Mapping visual features to semantic profiles for retrieval in medical imaging. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 457–465 (2015)Google Scholar
  8. 8.
    Gao, M., Bagci, U., Lu, L., Wu, A., Buty, M., Shin, H-.C., Roth, H., Papadakis, G.Z., Depeursinge, A., Summers, R.M., et al.: Holistic classification of CT attenuation patterns for interstitial lung diseases via deep convolutional neural networks. Comput. Meth. Biomech. Biomed. Eng. Imag. Vis. 1–6 (2016). Taylor & FrancisGoogle Scholar
  9. 9.
    Gong, Y., Wang, L., Guo, R., Lazebnik, S.: Multi-scale orderless pooling of deep convolutional activation features. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8695, pp. 392–407. Springer, Heidelberg (2014). doi: 10.1007/978-3-319-10584-0_26 Google Scholar
  10. 10.
    Wei, Y., Xia, W., Huang, J., Ni, B., Dong, J., Zhao, Y., Yan, S.: CNN: Single-label to multi-label. arXiv preprint arXiv:1406.5726 (2014)
  11. 11.
    Zhen, X., Islam, A., Bhaduri, M., Chan, I., Li, S.: Direct and simultaneous four-chamber volume estimation by multi-output regression. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9349, pp. 669–676. Springer, Heidelberg (2015). doi: 10.1007/978-3-319-24553-9_82 CrossRefGoogle Scholar
  12. 12.
    Zhen, X., Wang, Z., Islam, A., Bhaduri, M., Chan, I., Li, S.: Direct estimation of cardiac bi-ventricular volumes with regression forests. In: Golland, P., Hata, N., Barillot, C., Hornegger, J., Howe, R. (eds.) MICCAI 2014. LNCS, vol. 8674, pp. 586–593. Springer, Heidelberg (2014). doi: 10.1007/978-3-319-10470-6_73 Google Scholar
  13. 13.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS, pp. 1097–1105 (2012)Google Scholar
  14. 14.
    Cimpoi, M., Maji, S., Kokkinos, I., Vedaldi, A.: Deep filter banks for texture recognition, description, and segmentation. Int. J. Comput. Vis. 118(1), 65–94 (2016)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Perronnin, F., Sánchez, J., Mensink, T.: Improving the fisher kernel for large-scale image classification. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6314, pp. 143–156. Springer, Heidelberg (2010). doi: 10.1007/978-3-642-15561-1_11 CrossRefGoogle Scholar
  16. 16.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  17. 17.
    Chatfield, K., Simonyan, K., Vedaldi, A., Zisserman, A.: Return of the devil in the details: delving deep into convolutional nets. arXiv preprint arXiv:1405.3531 (2014)
  18. 18.
    Gao, M., Xu, Z., Lu, L., Wu, A., Summers, R., Mollura, D.: Segmentation label propagation using deep convolutional neural networks and dense conditional random fields. In: IEEE ISBI (2016)Google Scholar
  19. 19.
    Vedaldi, A., Lenc, K.: MatConvNet: convolutional neural networks for matlab. In: Proceedings of the 23rd Annual ACM Conference on Multimedia Conference, pp. 689–692. ACM (2015)Google Scholar

Copyright information

© US Government (outside the US) 2016

Authors and Affiliations

  • Mingchen Gao
    • 1
    Email author
  • Ziyue Xu
    • 1
  • Le Lu
    • 1
  • Adam P. Harrison
    • 1
  • Ronald M. Summers
    • 1
  • Daniel J. Mollura
    • 1
  1. 1.Department of Radiology and Imaging SciencesNational Institutes of Health (NIH)BethesdaUSA

Personalised recommendations