Risk Stratification of Lung Nodules Using 3D CNN-Based Multi-task Learning

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10265)


Risk stratification of lung nodules is a task of primary importance in lung cancer diagnosis. Any improvement in robust and accurate nodule characterization can assist in identifying cancer stage, prognosis, and improving treatment planning. In this study, we propose a 3D Convolutional Neural Network (CNN) based nodule characterization strategy. With a completely 3D approach, we utilize the volumetric information from a CT scan which would be otherwise lost in the conventional 2D CNN based approaches. In order to address the need for a large amount of training data for CNN, we resort to transfer learning to obtain highly discriminative features. Moreover, we also acquire the task dependent feature representation for six high-level nodule attributes and fuse this complementary information via a Multi-task learning (MTL) framework. Finally, we propose to incorporate potential disagreement among radiologists while scoring different nodule attributes in a graph regularized sparse multi-task learning. We evaluated our proposed approach on one of the largest publicly available lung nodule datasets comprising 1018 scans and obtained state-of-the-art results in regressing the malignancy scores.


Computer-Aided Diagnosis (CAD) Lung nodule characterization 3D Convolutional Neural Network Multi-task learning Transfer learning Computed Tomography (CT) Deep learning 


  1. 1.
    Stewart, B., Wild, C.P., et al.: World Cancer Report 2014. World, Mountain View (2016)Google Scholar
  2. 2.
    van Beek, E.J., Mirsadraee, S., Murchison, J.T.: Lung cancer screening: computed tomography or chest radiographs? World J. Radiol. 7(8), 189 (2015)CrossRefGoogle Scholar
  3. 3.
    Armato III, S., McLennan, G., Bidaut, L., McNitt-Gray, M.F., Meyer, C.R., Reeves, A.P., Zhao, B., Aberle, D.R., Henschke, C.I., Hoffman, E.A., et al.: The lung image database consortium (LIDC) and image database resource initiative (IDRI): a completed reference database of lung nodules on CT scans. Med. Phys. 38(2), 915–931 (2011)CrossRefGoogle Scholar
  4. 4.
    Shin, H., Roth, H.R., Gao, M., Lu, L., Xu, Z., Nogues, I., Yao, J., Mollura, D., Summers, R.M.: Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans. Med. Imaging 35(5), 1285–1298 (2016)CrossRefGoogle Scholar
  5. 5.
    Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3D convolutional networks. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp. 4489–4497. IEEE (2015)Google Scholar
  6. 6.
    Uchiyama, Y., Katsuragawa, S., Abe, H., Shiraishi, J., Li, F., Li, Q., Zhang, C.T., Suzuki, K., Doi, K.: Quantitative computerized analysis of diffuse lung disease in high-resolution computed tomography. Med. Phys. 30(9), 2440–2454 (2003)CrossRefGoogle Scholar
  7. 7.
    El-Baz, A., Nitzken, M., Khalifa, F., Elnakib, A., Gimel’farb, G., Falk, R., El-Ghar, M.A.: 3D shape analysis for early diagnosis of malignant lung nodules. In: Székely, G., Hahn, H.K. (eds.) IPMI 2011. LNCS, vol. 6801, pp. 772–783. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-22092-0_63 CrossRefGoogle Scholar
  8. 8.
    Han, F., Wang, H., Zhang, G., Han, H., Song, B., Li, L., Moore, W., Lu, H., Zhao, H., Liang, Z.: Texture feature analysis for computer-aided diagnosis on pulmonary nodules. J. Digit. Imaging 28(1), 99–115 (2015)CrossRefGoogle Scholar
  9. 9.
    Way, T.W., Hadjiiski, L.M., Sahiner, B., Chan, H.P., Cascade, P.N., Kazerooni, E.A., Bogot, N., Zhou, C.: Computer-aided diagnosis of pulmonary nodules on CT scans: segmentation and classification using 3D active contours. Med. Phys. 33(7), 2323–2337 (2006)CrossRefGoogle Scholar
  10. 10.
    Lee, M., Boroczky, L., Sungur-Stasik, K., Cann, A., Borczuk, A., Kawut, S., Powell, C.: Computer-aided diagnosis of pulmonary nodules using a two-step approach for feature selection and classifier ensemble construction. Artif. Intell. Med. 50(1), 43–53 (2010)CrossRefGoogle Scholar
  11. 11.
    Kumar, D., Wong, A., Clausi, D.A.: Lung nodule classification using deep features in CT images. In: 2015 12th Conference on Computer and Robot Vision (CRV), pp. 133–138. IEEE (2015)Google Scholar
  12. 12.
    Buty, M., Xu, Z., Gao, M., Bagci, U., Wu, A., Mollura, D.J.: Characterization of lung nodule malignancy using hybrid shape and appearance features. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9900, pp. 662–670. Springer, Cham (2016). doi: 10.1007/978-3-319-46720-7_77 CrossRefGoogle Scholar
  13. 13.
    Chen, S., Ni, D., Qin, J., Lei, B., Wang, T., Cheng, J.-Z.: Bridging computational features toward multiple semantic features with multi-task regression: a study of CT pulmonary nodules. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 53–60. Springer, Cham (2016). doi: 10.1007/978-3-319-46723-8_7 CrossRefGoogle Scholar
  14. 14.
    Furuya, K., Murayama, S., Soeda, H., Murakami, J., Ichinose, Y., Yauuchi, H., Katsuda, Y., Koga, M., Masuda, K.: New classification of small pulmonary nodules by margin characteristics on highresolution CT. Acta Radiol. 40(5), 496–504 (1999)CrossRefGoogle Scholar
  15. 15.
    Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., Fei-Fei, L.: Large-scale video classification with convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1725–1732 (2014)Google Scholar
  16. 16.
    Evgeniou, T., Pontil, M.: Regularized multi-task learning. In: Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 109–117. ACM (2004)Google Scholar
  17. 17.
    Zhou, J., Chen, J., Ye, J.: MALSAR: multi-task learning via structural regularization (2012)Google Scholar
  18. 18.
    Nesterov, Y.: Introductory Lectures on Convex Optimization: A Basic Course, vol. 87. Springer Science & Business Media, Berlin (2013)zbMATHGoogle Scholar
  19. 19.
    Parikh, N., Boyd, S., et al.: Proximal algorithms. Found. Trends® Optim. 1(3), 127–239 (2014)CrossRefGoogle Scholar
  20. 20.
    Oliva, A., Torralba, A.: Modeling the shape of the scene: a holistic representation of the spatial envelope. Int. J. Comput. Vis. 42(3), 145–175 (2001)CrossRefzbMATHGoogle Scholar
  21. 21.
    van der Maaten, L., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9(Nov), 2579–2605 (2008)zbMATHGoogle Scholar
  22. 22.
    Wang, Y.X.J., Gong, J.S., Suzuki, K., Morcos, S.K.: Evidence based imaging strategies for solitary pulmonary nodule. J. Thorac. Dis. 6(7), 872–887 (2014)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Center for Research in Computer Vision (CRCV)University of Central FloridaOrlandoUSA
  2. 2.CuraCloud CorporationSeattleUSA

Personalised recommendations