Skip to main content

Multi-model Ensemble Learning Architecture Based on 3D CNN for Lung Nodule Malignancy Suspiciousness Classification

Journal of Digital Imaging volume 33, pages 1242–1256 (2020)Cite this article

Abstract

Classification of benign and malignant in lung nodules using chest CT images is a key step in the diagnosis of early-stage lung cancer, as well as an effective way to improve the patients’ survival rate. However, due to the diversity of lung nodules and the visual similarity of lung nodules to their surrounding tissues, it is difficult to construct a robust classification model with conventional deep learning–based diagnostic methods. To address this problem, we propose a multi-model ensemble learning architecture based on 3D convolutional neural network (MMEL-3DCNN). This approach incorporates three key ideas: (1) Constructed multi-model network architecture can be well adapted to the heterogeneity of lung nodules. (2) The input that concatenated of the intensity image corresponding to the nodule mask, the original image, and the enhanced image corresponding to which can help training model to extract advanced feature with more discriminative capacity. (3) Select the corresponding model to different nodule size dynamically for prediction, which can improve the generalization ability of the model effectively. In addition, ensemble learning is applied in this paper to further improve the robustness of the nodule classification model. The proposed method has been experimentally verified on the public dataset, LIDC-IDRI. The experimental results show that the proposed MMEL-3DCNN architecture can obtain satisfactory classification results.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

43,34 €

Price includes VAT (Finland)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

References

  1. R.L. Siegel, K.D. Miller, A. Jemal, Cancer statistics, 2018, CA Cancer J Clin. 68 (2018) 7–30. https://doi.org/10.3322/caac.21442.

    Article  PubMed  Google Scholar 

  2. P.B. Bach, J.N. Mirkin, T.K. Oliver, C.G. Azzoli, D.A. Berry, O.W. Brawley, T. Byers, G.A. Colditz, M.K. Gould, J.R. Jett, A.L. Sabichi, R. Smith-Bindman, D.E. Wood, A. Qaseem, F.C. Detterbeck, Benefits and harms of CT screening for lung cancer: a systematic review, JAMA. 307 (2012) 2418–2429. https://doi.org/10.1001/jama.2012.5521.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  3. T. Atwater, C.M. Cook, P.P. Massion, The Pursuit of Noninvasive Diagnosis of Lung Cancer, Semin Respir Crit Care Med. 37 (2016) 670–680. https://doi.org/10.1055/s-0036-1592314.

    Article  PubMed  PubMed Central  Google Scholar 

  4. M.M. Wahidi, J.A. Govert, R.K. Goudar, M.K. Gould, D.C. McCrory, Evidence for the Treatment of Patients With Pulmonary Nodules: When Is It Lung Cancer?: ACCP Evidence-Based Clinical Practice Guidelines (2nd Edition), Chest. 132 (2007) 94S-107S. https://doi.org/10.1378/chest.07-1352.

    Article  PubMed  Google Scholar 

  5. M.K. Gould, L. Ananth, P.G. Barnett, A Clinical Model To Estimate the Pretest Probability of Lung Cancer in Patients With Solitary Pulmonary Nodules, Chest. 131 (2007) 383–388. https://doi.org/10.1378/chest.06-1261.

    Article  PubMed  PubMed Central  Google Scholar 

  6. M.K. Gould, J. Donington, W.R. Lynch, P.J. Mazzone, D.E. Midthun, D.P. Naidich, R.S. Wiener, Evaluation of Individuals With Pulmonary Nodules: When Is It Lung Cancer?: Diagnosis and Management of Lung Cancer, 3rd ed: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines, Chest. 143 (2013) e93S-e120S. https://doi.org/10.1378/chest.12-2351.

    Article  PubMed  PubMed Central  Google Scholar 

  7. K. Mao, Z. Deng, Lung Nodule Image Classification Based on Local Difference Pattern and Combined Classifier, Comput Math Methods Med. 2016 (2016) 1091279. https://doi.org/10.1155/2016/1091279.

    Article  PubMed  PubMed Central  Google Scholar 

  8. B.R. Froz, A.O. de Carvalho Filho, A.C. Silva, A.C. de Paiva, R. Acatauassú Nunes, M. Gattass, Lung nodule classification using artificial crawlers, directional texture and support vector machine, Expert Syst Appl. 69 (2017) 176–188. https://doi.org/10.1016/j.eswa.2016.10.039.

    Article  Google Scholar 

  9. G. Wei, H. Cao, H. Ma, S. Qi, W. Qian, Z. Ma, Content-based image retrieval for Lung Nodule Classification Using Texture Features and Learned Distance Metric, J Med Syst. 42 (2017) 13. https://doi.org/10.1007/s10916-017-0874-5.

    Article  PubMed  Google Scholar 

  10. R. Dey, Z. Lu, Y. Hong, Diagnostic classification of lung nodules using 3D neural networks, in: 2018 IEEE 15th Int. Symp. Biomed. Imaging (ISBI 2018), 2018: pp. 774–778. https://doi.org/10.1109/ISBI.2018.8363687.

  11. G. Polat, Y.S. Dogrusöz, U. Halici, Effect of input size on the classification of lung nodules using convolutional neural networks, in: 2018 26th Signal Process. Commun. Appl. Conf., 2018: pp. 1–4. https://doi.org/10.1109/SIU.2018.8404659.

  12. Y. Xie, J. Zhang, Y. Xia, M. Fulham, Y. Zhang, Fusing texture, shape and deep model-learned information at decision level for automated classification of lung nodules on chest CT, Inf Fusion. 42 (2018) 102–110. https://doi.org/10.1016/j.inffus.2017.10.005.

    Article  Google Scholar 

  13. M. Sakamoto, H. Nakano, K. Zhao, T. Sekiyama, Lung nodule classification by the combination of fusion classifier and cascaded convolutional neural networks, in: 2018 IEEE 15th Int. Symp. Biomed. Imaging (ISBI 2018), 2018: pp. 822–825. https://doi.org/10.1109/ISBI.2018.8363698.

  14. R.V.M. d. Nóbrega, S.A. Peixoto, S.P.P. d. Silva, P.P.R. Filho, Lung Nodule Classification via Deep Transfer Learning in CT Lung Images, in: 2018 IEEE 31st Int. Symp. Comput. Med. Syst., 2018: pp. 244–249. https://doi.org/10.1109/CBMS.2018.00050.

  15. D. Zhao, D. Zhu, J. Lu, Y. Luo, G. Zhang, Synthetic Medical Images Using F&BGAN for Improved Lung Nodules Classification by Multi-Scale VGG16., Symmetry (Basel). 10 (2018) 519. https://doi.org/10.3390/sym10100519.

    Article  Google Scholar 

  16. J. Lyu, S.H. Ling, Using Multi-level Convolutional Neural Network for Classification of Lung Nodules on CT images, in: 2018 40th Annu. Int. Conf. IEEE Eng. Med. Biol. Soc., 2018: pp. 686–689. https://doi.org/10.1109/EMBC.2018.8512376.

  17. K. Simonyan, A. Zisserman, Very Deep Convolutional Networks for Large-Scale Image Recognition., CoRR. abs/1409.1 (2014). http://arxiv.org/abs/1409.1556.

  18. K. He, X. Zhang, S. Ren, J. Sun, Deep Residual Learning for Image Recognition, in: 2016 IEEE Conf. Comput. Vis. Pattern Recognit., 2016: pp. 770–778. https://doi.org/10.1109/CVPR.2016.90.

  19. C. Szegedy, S. Ioffe, V. Vanhoucke, A.A. Alemi, Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning. BT - Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA., (2017) 4278–4284. http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14806.

  20. M.C. Lee, L. Boroczky, K. Sungur-Stasik, A.D. Cann, A.C. Borczuk, S.M. Kawut, C.A. Powell, Computer-aided diagnosis of pulmonary nodules using a two-step approach for feature selection and classifier ensemble construction, Artif Intell Med. 50 (2010) 43–53. https://doi.org/10.1016/j.artmed.2010.04.011.

    Article  PubMed  Google Scholar 

  21. S.M. Barros Netto, A.C. Silva, R. Acatauassú Nunes, M. Gattass, Analysis of directional patterns of lung nodules in computerized tomography using Getis statistics and their accumulated forms as malignancy and benignity indicators, Pattern Recognit Lett. 33 (2012) 1734–1740. https://doi.org/10.1016/j.patrec.2012.05.010.

    Article  Google Scholar 

  22. D.S. Elizabeth, H.K. Nehemiah, C.S.R. Raj, A. Kannan, Computer-aided diagnosis of lung cancer based on analysis of the significant slice of chest computed tomography image, IET Image Process. 6 (2012) 697–705. https://doi.org/10.1049/iet-ipr.2010.0521.

    Article  Google Scholar 

  23. A. Kaya, A.B. Can, A weighted rule based method for predicting malignancy of pulmonary nodules by nodule characteristics, J. Biomed. Inform. 56 (2015) 69–79. https://doi.org/10.1016/j.jbi.2015.05.011.

    Article  PubMed  Google Scholar 

  24. M. Firmino, G. Angelo, H. Morais, M.R. Dantas, R. Valentim, Computer-aided detection (CADe) and diagnosis (CADx) system for lung cancer with likelihood of malignancy, Biomed Eng Online. 15 (2016) 2. https://doi.org/10.1186/s12938-015-0120-7.

    Article  PubMed  PubMed Central  Google Scholar 

  25. R.W. de Sousa Costa, G.L.F. da Silva, A.O. de Carvalho Filho, A.C. Silva, A.C. de Paiva, M. Gattass, Classification of malignant and benign lung nodules using taxonomic diversity index and phylogenetic distance, Med. Biol. Eng. Comput. 56 (2018) 2125–2136. https://doi.org/10.1007/s11517-018-1841-0.

    Article  PubMed  Google Scholar 

  26. M.B. Rodrigues, R.V.M. Da Nóbrega, S.S.A. Alves, P.P.R. Filho, J.B.F. Duarte, A.K. Sangaiah, V.H.C. De Albuquerque, Health of Things Algorithms for Malignancy Level Classification of Lung Nodules, IEEE Access. 6 (2018) 18592–18601. https://doi.org/10.1109/ACCESS.2018.2817614.

    Article  Google Scholar 

  27. A.K. Dhara, S. Mukhopadhyay, A. Dutta, M. Garg, N. Khandelwal, P. Kumar, Classification of pulmonary nodules in lung CT images using shape and texture features, in: 2016: pp. 97852Y-9785–6.

  28. B. Sasidhar, G. Geetha, B.I. Khodanpur, D.R. Ramesh Babu, Automatic Classification of Lung Nodules into Benign or Malignant Using SVM Classifier BT - Proceedings of the 5th International Conference on Frontiers in Intelligent Computing: Theory and Applications, in: S.C. Satapathy, V. Bhateja, S.K. Udgata, P.K. Pattnaik (Eds.), Springer Singapore, Singapore, 2017: pp. 551–559.

  29. H. Hu, S. Nie, Classification of malignant-benign pulmonary nodules in lung CT images using an improved random forest (Use style: Paper title), in: 2017 13th Int. Conf. Nat. Comput. Fuzzy Syst. Knowl. Discov., 2017: pp. 2285–2290. https://doi.org/10.1109/FSKD.2017.8393127.

  30. G. Wei, H. Ma, W. Qian, F. Han, H. Jiang, S. Qi, M. Qiu, Lung nodule classification using local kernel regression models with out-of-sample extension, Biomed. Signal Process. Control. 40 (2018) 1–9. https://doi.org/10.1016/j.bspc.2017.08.026.

    CAS  Article  Google Scholar 

  31. J.C.M. Bobadilla, H. Pedrini, Lung Nodule Classification Based on Deep Convolutional Neural Networks BT - Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications, in: C. Beltrán-Castañón, I. Nyström, F. Famili (Eds.), Springer International Publishing, Cham, 2017: pp. 117–124.

    Google Scholar 

  32. W. Sun, B. Zheng, W. Qian, Automatic feature learning using multichannel ROI based on deep structured algorithms for computerized lung cancer diagnosis, Comput Biol Med. 89 (2017) 530–539. https://doi.org/10.1016/j.compbiomed.2017.04.006.

    Article  PubMed  Google Scholar 

  33. Y. Xie, Y. Xia, J. Zhang, D.D. Feng, M. Fulham, W. Cai, Transferable Multi-model Ensemble for Benign-Malignant Lung Nodule Classification on Chest CT BT - Medical Image Computing and Computer-Assisted Intervention − MICCAI 2017, in: M. Descoteaux, L. Maier-Hein, A. Franz, P. Jannin, D.L. Collins, S. Duchesne (Eds.), Springer International Publishing, Cham, 2017: pp. 656–664.

    Google Scholar 

  34. D. Lückehe, G. von Voigt, Evolutionary image simplification for lung nodule classification with convolutional neural networks, Int. J. Comput. Assist. Radiol. Surg. 13 (2018) 1499–1513. https://doi.org/10.1007/s11548-018-1794-7.

    Article  PubMed  Google Scholar 

  35. M. Buty, Z. Xu, M. Gao, U. Bagci, A. Wu, D.J. Mollura, Characterization of Lung Nodule Malignancy Using Hybrid Shape and Appearance Features BT - Medical Image Computing and Computer-Assisted Intervention – MICCAI 2016, in: S. Ourselin, L. Joskowicz, M.R. Sabuncu, G. Unal, W. Wells (Eds.), Springer International Publishing, Cham, 2016: pp. 662–670.

    Google Scholar 

  36. J.L. Causey, J. Zhang, S. Ma, B. Jiang, J.A. Qualls, D.G. Politte, F. Prior, S. Zhang, X. Huang, Highly accurate model for prediction of lung nodule malignancy with CT scans, Sci Rep. 8 (2018) 9286. https://doi.org/10.1038/s41598-018-27569-w.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  37. K. Liu, G. Kang, Multiview convolutional neural networks for lung nodule classification., Int J Imaging Syst Technol. 27 (2017) 12–22. https://doi.org/10.1002/ima.22206.

    Article  Google Scholar 

  38. H. Lee, H. Lee, M. Park, J. Kim, Contextual convolutional neural networks for lung nodule classification using Gaussian-weighted average image patches, in: Proc.SPIE, 2017. https://doi.org/10.1117/12.2253978.

  39. W. Sun, B. Zheng, X. Huang, W. Qian, Balance the nodule shape and surroundings: a new multichannel image based convolutional neural network scheme on lung nodule diagnosis, in: Proc.SPIE, 2017. https://doi.org/10.1117/12.2251297.

  40. Y. Liu, P. Hao, P. Zhang, X. Xu, J. Wu, W. Chen, Dense Convolutional Binary-Tree Networks for Lung Nodule Classification, IEEE Access. 6 (2018) 49080–49088. https://doi.org/10.1109/ACCESS.2018.2865544.

    Article  Google Scholar 

  41. X. Yan, J. Pang, H. Qi, Y. Zhu, C. Bai, X. Geng, M. Liu, D. Terzopoulos, X. Ding, Classification of Lung Nodule Malignancy Risk on Computed Tomography Images Using Convolutional Neural Network: A Comparison Between 2D and 3D Strategies BT - Computer Vision – ACCV 2016 Workshops, in: C.-S. Chen, J. Lu, K.-K. Ma (Eds.), Springer International Publishing, Cham, 2017: pp. 91–101.

    Google Scholar 

  42. S. Shen, S.X. Han, D.R. Aberle, A.A.T. Bui, W. Hsu, An Interpretable Deep Hierarchical Semantic Convolutional Neural Network for Lung Nodule Malignancy Classification., CoRR. abs/1806.0 (2018). http://arxiv.org/abs/1806.00712.

  43. X. Li, Y. Kao, W. Shen, X. Li, G. Xie, Lung nodule malignancy prediction using multi-task convolutional neural network, in: Proc.SPIE, 2017. https://doi.org/10.1117/12.2253836.

  44. D. Ardila, A.P. Kiraly, S. Bharadwaj, B. Choi, J.J. Reicher, L. Peng, D. Tse, M. Etemadi, W. Ye, G. Corrado, D.P. Naidich, S. Shetty, End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography, Nat. Med. 25 (2019) 954–961. https://doi.org/10.1038/s41591-019-0447-x.

    CAS  Article  PubMed  Google Scholar 

  45. G. Kang, K. Liu, B. Hou, N. Zhang, 3D multi-view convolutional neural networks for lung nodule classification, PLoS One. 12 (2017) e0188290. doi:https://doi.org/10.1371/journal.pone.0188290.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  46. W. Shen, M. Zhou, F. Yang, C. Yang, J. Tian, Multi-scale Convolutional Neural Networks for Lung Nodule Classification, Inf Process Med Imaging. 24 (2015) 588–599. http://europepmc.org/abstract/MED/26221705.

    PubMed  Google Scholar 

  47. W. Shen, M. Zhou, F. Yang, D. Yu, D. Dong, C. Yang, Y. Zang, J. Tian, Multi-crop Convolutional Neural Networks for lung nodule malignancy suspiciousness classification, Pattern Recognit. 61 (2017) 663–673. https://doi.org/10.1016/j.patcog.2016.05.029.

    Article  Google Scholar 

  48. S. Ioffe, C. Szegedy, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. BT - Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, (2015) 448–456. http://jmlr.org/proceedings/papers/v37/ioffe15.html.

    Google Scholar 

  49. K. He, X. Zhang, S. Ren, J. Sun, Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification, in: 2015 IEEE Int. Conf. Comput. Vis., 2015: pp. 1026–1034. https://doi.org/10.1109/ICCV.2015.123.

  50. H. Mzoughi, I. Njeh, M. Ben Slima, A. Ben Hamida, Histogram equalization-based techniques for contrast enhancement of MRI brain Glioma tumor images: Comparative study, in: 2018 4th Int. Conf. Adv. Technol. Signal Image Process., 2018: pp. 1–6. https://doi.org/10.1109/ATSIP.2018.8364471.

  51. P. Amorim, T. Moraes, J. Silva, H. Pedrini, 3D Adaptive Histogram Equalization Method for Medical Volumes. BT - Proceedings of the 13th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2018) - Volume 4: VISAPP, Funchal, Madeira, Po, (2018) 363–370. https://doi.org/10.5220/0006615303630370.

  52. S. Roy, P. Ghosh, S.K. Bandyopadhyay, Contour Extraction and Segmentation of Cerebral Hemorrhage from MRI of Brain by Gamma Transformation Approach BT - Proceedings of the 3rd International Conference on Frontiers of Intelligent Computing: Theory and Applications (FICTA) 2014, in: S.C. Satapathy, B.N. Biswal, S.K. Udgata, J.K. Mandal (Eds.), Springer International Publishing, Cham, 2015: pp. 383–394.

  53. T. Zhan, M. Gong, X. Jiang, S. Li, Log-Based Transformation Feature Learning for Change Detection in Heterogeneous Images, IEEE Geosci Remote Sens Lett. 15 (2018) 1352–1356. https://doi.org/10.1109/LGRS.2018.2843385.

    Article  Google Scholar 

  54. D.P. Kingma, J. Ba, Adam: A Method for Stochastic Optimization., CoRR. abs/1412.6 (2014). http://arxiv.org/abs/1412.6980.

  55. R. Caruana, S. Lawrence, C.L. Giles, Overfitting in Neural Nets: Backpropagation, Conjugate Gradient, and Early Stopping. BT - Advances in Neural Information Processing Systems 13, Papers from Neural Information Processing Systems (NIPS) 2000, Denver, CO, USA, (2000) 402–408. http://papers.nips.cc/paper/1895-overfitting-in-neural-nets-backpropagation-conjugate-gradient-and-early-stopping.

  56. F. Han, H. Wang, G. Zhang, H. Han, B. Song, L. Li, W. Moore, H. Lu, H. Zhao, Z. Liang, Texture Feature Analysis for Computer-Aided Diagnosis on Pulmonary Nodules, J. Digit. Imaging. 28 (2015) 99–115. https://doi.org/10.1007/s10278-014-9718-8.

    Article  PubMed  Google Scholar 

  57. F. Han, G. Zhang, H. Wang, B. Song, H. Lu, D. Zhao, H. Zhao, Z. Liang, A texture feature analysis for diagnosis of pulmonary nodules using LIDC-IDRI database, in: 2013 IEEE Int. Conf. Med. Imaging Phys. Eng., 2013: pp. 14–18. https://doi.org/10.1109/ICMIPE.2013.6864494.

Download references

Acknowledgments

The authors acknowledge the National Cancer Institute and the Foundation for the National Institutes of Health and their critical role in the creation of the free publicly available LIDC-IDRI Database used in this study.

Funding

The National Key R&D Program of China (Grant No. 2017YFC0112804) supported this work.

Author information

Affiliations

  1. School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, 430074, China

    Hong Liu, Haichao Cao, Enmin Song, Guangzhi Ma, Xiangyang Xu, Renchao Jin & Chuhua Liu

  2. Laboratory for Machine Vision and Security Research, Kennesaw State University, Kennesaw, GA, USA

    Chih-Cheng Hung

Authors
  1. Hong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Haichao Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Enmin Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Guangzhi Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiangyang Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Renchao Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Chuhua Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Chih-Cheng Hung
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Enmin Song.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflicts of interest.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Liu, H., Cao, H., Song, E. et al. Multi-model Ensemble Learning Architecture Based on 3D CNN for Lung Nodule Malignancy Suspiciousness Classification. J Digit Imaging 33, 1242–1256 (2020). https://doi.org/10.1007/s10278-020-00372-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10278-020-00372-8

Keywords

  • Benign and malignant classification
  • Computer-aided diagnosis
  • Image enhancement
  • Multi-model ensemble architecture
  • 3D CNN