Skip to main content

CT-based deep learning model to differentiate invasive pulmonary adenocarcinomas appearing as subsolid nodules among surgical candidates: comparison of the diagnostic performance with a size-based logistic model and radiologists

European Radiology volume 30, pages 3295–3305 (2020)Cite this article

Abstract

Objectives

To evaluate the deep learning models for differentiating invasive pulmonary adenocarcinomas (IACs) among subsolid nodules (SSNs) considered for resection in a retrospective diagnostic cohort in comparison with a size-based logistic model and expert radiologists.

Methods

This study included 525 patients (309 women; median, 62 years) to develop models, and an independent cohort of 101 patients (57 women; median, 66 years) was used for validation. A size-based logistic model and deep learning models using 2.5-dimension (2.5D) and three-dimension (3D) CT images were developed to discriminate IAC from less invasive pathologies. Overall performance, discrimination, and calibration were assessed. Diagnostic performances of the three thoracic radiologists were compared with those of the deep learning model.

Results

The overall performances of the deep learning models (Brier score, 0.122 for the 2.5D DenseNet and 0.121 for the 3D DenseNet) were superior to those of the size-based logistic model (Brier score, 0.198). The area under the receiver operating characteristic curve (AUC) of the 2.5D DenseNet (0.921) was significantly higher than that of the 3D DenseNet (0.835; p = 0.037) and the size-based logistic model (0.836; p = 0.009). At equally high sensitivities of 90%, the 2.5D DenseNet showed significantly higher specificity (88.2%; all p < 0.05) and positive predictive value (97.4%; all p < 0.05) than other models. Model calibration was poor for all models (all p < 0.05). The 2.5D DenseNet had a comparable performance with the radiologists (AUC, 0.848–0.910).

Conclusion

The 2.5D DenseNet model could be used as a highly sensitive and specific diagnostic tool to differentiate IACs among SSNs for surgical candidates.

Key Points

• The deep learning model developed using 2.5D DenseNet showed higher overall performance and discrimination than the size-based logistic model for the differentiation of invasive adenocarcinomas among subsolid nodules for surgical candidates.

• The 2.5D DenseNet demonstrated a thoracic radiologist–level diagnostic performance and had higher specificity (88.2%) at equal sensitivities (90%) than the size-based logistic model (specificity, 52.9%).

• The 2.5D DenseNet could be used to reduce potential overtreatment for the indolent subsolid nodules or to select candidates for sublobar resection instead of the standard lobectomy.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

43,34 €

Price includes VAT (Finland)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Abbreviations

2.5D:

2.5-dimension

3D:

Three-dimension

AAH:

Atypical adenomatous hyperplasia

AIS:

Adenocarcinoma in situ

AUC:

Area under the receiver operating characteristic curve

CI:

Confidence interval

IAC:

Invasive adenocarcinoma

IQR:

Interquartile range

MIA:

Minimally invasive adenocarcinoma

NPV:

Negative predictive value

pGGN:

Pure ground-glass nodule

PPV:

Positive predictive value

PSN:

Part-solid nodule

SSN:

Subsolid nodule

References

  1. Travis WD, Asamura H, Bankier AA et al (2016) The IASLC Lung Cancer Staging Project: proposals for coding T categories for subsolid nodules and assessment of tumor size in part-solid tumors in the forthcoming eighth edition of the TNM classification of lung cancer. J Thorac Oncol 11:1204–1223

    Article  Google Scholar 

  2. Asamura H, Aokage K, Yotsukura M (2017) Wedge resection versus anatomic resection: extent of surgical resection for stage I and II lung cancer. Am Soc Clin Oncol Educ Book 37:426–433

    Article  Google Scholar 

  3. Liu S, Wang R, Zhang Y et al (2016) Precise diagnosis of intraoperative frozen section is an effective method to guide resection strategy for peripheral small-sized lung adenocarcinoma. J Clin Oncol 34:307–313

    Article  Google Scholar 

  4. Boughey JC, Keeney GL, Radensky P, Song CP, Habermann EB (2016) Economic implications of widespread expansion of frozen section margin analysis to guide surgical resection in women with breast cancer undergoing breast-conserving surgery. J Oncol Pract 12:e413–e422

    Article  Google Scholar 

  5. Zhu E, Xie H, Dai C et al (2018) Intraoperatively measured tumor size and frozen section results should be considered jointly to predict the final pathology for lung adenocarcinoma. Mod Pathol 31:1391–1399

    Article  Google Scholar 

  6. Kim H, Goo JM, Park CM (2019) A simple prediction model using size measures for discrimination of invasive adenocarcinomas among incidental pulmonary subsolid nodules considered for resection. Eur Radiol 29:1674–1683

    Article  Google Scholar 

  7. Kim H, Park CM, Hwang EJ, Ahn SY, Goo JM (2018) Pulmonary subsolid nodules: value of semi-automatic measurement in diagnostic accuracy, diagnostic reproducibility and nodule classification agreement. Eur Radiol 28:2124–2133

    Article  Google Scholar 

  8. Lee SM, Park CM, Goo JM, Lee HJ, Wi JY, Kang CH (2013) Invasive pulmonary adenocarcinomas versus preinvasive lesions appearing as ground-glass nodules: differentiation by using CT features. Radiology 268:265–273

    Article  Google Scholar 

  9. Zhan Y, Peng X, Shan F et al (2019) Attenuation and morphologic characteristics distinguishing a ground-glass nodule measuring 5-10 mm in diameter as invasive lung adenocarcinoma on thin-slice CT. AJR Am J Roentgenol 213:1–9

    Article  Google Scholar 

  10. Zhang Y, Qiang JW, Ye JD, Ye XD, Zhang J (2014) High resolution CT in differentiating minimally invasive component in early lung adenocarcinoma. Lung Cancer 84:236–241

    Article  Google Scholar 

  11. Chae HD, Park CM, Park SJ, Lee SM, Kim KG, Goo JM (2014) Computerized texture analysis of persistent part-solid ground-glass nodules: differentiation of preinvasive lesions from invasive pulmonary adenocarcinomas. Radiology 273:285–293

    Article  Google Scholar 

  12. Fan L, Fang M, Li Z et al (2019) Radiomics signature: a biomarker for the preoperative discrimination of lung invasive adenocarcinoma manifesting as a ground-glass nodule. Eur Radiol 29:889–897

    Article  Google Scholar 

  13. Kim H, Park CM, Gwak J et al (2019) Effect of CT reconstruction algorithm on the diagnostic performance of radiomics models: a task-based approach for pulmonary subsolid nodules. AJR Am J Roentgenol 212:505–512

    Article  Google Scholar 

  14. She Y, Zhang L, Zhu H et al (2018) The predictive value of CT-based radiomics in differentiating indolent from invasive lung adenocarcinoma in patients with pulmonary nodules. Eur Radiol 28:5121–5128

    Article  Google Scholar 

  15. Zhao W, Xu Y, Yang Z et al (2019) Development and validation of a radiomics nomogram for identifying invasiveness of pulmonary adenocarcinomas appearing as subcentimeter ground-glass opacity nodules. Eur J Radiol 112:161–168

    Article  Google Scholar 

  16. Wang S, Wang R, Zhang S et al (2018) 3D convolutional neural network for differentiating pre-invasive lesions from invasive adenocarcinomas appearing as ground-glass nodules with diameters </=3 cm using HRCT. Quant Imaging Med Surg 8:491–499

    CAS  Article  Google Scholar 

  17. Yanagawa M, Niioka H, Hata A et al (2019) Application of deep learning (3-dimensional convolutional neural network) for the prediction of pathological invasiveness in lung adenocarcinoma: a preliminary study. Medicine (Baltimore) 98:e16119

    Article  Google Scholar 

  18. Zhao W, Yang J, Sun Y et al (2018) 3D deep learning from CT scans predicts tumor invasiveness of subcentimeter pulmonary adenocarcinomas. Cancer Res 78:6881–6889

    CAS  Article  Google Scholar 

  19. Austin PC, van Klaveren D, Vergouwe Y, Nieboer D, Lee DS, Steyerberg EW (2017) Validation of prediction models: examining temporal and geographic stability of baseline risk and estimated covariate effects. Diagn Progn Res 1:12

    Article  Google Scholar 

  20. Moons KG, Altman DG, Reitsma JB et al (2015) Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med 162:W1–W73

    Article  Google Scholar 

  21. Naidich DP, Bankier AA, MacMahon H et al (2013) Recommendations for the management of subsolid pulmonary nodules detected at CT: a statement from the Fleischner Society. Radiology 266:304–317

    Article  Google Scholar 

  22. Dunnmon JA, Yi D, Langlotz CP, Re C, Rubin DL, Lungren MP (2019) Assessment of convolutional neural networks for automated classification of chest radiographs. Radiology 290:537–544

    Article  Google Scholar 

  23. Rajpurkar P, Irvin J, Ball RL et al (2018) Deep learning for chest radiograph diagnosis: a retrospective comparison of the CheXNeXt algorithm to practicing radiologists. PLoS Med 15:e1002686

    Article  Google Scholar 

  24. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition 4700–4708

  25. Bui TD, Shin J, Moon T (2017) 3D densely convolutional networks for volumetric segmentation. arXiv:170903199

  26. Steyerberg EW, Vickers AJ, Cook NR et al (2010) Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 21:128–138

    Article  Google Scholar 

  27. Rufibach K (2010) Use of Brier score to assess binary predictions. J Clin Epidemiol 63:938–939

    Article  Google Scholar 

  28. DeLong ER, DeLong DM, Clarke-Pearson DL (1988) Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44:837–845

    CAS  Article  Google Scholar 

  29. McNemar Q (1947) Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12:153–157

    CAS  Article  Google Scholar 

  30. Leisenring W, Alonzo T, Pepe MS (2000) Comparisons of predictive values of binary medical diagnostic tests for paired designs. Biometrics 56:345–351

    CAS  Article  Google Scholar 

  31. Walsh CG, Sharman K, Hripcsak G (2017) Beyond discrimination: a comparison of calibration methods and clinical usefulness of predictive models of readmission risk. J Biomed Inform 76:9–18

    Article  Google Scholar 

  32. Tammemagi MC, Ten Haaf K, Toumazis I et al (2019) Development and validation of a multivariable lung cancer risk prediction model that includes low-dose computed tomography screening results: a secondary analysis of data from the National Lung Screening Trial. JAMA Netw Open 2:e190204

    Article  Google Scholar 

  33. Chung JH, Choe G, Jheon S et al (2009) Epidermal growth factor receptor mutation and pathologic-radiologic correlation between multiple lung nodules with ground-glass opacity differentiates multicentric origin from intrapulmonary spread. J Thorac Oncol 4:1490–1495

    Article  Google Scholar 

  34. Gillies RJ, Kinahan PE, Hricak H (2016) Radiomics: images are more than pictures, they are data. Radiology 278:563–577

    Article  Google Scholar 

  35. Bankier AA, MacMahon H, Goo JM, Rubin GD, Schaefer-Prokop CM, Naidich DP (2017) Recommendations for measuring pulmonary nodules at CT: a statement from the Fleischner Society. Radiology 285:584–600

    Article  Google Scholar 

  36. MacMahon H, Naidich DP, Goo JM et al (2017) Guidelines for management of incidental pulmonary nodules detected on CT images: from the Fleischner Society 2017. Radiology 284:228–243

    Article  Google Scholar 

  37. Lee KH, Goo JM, Park SJ et al (2014) Correlation between the size of the solid component on thin-section CT and the invasive component on pathology in small lung adenocarcinomas manifesting as ground-glass nodules. J Thorac Oncol 9:74–82

    Article  Google Scholar 

  38. Kim DW, Jang HY, Kim KW, Shin Y, Park SH (2019) Design characteristics of studies reporting the performance of artificial intelligence algorithms for diagnostic analysis of medical images: results from recently published papers. Korean J Radiol 20:405–410

    Article  Google Scholar 

  39. Park SH, Han K (2018) Methodologic guide for evaluating clinical performance and effect of artificial intelligence technology for medical diagnosis and prediction. Radiology 286:800–809

    Article  Google Scholar 

Download references

Acknowledgments

We sincerely express our gratitude to Sunkyung Jeon, Jong Hyuk Lee, Su Yeon Ahn, Roh-Eul Yoo, Hyun-ju Lim, Juil Park, Woo Hyeon Lim, and Myunghee Lee for data acquisition. We are also grateful to Jung Hee Hong, Eui Jin Hwang, and Soon Ho Yoon for participating in the observer performance test.

Funding

This study was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF), funded by the Ministry of Science, ICT & Future Planning (grant number: 2017R1A2B4008517).

Author information

Author notes

  1. Hyungjin Kim and Dongheon Lee share joint first authorship as they contributed equally to this work.

Affiliations

  1. Department of Radiology, Seoul National University Hospital, Seoul National University College of Medicine, 101, Daehak-ro, Jongno-gu, Seoul, 03080, South Korea

    Hyungjin Kim, Jin Mo Goo & Chang Min Park

  2. Institute of Radiation Medicine, Seoul National University Medical Research Center, 101, Daehak-ro, Jongno-gu, Seoul, 03080, South Korea

    Hyungjin Kim, Jin Mo Goo & Chang Min Park

  3. Interdisciplinary Program in Bioengineering, Graduate School, Seoul National University, 101, Daehak-ro, Jongno-gu, Seoul, 03080, South Korea

    Dongheon Lee, Woo Sang Cho & Hee Chan Kim

  4. Department of Biomedical Engineering, Seoul National University College of Medicine, 101, Daehak-ro, Jongno-gu, Seoul, 03080, South Korea

    Jung Chan Lee & Hee Chan Kim

  5. Institute of Medical & Biological Engineering, Medical Research Center, Seoul National University, 101, Daehak-ro, Jongno-gu, Seoul, 03080, South Korea

    Jung Chan Lee & Hee Chan Kim

  6. Cancer Research Institute, Seoul National University, 101, Daehak-ro, Jongno-gu, Seoul, 03080, South Korea

    Jin Mo Goo & Chang Min Park

Authors
  1. Hyungjin Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dongheon Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Woo Sang Cho
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jung Chan Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jin Mo Goo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hee Chan Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Chang Min Park
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Hee Chan Kim or Chang Min Park.

Ethics declarations

Guarantor

The scientific guarantor of this publication is Chang Min Park.

Conflict of interest

Activities related to the present article: none.

Activities not related to the present article: H.K., J.M.G., and C.M.P received research grants from Lunit Inc. (Seoul, South Korea).

Statistics and biometry

No complex statistical methods were necessary for this paper.

Informed consent

Written informed consent was waived by the Institutional Review Board.

Ethical approval

Institutional Review Board approval was obtained.

Study subjects or cohorts overlap

Some study subjects or cohorts have been previously reported in journal articles (Eur Radiol 2016 26:4465–4474; Eur J Radiol 2016 85:1174–1180; Eur Radiol 2017 27:3266–3274; Eur Radiol 2017 27:1369–1376; Eur Radiol 2018 28:2124–2133; Eur Radiol 2019 29:1674–1683; Eur Radiol 2019 29:1586–1594).

Methodology

• retrospective

• diagnostic or prognostic study

• performed at one institution

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

ESM 1

(DOCX 42 kb)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Kim, H., Lee, D., Cho, W.S. et al. CT-based deep learning model to differentiate invasive pulmonary adenocarcinomas appearing as subsolid nodules among surgical candidates: comparison of the diagnostic performance with a size-based logistic model and radiologists. Eur Radiol 30, 3295–3305 (2020). https://doi.org/10.1007/s00330-019-06628-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00330-019-06628-4

Keywords

  • Adenocarcinoma
  • Multidetector computed tomography
  • Computer-assisted radiographic image interpretation
  • Artificial intelligence
  • Logistic model