Abstract
Objectives
To evaluate the deep learning models for differentiating invasive pulmonary adenocarcinomas (IACs) among subsolid nodules (SSNs) considered for resection in a retrospective diagnostic cohort in comparison with a size-based logistic model and expert radiologists.
Methods
This study included 525 patients (309 women; median, 62 years) to develop models, and an independent cohort of 101 patients (57 women; median, 66 years) was used for validation. A size-based logistic model and deep learning models using 2.5-dimension (2.5D) and three-dimension (3D) CT images were developed to discriminate IAC from less invasive pathologies. Overall performance, discrimination, and calibration were assessed. Diagnostic performances of the three thoracic radiologists were compared with those of the deep learning model.
Results
The overall performances of the deep learning models (Brier score, 0.122 for the 2.5D DenseNet and 0.121 for the 3D DenseNet) were superior to those of the size-based logistic model (Brier score, 0.198). The area under the receiver operating characteristic curve (AUC) of the 2.5D DenseNet (0.921) was significantly higher than that of the 3D DenseNet (0.835; p = 0.037) and the size-based logistic model (0.836; p = 0.009). At equally high sensitivities of 90%, the 2.5D DenseNet showed significantly higher specificity (88.2%; all p < 0.05) and positive predictive value (97.4%; all p < 0.05) than other models. Model calibration was poor for all models (all p < 0.05). The 2.5D DenseNet had a comparable performance with the radiologists (AUC, 0.848–0.910).
Conclusion
The 2.5D DenseNet model could be used as a highly sensitive and specific diagnostic tool to differentiate IACs among SSNs for surgical candidates.
Key Points
• The deep learning model developed using 2.5D DenseNet showed higher overall performance and discrimination than the size-based logistic model for the differentiation of invasive adenocarcinomas among subsolid nodules for surgical candidates.
• The 2.5D DenseNet demonstrated a thoracic radiologist–level diagnostic performance and had higher specificity (88.2%) at equal sensitivities (90%) than the size-based logistic model (specificity, 52.9%).
• The 2.5D DenseNet could be used to reduce potential overtreatment for the indolent subsolid nodules or to select candidates for sublobar resection instead of the standard lobectomy.
Similar content being viewed by others
Abbreviations
- 2.5D:
-
2.5-dimension
- 3D:
-
Three-dimension
- AAH:
-
Atypical adenomatous hyperplasia
- AIS:
-
Adenocarcinoma in situ
- AUC:
-
Area under the receiver operating characteristic curve
- CI:
-
Confidence interval
- IAC:
-
Invasive adenocarcinoma
- IQR:
-
Interquartile range
- MIA:
-
Minimally invasive adenocarcinoma
- NPV:
-
Negative predictive value
- pGGN:
-
Pure ground-glass nodule
- PPV:
-
Positive predictive value
- PSN:
-
Part-solid nodule
- SSN:
-
Subsolid nodule
References
Travis WD, Asamura H, Bankier AA et al (2016) The IASLC Lung Cancer Staging Project: proposals for coding T categories for subsolid nodules and assessment of tumor size in part-solid tumors in the forthcoming eighth edition of the TNM classification of lung cancer. J Thorac Oncol 11:1204–1223
Asamura H, Aokage K, Yotsukura M (2017) Wedge resection versus anatomic resection: extent of surgical resection for stage I and II lung cancer. Am Soc Clin Oncol Educ Book 37:426–433
Liu S, Wang R, Zhang Y et al (2016) Precise diagnosis of intraoperative frozen section is an effective method to guide resection strategy for peripheral small-sized lung adenocarcinoma. J Clin Oncol 34:307–313
Boughey JC, Keeney GL, Radensky P, Song CP, Habermann EB (2016) Economic implications of widespread expansion of frozen section margin analysis to guide surgical resection in women with breast cancer undergoing breast-conserving surgery. J Oncol Pract 12:e413–e422
Zhu E, Xie H, Dai C et al (2018) Intraoperatively measured tumor size and frozen section results should be considered jointly to predict the final pathology for lung adenocarcinoma. Mod Pathol 31:1391–1399
Kim H, Goo JM, Park CM (2019) A simple prediction model using size measures for discrimination of invasive adenocarcinomas among incidental pulmonary subsolid nodules considered for resection. Eur Radiol 29:1674–1683
Kim H, Park CM, Hwang EJ, Ahn SY, Goo JM (2018) Pulmonary subsolid nodules: value of semi-automatic measurement in diagnostic accuracy, diagnostic reproducibility and nodule classification agreement. Eur Radiol 28:2124–2133
Lee SM, Park CM, Goo JM, Lee HJ, Wi JY, Kang CH (2013) Invasive pulmonary adenocarcinomas versus preinvasive lesions appearing as ground-glass nodules: differentiation by using CT features. Radiology 268:265–273
Zhan Y, Peng X, Shan F et al (2019) Attenuation and morphologic characteristics distinguishing a ground-glass nodule measuring 5-10 mm in diameter as invasive lung adenocarcinoma on thin-slice CT. AJR Am J Roentgenol 213:1–9
Zhang Y, Qiang JW, Ye JD, Ye XD, Zhang J (2014) High resolution CT in differentiating minimally invasive component in early lung adenocarcinoma. Lung Cancer 84:236–241
Chae HD, Park CM, Park SJ, Lee SM, Kim KG, Goo JM (2014) Computerized texture analysis of persistent part-solid ground-glass nodules: differentiation of preinvasive lesions from invasive pulmonary adenocarcinomas. Radiology 273:285–293
Fan L, Fang M, Li Z et al (2019) Radiomics signature: a biomarker for the preoperative discrimination of lung invasive adenocarcinoma manifesting as a ground-glass nodule. Eur Radiol 29:889–897
Kim H, Park CM, Gwak J et al (2019) Effect of CT reconstruction algorithm on the diagnostic performance of radiomics models: a task-based approach for pulmonary subsolid nodules. AJR Am J Roentgenol 212:505–512
She Y, Zhang L, Zhu H et al (2018) The predictive value of CT-based radiomics in differentiating indolent from invasive lung adenocarcinoma in patients with pulmonary nodules. Eur Radiol 28:5121–5128
Zhao W, Xu Y, Yang Z et al (2019) Development and validation of a radiomics nomogram for identifying invasiveness of pulmonary adenocarcinomas appearing as subcentimeter ground-glass opacity nodules. Eur J Radiol 112:161–168
Wang S, Wang R, Zhang S et al (2018) 3D convolutional neural network for differentiating pre-invasive lesions from invasive adenocarcinomas appearing as ground-glass nodules with diameters </=3 cm using HRCT. Quant Imaging Med Surg 8:491–499
Yanagawa M, Niioka H, Hata A et al (2019) Application of deep learning (3-dimensional convolutional neural network) for the prediction of pathological invasiveness in lung adenocarcinoma: a preliminary study. Medicine (Baltimore) 98:e16119
Zhao W, Yang J, Sun Y et al (2018) 3D deep learning from CT scans predicts tumor invasiveness of subcentimeter pulmonary adenocarcinomas. Cancer Res 78:6881–6889
Austin PC, van Klaveren D, Vergouwe Y, Nieboer D, Lee DS, Steyerberg EW (2017) Validation of prediction models: examining temporal and geographic stability of baseline risk and estimated covariate effects. Diagn Progn Res 1:12
Moons KG, Altman DG, Reitsma JB et al (2015) Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med 162:W1–W73
Naidich DP, Bankier AA, MacMahon H et al (2013) Recommendations for the management of subsolid pulmonary nodules detected at CT: a statement from the Fleischner Society. Radiology 266:304–317
Dunnmon JA, Yi D, Langlotz CP, Re C, Rubin DL, Lungren MP (2019) Assessment of convolutional neural networks for automated classification of chest radiographs. Radiology 290:537–544
Rajpurkar P, Irvin J, Ball RL et al (2018) Deep learning for chest radiograph diagnosis: a retrospective comparison of the CheXNeXt algorithm to practicing radiologists. PLoS Med 15:e1002686
Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition 4700–4708
Bui TD, Shin J, Moon T (2017) 3D densely convolutional networks for volumetric segmentation. arXiv:170903199
Steyerberg EW, Vickers AJ, Cook NR et al (2010) Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 21:128–138
Rufibach K (2010) Use of Brier score to assess binary predictions. J Clin Epidemiol 63:938–939
DeLong ER, DeLong DM, Clarke-Pearson DL (1988) Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44:837–845
McNemar Q (1947) Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12:153–157
Leisenring W, Alonzo T, Pepe MS (2000) Comparisons of predictive values of binary medical diagnostic tests for paired designs. Biometrics 56:345–351
Walsh CG, Sharman K, Hripcsak G (2017) Beyond discrimination: a comparison of calibration methods and clinical usefulness of predictive models of readmission risk. J Biomed Inform 76:9–18
Tammemagi MC, Ten Haaf K, Toumazis I et al (2019) Development and validation of a multivariable lung cancer risk prediction model that includes low-dose computed tomography screening results: a secondary analysis of data from the National Lung Screening Trial. JAMA Netw Open 2:e190204
Chung JH, Choe G, Jheon S et al (2009) Epidermal growth factor receptor mutation and pathologic-radiologic correlation between multiple lung nodules with ground-glass opacity differentiates multicentric origin from intrapulmonary spread. J Thorac Oncol 4:1490–1495
Gillies RJ, Kinahan PE, Hricak H (2016) Radiomics: images are more than pictures, they are data. Radiology 278:563–577
Bankier AA, MacMahon H, Goo JM, Rubin GD, Schaefer-Prokop CM, Naidich DP (2017) Recommendations for measuring pulmonary nodules at CT: a statement from the Fleischner Society. Radiology 285:584–600
MacMahon H, Naidich DP, Goo JM et al (2017) Guidelines for management of incidental pulmonary nodules detected on CT images: from the Fleischner Society 2017. Radiology 284:228–243
Lee KH, Goo JM, Park SJ et al (2014) Correlation between the size of the solid component on thin-section CT and the invasive component on pathology in small lung adenocarcinomas manifesting as ground-glass nodules. J Thorac Oncol 9:74–82
Kim DW, Jang HY, Kim KW, Shin Y, Park SH (2019) Design characteristics of studies reporting the performance of artificial intelligence algorithms for diagnostic analysis of medical images: results from recently published papers. Korean J Radiol 20:405–410
Park SH, Han K (2018) Methodologic guide for evaluating clinical performance and effect of artificial intelligence technology for medical diagnosis and prediction. Radiology 286:800–809
Acknowledgments
We sincerely express our gratitude to Sunkyung Jeon, Jong Hyuk Lee, Su Yeon Ahn, Roh-Eul Yoo, Hyun-ju Lim, Juil Park, Woo Hyeon Lim, and Myunghee Lee for data acquisition. We are also grateful to Jung Hee Hong, Eui Jin Hwang, and Soon Ho Yoon for participating in the observer performance test.
Funding
This study was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF), funded by the Ministry of Science, ICT & Future Planning (grant number: 2017R1A2B4008517).
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
Guarantor
The scientific guarantor of this publication is Chang Min Park.
Conflict of interest
Activities related to the present article: none.
Activities not related to the present article: H.K., J.M.G., and C.M.P received research grants from Lunit Inc. (Seoul, South Korea).
Statistics and biometry
No complex statistical methods were necessary for this paper.
Informed consent
Written informed consent was waived by the Institutional Review Board.
Ethical approval
Institutional Review Board approval was obtained.
Study subjects or cohorts overlap
Some study subjects or cohorts have been previously reported in journal articles (Eur Radiol 2016 26:4465–4474; Eur J Radiol 2016 85:1174–1180; Eur Radiol 2017 27:3266–3274; Eur Radiol 2017 27:1369–1376; Eur Radiol 2018 28:2124–2133; Eur Radiol 2019 29:1674–1683; Eur Radiol 2019 29:1586–1594).
Methodology
• retrospective
• diagnostic or prognostic study
• performed at one institution
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
ESM 1
(DOCX 42 kb)
Rights and permissions
About this article
Cite this article
Kim, H., Lee, D., Cho, W.S. et al. CT-based deep learning model to differentiate invasive pulmonary adenocarcinomas appearing as subsolid nodules among surgical candidates: comparison of the diagnostic performance with a size-based logistic model and radiologists. Eur Radiol 30, 3295–3305 (2020). https://doi.org/10.1007/s00330-019-06628-4
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00330-019-06628-4