Abstract
Objectives
Post-imaging mathematical prediction models (MPMs) provide guidance for the management of solid pulmonary nodules by providing a lung cancer risk score from demographic and radiologists-indicated imaging characteristics. We hypothesized calibrating the MPM risk score threshold to a local study cohort would result in improved performance over the original recommended MPM thresholds. We compared the pre- and post-calibration performance of four MPM models and determined if improvement in MPM prediction occurs as nodules are imaged longitudinally.
Materials and methods
A common cohort of 317 individuals with computed tomography-detected, solid nodules (80 malignant, 237 benign) were used to evaluate the MPM performance. We created a web-based application for this study that allows others to easily calibrate thresholds and analyze the performance of MPMs on their local cohort. Thirty patients with repeated imaging were tested for improved performance longitudinally.
Results
Using calibrated thresholds, Mayo Clinic and Brock University (BU) MPMs performed the best (AUC = 0.63, 0.61) compared to the Veteran’s Affairs (0.51) and Peking University (0.55). Only BU had consensus with the original MPM threshold; the other calibrated thresholds improved MPM accuracy. No significant improvements in accuracy were found longitudinally between time points.
Conclusions
Calibration to a common cohort can select the best-performing MPM for your institution. Without calibration, BU has the most stable performance in solid nodules ≥ 8 mm but has only moderate potential to refine subjects into appropriate workup. Application of MPM is recommended only at initial evaluation as no increase in accuracy was achieved over time.
Key Points
• Post-imaging lung cancer risk mathematical predication models (MPMs) perform poorly on local populations without calibration.
• An application is provided to facilitate calibration to new study cohorts: the Mayo Clinic model, the U.S. Department of Veteran’s Affairs model, the Brock University model, and the Peking University model.
• No significant improvement in risk prediction occurred in nodules with repeated imaging sessions, indicating the potential value of risk prediction application is limited to the initial evaluation.
Similar content being viewed by others
Abbreviations
- AUC:
-
Area under the curve
- BTS:
-
British Thoracic Society
- BU:
-
Brock University model
- COPD:
-
Chronic obstructive pulmonary disorder
- CT:
-
Computed tomography
- MAD:
-
Median absolute deviation
- MC:
-
Mayo Clinic model
- MPMs :
-
Mathematical prediction models
- PR:
-
Precision recall
- PU:
-
Peking University model
- ROC:
-
Receiver-operator characteristic
- TP_1:
-
Initial imaging encounter on which the pulmonary nodule was identified
- TP_F:
-
Final imaging encounter before pulmonary nodule diagnosis
- VA:
-
U.S. Department of Veterans Affairs model
References
Siegel RL, Miller KD, Jemal A (2017) Cancer statistics, 2017. CA Cancer J Clin 67:7–30
MacMahon H, Naidich DP, Goo JM et al (2017) Guidelines for management of incidental pulmonary nodules detected on CT images: from the Fleischner Society 2017. Radiology 284:228–243
American College of Radiology (2014). Lung CT screening reporting and data system (Lung-RADS). Available via: https://www.acr.org/Clinical-Resources/Reporting-and-Data-Systems/Lung-Rads. Accessed 12/12/2018
Gould MK, Donington J, Lynch WR et al (2013) Evaluation of individuals with pulmonary nodules: when is it lung cancer? Diagnosis and management of lung cancer, 3rd ed: American College of Chest Physicians evidence-based clinical practice guidelines. Chest 143:e93S–e120S
Mehta HJ, Mohammed TL, Jantz MA (2017) The American College of Radiology Lung Imaging Reporting and Data System: potential drawbacks and need for revision. Chest 151:539–543
Pinsky PF, Gierada DS, Black W et al (2015) Performance of Lung-RADS in the National Lung Screening Trial: a retrospective assessment. Ann Intern Med 162:485–491
van Riel SJ, Ciompi F, Jacobs C et al (2017) Malignancy risk estimation of screen-detected nodules at baseline CT: comparison of the PanCan model, Lung-RADS and NCCN guidelines. Eur Radiol 27:4019–4029
Gray EP, Teare MD, Stevens J, Archer R (2016) Risk prediction models for lung cancer: a systematic review. Clin Lung Cancer 17:95–106
Swensen SJ, Silverstein MD, Ilstrup DM, Schleck CD, Edell ES (1997) The probability of malignancy in solitary pulmonary nodules. Application to small radiologically indeterminate nodules. Arch Intern Med 157:849–855
Gould MK, Ananth L, Barnett PG, Veterans Affairs SNAP Cooperative Study Group (2007) A clinical model to estimate the pretest probability of lung cancer in patients with solitary pulmonary nodules. Chest 131:383–388
McWilliams A, Tammemagi MC, Mayo JR et al (2013) Probability of cancer in pulmonary nodules detected on first screening CT. N Engl J Med 369:910–919
Li Y, Wang J (2012) A mathematical model for predicting malignancy of solitary pulmonary nodules. World J Surg 36:830–835
Herder GJ, van Tinteren H, Golding RP et al (2005) Clinical prediction model to characterize pulmonary nodules: validation and added value of 18F-fluorodeoxyglucose positron emission tomography. Chest 128:2490–2496
Baldwin DR, Callister ME (2015) The British Thoracic Society guidelines on the investigation and management of pulmonary nodules. Thorax 70:794–798
Hammer MM, Nachiappan AC, Barbosa EJM Jr (2018) Limited utility of pulmonary nodule risk calculators for managing large nodules. Curr Probl Diagn Radiol 47:23–27
Al-Ameri A, Malhotra P, Thygesen H et al (2015) Risk of malignancy in pulmonary nodules: a validation study of four prediction models. Lung Cancer 89:27–30
Mehta HJ, Ravenel JG, Shaftman SR et al (2014) The utility of nodule volume in the context of malignancy prediction for small pulmonary nodules. Chest 145:464–472
Perandini S, Soardi GA, Motton M, Montemezzi S (2015) Critique of Al-Ameri et al. (2015) - risk of malignancy in pulmonary nodules: a validation study of four prediction models. Lung Cancer 90:118–119
Regan EA, Hokanson JE, Murphy JR et al (2010) Genetic epidemiology of COPD (COPDGene) study design. COPD 7:32–43
Schwartz AG, Lusk CM, Wenzlaff AS et al (2016) Risk of lung cancer associated with COPD phenotype based on quantitative image analysis. Cancer Epidemiol Biomarkers Prevv 25:1341–1347
Swensen SJ, Silverstein MD, Edell ES et al (1999) Solitary pulmonary nodules: clinical prediction model versus physicians. Mayo Clin Proc 74:319–329
Cummings SR, Lillington GA, Richard RJ (1986) Managing solitary pulmonary nodules. The choice of strategy is a “close call”. Am Rev Respir Dis 134:453–460
Chung K, Mets OM, Gerke PK et al (2018) Brock malignancy risk calculator for pulmonary nodules: validation outside a lung cancer screening population. Thorax 73:857–863
Youden WJ (1950) Index for rating diagnostic tests. Cancer 3:32–35
Jesse D, Goadrich M (2006) The relationship between precision-recall and ROC curves. Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, 25–29 June 2006, pp 233–240
McNemar Q (1947) Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12:153–157
Callister ME, Baldwin DR, Akram AR et al (2015) British Thoracic Society guidelines for the investigation and management of pulmonary nodules. Thorax 70(Suppl 2):ii1–ii54
McNitt-Gray MF, Kim GH, Zhao B et al (2015) Determining the variability of lesion size measurements from CT patient data sets acquired under “no change” conditions. Transl Oncol 8:55–64
Lin H, Huang C, Wang W, Luo J, Yang X, Liu Y (2017) Measuring interobserver disagreement in rating diagnostic characteristics of pulmonary nodule using the lung imaging database consortium and image database resource initiative. Acad Radiol 24:401–410
Maiga AW, Deppen SA, Massion PP et al (2018) Communication about the probability of cancer in indeterminate pulmonary nodules. JAMA Surg 153:353–357
Dilger SK, Uthoff J, Judisch A et al (2015) Improved pulmonary nodule classification utilizing quantitative lung parenchyma features. J Med Imaging (Bellingham) 2:041004
Dhara AK, Mukhopadhyay S, Dutta A, Garg M, Khandelwal N (2016) A combination of shape and texture features for classification of pulmonary nodules in lung CT images. J Digit Imaging 29:466–475
Ferreira JR Jr, Oliveira MC, de Azevedo-Marques PM (2017) Characterization of pulmonary nodules based on features of margin sharpness and texture. J Digit Imaging 31:451–463
Nibali A, He Z, Wollersheim D (2017) Pulmonary nodule classification with deep residual networks. Int J Comput Assist Radiol Surg 12:1799–1808
Sun W, Zheng B, Qian W (2017) Automatic feature learning using multichannel ROI based on deep structured algorithms for computerized lung cancer diagnosis. Comput Biol Med 89:530–539
Way TW, Sahiner B, Chan HP et al (2009) Computer-aided diagnosis of pulmonary nodules on CT scans: improvement of classification performance with nodule surface features. Med Phys 36:3086–3098
Zhu Y, Tan Y, Hua Y, Wang M, Zhang G, Zhang J (2010) Feature selection and performance evaluation of support vector machine (SVM)-based classifier for differentiating benign and malignant pulmonary nodules by computed tomography. J Digit Imaging 23:51–65
Acknowledgements
We thank Kimberly Sprenger, Debra O’Connel-Moore, Mark Escher, Patrick Thalken, and Kimberly Schroeder for technical assistance.
Funding
This work was supported in part by Grant IRG-77-004-34 from the American Cancer Society, administered through the Holden Comprehensive Cancer Center at the University of Iowa. The COPDGene Study was supported by NHLBI U01 HL089897 and U01 HL089856. The COPDGene study (NCT00608764) is also supported by the COPD Foundation through contributions made to an Industry Advisory Committee comprised of AstraZeneca, Boehringer-Ingelheim, GlaxoSmithKline, Novartis, Pfizer, Siemens, and Sunovion. The INHALE study was supported by Award Number R01CA141769 and P30CA022453 from the National Cancer Institute, Health and Human Services Award HHSN26120130011I, and the Herrick Foundation.
Author information
Authors and Affiliations
Consortia
Corresponding author
Ethics declarations
Guarantor
The scientific guarantor of this publication is Dr. Jessica C. Sieren.
Conflict of interest
The authors declare that they have no conflict of interest.
Statistics and biometry
The first and last authors, as biomedical engineers, have experience with biostatistics methods. No complex statistical methods were necessary for this paper.
Informed consent
The University of Iowa Institutional Review Board has approved this study (IRB 201603824). Informed consent was obtained from the research cohort participants through the parent studies, COPDgene and INHALE (including the approval of collected data for expanded research questions beyond the parent study purpose). For the retrospective clinical cohort, written informed consent was waived by the Institutional Review Board.
Ethical approval
Institutional Review Board approval was obtained.
Study subjects or cohorts overlap
Forty subjects from study subjects or cohorts have been previously reported by our lab in a machine learning approach development [19, 20, 31].
Methodology
• Retrospective
• observational
• performed at one institution
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
ESM 1
(DOCX 225 kb)
Rights and permissions
About this article
Cite this article
Uthoff, J., Koehn, N., Larson, J. et al. Post-imaging pulmonary nodule mathematical prediction models: are they clinically relevant?. Eur Radiol 29, 5367–5377 (2019). https://doi.org/10.1007/s00330-019-06168-x
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00330-019-06168-x