Skip to main content
SpringerLink
Log in

Analysis of Validation Performance of a Machine Learning Classifier in Interstitial Lung Disease Cases Without Definite or Probable Usual Interstitial Pneumonia Pattern on CT Using Clinical and Pathology-Supported Diagnostic Labels

  • Published:
Journal of Imaging Informatics in Medicine Aims and scope Submit manuscript

Abstract

We previously validated Fibresolve, a machine learning classifier system that non-invasively predicts idiopathic pulmonary fibrosis (IPF) diagnosis. The system incorporates an automated deep learning algorithm that analyzes chest computed tomography (CT) imaging to assess for features associated with idiopathic pulmonary fibrosis. Here, we assess performance in assessment of patterns beyond those that are characteristic features of usual interstitial pneumonia (UIP) pattern. The machine learning classifier was previously developed and validated using standard training, validation, and test sets, with clinical plus pathologically determined ground truth. The multi-site 295-patient validation dataset was used for focused subgroup analysis in this investigation to evaluate the classifier’s performance range in cases with and without radiologic UIP and probable UIP designations. Radiologic assessment of specific features for UIP including the presence and distribution of reticulation, ground glass, bronchiectasis, and honeycombing was used for assignment of radiologic pattern. Output from the classifier was assessed within various UIP subgroups. The machine learning classifier was able to classify cases not meeting the criteria for UIP or probable UIP as IPF with estimated sensitivity of 56–65% and estimated specificity of 92–94%. Example cases demonstrated non-basilar-predominant as well as ground glass patterns that were indeterminate for UIP by subjective imaging criteria but for which the classifier system was able to correctly identify the case as IPF as confirmed by multidisciplinary discussion generally inclusive of histopathology. The machine learning classifier Fibresolve may be helpful in the diagnosis of IPF in cases without radiological UIP and probable UIP patterns.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data Availability

The data analyzed during the study were provided by a third party. Requests for data should be directed to the provider indicated in the “Acknowledgements.”

References

  1. Jeganathan N, Sathananthan M. The prevalence and burden of interstitial lung diseases in the USA. ERJ Open Res [Internet]. 2022 Jan 1 [cited 2023 Jan 25];8(1). Available from: https://openres.ersjournals.com/content/8/1/00630-2021

  2. Soffer S, Morgenthau AS, Shimon O, Barash Y, Konen E, Glicksberg BS, Klang E. Artificial Intelligence for Interstitial Lung Disease Analysis on Chest Computed Tomography: A Systematic Review. Acad Radiol. 2022 Feb;29 Suppl 2:S226–35.

    Article  PubMed  Google Scholar 

  3. Raghu G, Remy-Jardin M, Myers JL, Richeldi L, Ryerson CJ, Lederer DJ, Behr J, Cottin V, Danoff SK, Morell F, Flaherty KR, Wells A, Martinez FJ, Azuma A, Bice TJ, Bouros D, Brown KK, Collard HR, Duggal A, Galvin L, Inoue Y, Jenkins RG, Johkoh T, Kazerooni EA, Kitaichi M, Knight SL, Mansour G, Nicholson AG, Pipavath SNJ, Buendía-Roldán I, Selman M, Travis WD, Walsh S, Wilson KC, American Thoracic Society, European Respiratory Society, Japanese Respiratory Society, and Latin American Thoracic Society. Diagnosis of Idiopathic Pulmonary Fibrosis. An Official ATS/ERS/JRS/ALAT Clinical Practice Guideline. Am J Respir Crit Care Med. 2018 Sep 1;198(5):e44–68.

  4. Hutchinson JP, Fogarty AW, McKeever TM, Hubbard RB. In-Hospital Mortality after Surgical Lung Biopsy for Interstitial Lung Disease in the United States. 2000 to 2011. Am J Respir Crit Care Med. 2016 May 15;193(10):1161–7.

  5. Walsh SLF, Calandriello L, Sverzellati N, Wells AU, Hansell DM, UIP Observer Consort. Interobserver agreement for the ATS/ERS/JRS/ALAT criteria for a UIP pattern on CT. Thorax. 2016 Jan;71(1):45–51.

  6. Delayed access and survival in idiopathic pulmonary fibrosis: a cohort study - PubMed [Internet]. [cited 2023 Jan 25]. Available from: https://pubmed.ncbi.nlm.nih.gov/21719755/

  7. Machine Learning for Pulmonary and Critical Care Medicine: A Narrative Review - PubMed [Internet]. [cited 2023 Jan 25]. Available from: https://pubmed.ncbi.nlm.nih.gov/32048244/

  8. Saba L, Biswas M, Kuppili V, Cuadrado Godia E, Suri HS, Edla DR, Omerzu T, Laird JR, Khanna NN, Mavrogeni S, Protogerou A, Sfikakis PP, Viswanathan V, Kitas GD, Nicolaides A, Gupta A, Suri JS. The present and future of deep learning in radiology. Eur J Radiol. 2019 May 1;114:14–24.

    Article  Google Scholar 

  9. Shaish H, Ahmed FS, Lederer D, D’Souza B, Armenta P, Salvatore M, Saqi A, Huang S, Jambawalikar S, Mutasa S. Deep Learning of Computed Tomography Virtual Wedge Resection for Prediction of Histologic Usual Interstitial Pneumonitis. Ann Am Thorac Soc. 2021 Jan;18(1):51–9.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Christe A, Peters AA, Drakopoulos D, Heverhagen JT, Geiser T, Stathopoulou T, Christodoulidis S, Anthimopoulos M, Mougiakakou SG, Ebner L. Computer-Aided Diagnosis of Pulmonary Fibrosis Using Deep Learning and CT Images. Invest Radiol. 2019 Oct;54(10):627–32.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Walsh SLF, Mackintosh JA, Calandriello L, Silva M, Sverzellati N, Larici AR, Humphries SM, Lynch DA, Jo HE, Glaspole I, Grainge C, Goh N, Hopkins PMA, Moodley Y, Reynolds PN, Zappala C, Keir G, Cooper WA, Mahar AM, Ellis S, Wells AU, Corte TJ. Deep Learning-based Outcome Prediction in Progressive Fibrotic Lung Disease Using High-Resolution Computed Tomography. Am J Respir Crit Care Med. 2022 Oct 1;206(7):883–91.

    Article  PubMed  Google Scholar 

  12. Aboeleneen AE, Patel MK, Al-maadeed S. Pulmonary Fibrosis Progression Prediction Using Image Processing and Machine Learning. In: Alja’am J, Al-Maadeed S, Halabi O, editors. Emerging Technologies in Biomedical Engineering and Sustainable TeleMedicine [Internet]. Cham: Springer International Publishing; 2021 [cited 2022 Dec 26]. p. 159–77. (Advances in Science, Technology & Innovation). Available from: https://doi.org/10.1007/978-3-030-14647-4_11

  13. Mandal S, Balas VE, Shaw RN, Ghosh A. Prediction Analysis of Idiopathic Pulmonary Fibrosis Progression from OSIC Dataset. In: 2020 IEEE International Conference on Computing, Power and Communication Technologies (GUCON). 2020. p. 861–5.

  14. Liu Q, Sun D, Wang Y, Li P, Jiang T, Dai L, Duo M, Wu R, Cheng Z. Use of machine learning models to predict prognosis of combined pulmonary fibrosis and emphysema in a Chinese population. BMC Pulm Med. 2022 Aug 29;22(1):327.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  15. Walsh SLF, Calandriello L, Silva M, Sverzellati N. Deep learning for classifying fibrotic lung disease on high-resolution computed tomography: a case-cohort study. Lancet Respir Med. 2018 Nov 1;6(11):837–45.

    Article  PubMed  Google Scholar 

  16. Khemasuwan D, Sorensen JS, Colt HG. Artificial intelligence in pulmonary medicine: computer vision, predictive model and COVID-19. Eur Respir Rev [Internet]. 2020 Sep 30 [cited 2022 Dec 26];29(157). Available from: https://err.ersjournals.com/content/29/157/200181

  17. Koo CW, Larson NB, Parris-Skeete CT, Karwoski RA, Kalra S, Bartholmai BJ, Carmona EM. Prospective machine learning CT quantitative evaluation of idiopathic pulmonary fibrosis in patients undergoing anti-fibrotic treatment using low- and ultra-low-dose CT. Clin Radiol. 2022 Mar 1;77(3):e208–14.

    Article  CAS  PubMed  Google Scholar 

  18. Subramanian DR, Rubi DRD, Dedeepya M, Gorugantu SKVM. Quantitative Progression analysis of Post-Acute Sequelae of COVID-19, Pulmonary Fibrosis (PASC-PF) and Artificial Intelligence driven CT scoring of Lung involvement in Covid-19 infection using HRCT-Chest images. Med Res Arch [Internet]. 2022 Oct 31 [cited 2022 Dec 27];10(10). Available from: https://esmed.org/MRA/mra/article/view/3145

  19. Handa T, Tanizawa K, Oguma T, Uozumi R, Watanabe K, Tanabe N, Niwamoto T, Shima H, Mori R, Nobashi TW, Sakamoto R, Kubo T, Kurosaki A, Kishi K, Nakamoto Y, Hirai T. Novel Artificial Intelligence-based Technology for Chest Computed Tomography Analysis of Idiopathic Pulmonary Fibrosis. Ann Am Thorac Soc. 2022 Mar;19(3):399–406.

    Article  PubMed  Google Scholar 

  20. Shi Y, Wong WK, Goldin JG, Brown MS, Kim GHJ. Prediction of progression in idiopathic pulmonary fibrosis using CT scans at baseline: A quantum particle swarm optimization - Random forest approach. Artif Intell Med. 2019 Sep 1;100:101709.

    Article  Google Scholar 

  21. Milam ME, Koo CW. The current status and future of FDA-approved artificial intelligence tools in chest radiology in the United States. Clin Radiol [Internet]. 2022 Sep 28 [cited 2022 Dec 26]; Available from: https://www.sciencedirect.com/science/article/pii/S0009926022005141

  22. Yousef Ahmad, DO, MSc, Angad Kalra, MS, Joshua Mooney, MD, Isabel E Allen, PhD, Julia Seaman, PhD, Michael Muelly, MD, Joshua J. Reicher, MD. A Machine Learning System to Improve Non-Invasive Diagnosis of Idiopathic Pulmonary Fibrosis. 2023 Am Thorac Soc Conf Present.

  23. Maddali MV, Kalra A, Muelly M, Reicher JJ (2023) Development and validation of a CT-based deep learning algorithm to augment non-invasive diagnosis of idiopathic pulmonary fibrosis. Respir Med 219:107428. https://doi.org/10.1016/j.rmed.2023.107428

  24. Manoj V. Maddali, MD, Angad Kalra, MS, Michael Muelly, MD, Joshua J. Reicher, MD. Development and Validation of a CT-based Deep Learning Algorithm to Augment Non-invasive Diagnosis of Idiopathic Pulmonary Fibrosis. 2023 Am Thorac Soc Conf Present.

  25. The Role of Surgical Lung Biopsy in the Diagnosis of Fibrotic Interstitial Lung Disease: Perspective from the Pulmonary Fibrosis Foundation - PubMed [Internet]. [cited 2023 Jan 25]. Available from: https://pubmed.ncbi.nlm.nih.gov/34004127/

  26. Smith ML, Hariri LP, Mino-Kenudson M, Dacic S, Attanoos R, Borczuk A, Colby TV, Cooper W, Jones KD, Leslie KO, Mahar A, Larsen BT, Cavazza A, Fukuoka J, Roden AC, Sholl LM, Tazelaar HD, Churg A, Beasley MB. Histopathologic Assessment of Suspected Idiopathic Pulmonary Fibrosis: Where We Are and Where We Need to Go. Arch Pathol Lab Med. 2020 Dec 1;144(12):1477–89.

    Article  PubMed  Google Scholar 

  27. Aziz ZA, Wells AU, Hansell DM, Bain GA, Copley SJ, Desai SR, Ellis SM, Gleeson FV, Grubnic S, Nicholson AG, Padley SPG, Pointon KS, Reynolds JH, Robertson RJH, Rubens MB. HRCT diagnosis of diffuse parenchymal lung disease: inter-observer variation. Thorax. 2004 Jun;59(6):506–11.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  28. Duffy G, Clarke SL, Christensen M, He B, Yuan N, Cheng S, Ouyang D. Confounders mediate AI prediction of demographics in medical imaging. Npj Digit Med. 2022 Dec 22;5(1):1–6.

    Article  Google Scholar 

  29. Jacob J, Bartholmai BJ, Rajagopalan S, van Moorsel CHM, van Es HW, van Beek FT, Struik MHL, Kokosi M, Egashira R, Brun AL, Nair A, Walsh SLF, Cross G, Barnett J, de Lauretis A, Judge EP, Desai S, Karwoski R, Ourselin S, Renzoni E, Maher TM, Altmann A, Wells AU. Predicting Outcomes in Idiopathic Pulmonary Fibrosis Using Automated Computed Tomographic Analysis. Am J Respir Crit Care Med. 2018 Sep 15;198(6):767–76.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Trovato G, Russo M. Artificial Intelligence (AI) and Lung Ultrasound in Infectious Pulmonary Disease. Front Med [Internet]. 2021 [cited 2023 Jan 25];8. Available from: https://www.frontiersin.org/articles/https://doi.org/10.3389/fmed.2021.706794

  31. Savadjiev P, Gallix B, Rezanejad M, Bhatnagar S, Semionov A, Siddiqi K, Forghani R, Reinhold C, Eidelman DH, Dandurand RJ. Improved Detection of Chronic Obstructive Pulmonary Disease at Chest CT Using the Mean Curvature of Isophotes. Radiol Artif Intell. 2022 Jan;4(1):e210105.

    Article  PubMed  Google Scholar 

  32. Walsh SLF, Wells AU, Desai SR, Poletti V, Piciucchi S, Dubini A, Nunes H, Valeyre D, Brillet PY, Kambouchner M, Morais A, Pereira JM, Moura CS, Grutters JC, van den Heuvel DA, van Es HW, van Oosterhout MF, Seldenrijk CA, Bendstrup E, Rasmussen F, Madsen LB, Gooptu B, Pomplun S, Taniguchi H, Fukuoka J, Johkoh T, Nicholson AG, Sayer C, Edmunds L, Jacob J, Kokosi MA, Myers JL, Flaherty KR, Hansell DM. Multicentre evaluation of multidisciplinary team meeting agreement on diagnosis in diffuse parenchymal lung disease: a case-cohort study. Lancet Respir Med. 2016 Jul;4(7):557–65.

    Article  PubMed  Google Scholar 

  33. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, Lijmer JG, Moher D, Rennie D, de Vet HCW, Standards for Reporting of Diagnostic Accuracy. Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. Standards for Reporting of Diagnostic Accuracy. Clin Chem. 2003 Jan;49(1):1–6.

  34. Depeursinge A, Vargas A, Platon A, Geissbuhler A, Poletti PA, Muller H. Building a reference multimedia database for interstitial lung disease. Computerized Medical Imaging and Graphics. 36(3):227-238, 2012.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

Software validation and performance analyses included public and non-public data sources. Public data sources included data provided by the Lung Tissue Research Consortium (LTRC) supported by the National Heart, Lung, and Blood Institute (NHLBI); multimedia database of interstitial lung diseases [34]; and the Open Source Imaging Consortium (OSIC). We thank Diego Ardila for his contributions to important concepts behind the science and machine learning engineering of the software system.

Funding

This study was funded by Imvaria Inc.

Author information

Authors and Affiliations

  1. Stanford School of Medicine, 291 Campus Drive, Stanford, CA, USA

    Marcello Chang

  2. Imvaria Inc, Berkeley, CA, USA

    Joshua J. Reicher & Michael Muelly

  3. Imvaria Inc, Vancouver, BC, Canada

    Angad Kalra

  4. Department of Pulmonary and Critical Care, University of Cincinnati Medical Center, Cincinnati, USA

    Yousef Ahmad

Authors
  1. Marcello Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Joshua J. Reicher
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Angad Kalra
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michael Muelly
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yousef Ahmad
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

M.C., co-author and drafting and revision of the manuscript. J.J.R., primary investigator and conception and design of the work and drafting and final approval of the manuscript. A.K., co-author and design of the work and acquisition or analysis of data. M.M., co-author and design of the work and acquisition or analysis of data. Y.A., co-author and drafting and revision of the manuscript.

Corresponding author

Correspondence to Joshua J. Reicher.

Ethics declarations

Ethics Approval

All datasets were acquired via 3rd parties under the IRB.

Competing Interests

Dr. Reicher, Dr. Muelly, and Mr. Kalra have a financial interest in Imvaria.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Highlights

• Key Points The Fibresolve system demonstrated consistent and clinically meaningful IPF diagnostic performance using CT in ILD cases without definite or probable usual interstitial pneumonia (UIP) pattern that would otherwise warrant invasive diagnostic tests, through incorporation of clinical and pathologic data into supervised model training labels.

• Summary The Fibresolve AI system maintained specificity in IPF diagnosis across different radiological patterns while improving sensitivity, suggesting it may reduce morbidity, mortality, and time delays from invasive diagnostic testing.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chang, M., Reicher, J.J., Kalra, A. et al. Analysis of Validation Performance of a Machine Learning Classifier in Interstitial Lung Disease Cases Without Definite or Probable Usual Interstitial Pneumonia Pattern on CT Using Clinical and Pathology-Supported Diagnostic Labels. J Digit Imaging. Inform. med. 37, 297–307 (2024). https://doi.org/10.1007/s10278-023-00914-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10278-023-00914-w

Keywords

Search

Navigation