Journal of Digital Imaging

, Volume 29, Issue 4, pp 476–487 | Cite as

A Comparison of Lung Nodule Segmentation Algorithms: Methods and Results from a Multi-institutional Study

  • Jayashree Kalpathy-Cramer
  • Binsheng Zhao
  • Dmitry Goldgof
  • Yuhua Gu
  • Xingwei Wang
  • Hao Yang
  • Yongqiang Tan
  • Robert Gillies
  • Sandy NapelEmail author


Tumor volume estimation, as well as accurate and reproducible borders segmentation in medical images, are important in the diagnosis, staging, and assessment of response to cancer therapy. The goal of this study was to demonstrate the feasibility of a multi-institutional effort to assess the repeatability and reproducibility of nodule borders and volume estimate bias of computerized segmentation algorithms in CT images of lung cancer, and to provide results from such a study. The dataset used for this evaluation consisted of 52 tumors in 41 CT volumes (40 patient datasets and 1 dataset containing scans of 12 phantom nodules of known volume) from five collections available in The Cancer Imaging Archive. Three academic institutions developing lung nodule segmentation algorithms submitted results for three repeat runs for each of the nodules. We compared the performance of lung nodule segmentation algorithms by assessing several measurements of spatial overlap and volume measurement. Nodule sizes varied from 29 μl to 66 ml and demonstrated a diversity of shapes. Agreement in spatial overlap of segmentations was significantly higher for multiple runs of the same algorithm than between segmentations generated by different algorithms (p < 0.05) and was significantly higher on the phantom dataset compared to the other datasets (p < 0.05). Algorithms differed significantly in the bias of the measured volumes of the phantom nodules (p < 0.05) underscoring the need for assessing performance on clinical data in addition to phantoms. Algorithms that most accurately estimated nodule volumes were not the most repeatable, emphasizing the need to evaluate both their accuracy and precision. There were considerable differences between algorithms, especially in a subset of heterogeneous nodules, underscoring the recommendation that the same software be used at all time points in longitudinal studies.


Segmentation Infrastructure Lung cancer Computed tomography Quantitative imaging 



U.S. Department of Health and Human Services, National Institutes of Health, National Cancer Institute (R01 CA160251), (R01 CA149490), (U01 CA140207), (U01 CA143062), (U01 CA154601), (U24 CA180927) and (U24 CA180918).


  1. 1.
    Torre LA, Bray F, Siegel RL, Ferlay J, Lortet-Tieulent J, Jemal A: Global cancer statistics, 2012. CA Cancer J Clin 65(2):87–108, 2015. doi: 10.3322/caac.21262 CrossRefPubMedGoogle Scholar
  2. 2.
    Ravenel JG: Evidence-based imaging in lung cancer: a systematic review. J Thorac Imaging 27(5):315–324, 2012. doi: 10.1097/RTI.0b013e318254a198 CrossRefPubMedGoogle Scholar
  3. 3.
    Rivera MP, Mehta AC, Wahidi MM: Establishing the diagnosis of lung cancer: Diagnosis and management of lung cancer, 3rd ed: American College of Chest Physicians evidence-based clinical practice guidelines. Chest 143(5 Suppl):e142S–e165S, 2013. doi: 10.1378/chest.12-2353 CrossRefPubMedGoogle Scholar
  4. 4.
    Nair A, Hansell DM: European and North American lung cancer screening experience and implications for pulmonary nodule management. Eur Radiol 21(12):2445–2454, 2011. doi: 10.1007/s00330-011-2219-y CrossRefPubMedGoogle Scholar
  5. 5.
    National Lung Screening Trial Research Team, Aberle DR, Adams AM, Berg CD, Black WC, Clapp JD, Fagerstrom RM, Gareen IF, Gatsonis C, Marcus PM, Sicks JD: Reduced lung-cancer mortality with low-dose computed tomographic screening. N Engl J Med 365(5):395–409, 2011. doi: 10.1056/NEJMoa1102873 CrossRefGoogle Scholar
  6. 6.
    National Lung Screening Trial Research Team, Church TR, Black WC, Aberle DR, Berg CD, Clingan KL, Duan F, Fagerstrom RM, Gareen IF, Gierada DS, Jones GC, Mahon I, Marcus PM, Sicks JD, Jain A, Baum S: Results of initial low-dose computed tomographic screening for lung cancer. N Engl J Med 368(21):1980–1991, 2013. doi: 10.1056/NEJMoa1209120 CrossRefGoogle Scholar
  7. 7.
    MacMahon H, Austin JH, Gamsu G, Herold CJ, Jett JR, Naidich DP, Patz Jr, EF, Swensen SJ, Fleischner S: Guidelines for management of small pulmonary nodules detected on CT scans: a statement from the Fleischner Society. Radiology 237(2):395–400, 2005. doi: 10.1148/radiol.2372041887 CrossRefPubMedGoogle Scholar
  8. 8.
    Revel MP: Avoiding overdiagnosis in lung cancer screening: the volume doubling time strategy. Eur Respir J 42(6):1459–1463, 2013. doi: 10.1183/09031936.00157713 CrossRefPubMedGoogle Scholar
  9. 9.
    Patel VK, Naik SK, Naidich DP, Travis WD, Weingarten JA, Lazzaro R, Gutterman DD, Wentowski C, Grosu HB, Raoof S: A practical algorithmic approach to the diagnosis and management of solitary pulmonary nodules: part 2: pretest probability and algorithm. Chest 143(3):840–846, 2013. doi: 10.1378/chest.12-1487 CrossRefPubMedGoogle Scholar
  10. 10.
    Infante M, Berghmans T, Heuvelmans MA, Hillerdal G, Oudkerk M: Slow-growing lung cancer as an emerging entity: from screening to clinical management. Eur Respir J 42(6):1706–1722, 2013. doi: 10.1183/09031936.00186212 CrossRefPubMedGoogle Scholar
  11. 11.
    Eisenhauer EA, Therasse P, Bogaerts J, Schwartz LH, Sargent D, Ford R, Dancey J, Arbuck S, Gwyther S, Mooney M, Rubinstein L, Shankar L, Dodd L, Kaplan R, Lacombe D, Verweij J: New response evaluation criteria in solid tumours: Revised RECIST guideline (version 1.1). 45(2):228–247, 2009. doi: 10.1016/j.ejca.2008.10.026
  12. 12.
    Revel MP, Bissery A, Bienvenu M, Aycard L, Lefort C, Frija G: Are two-dimensional CT measurements of small noncalcified pulmonary nodules reliable? Radiology 231(2):453–458, 2004. doi: 10.1148/radiol.2312030167 CrossRefPubMedGoogle Scholar
  13. 13.
    Reeves AP, Biancardi AM, Apanasovich TV, Meyer CR, MacMahon H, van Beek EJ, Kazerooni EA, Yankelevitz D, McNitt-Gray MF, McLennan G, Armato 3rd, SG, Henschke CI, Aberle DR, Croft BY, Clarke LP: The Lung Image Database Consortium (LIDC): a comparison of different size metrics for pulmonary nodule measurements. Acad Radiol 14(12):1475–1485, 2007. doi: 10.1016/j.acra.2007.09.005 CrossRefPubMedPubMedCentralGoogle Scholar
  14. 14.
    Marten K, Auer F, Schmidt S, Kohl G, Rummeny EJ: Inadequacy of manual measurements compared to automated CT volumetry in assessment of treatment response of pulmonary metastases using RECIST criteria - Springer. European. 2006Google Scholar
  15. 15.
    Zhao YR, Ooijen PMv, Dorrius MD, Heuvelmans M, de Bock GH, Vliegenthart R, Oudkerk M: Comparison of three software systems for semi-automatic volumetry of pulmonary nodules on baseline and follow-up CT examinations. Acta radiologica (Stockholm, Sweden : 1987). 2013. doi: 10.1177/0284185113508177
  16. 16.
    Ashraf H, de Hoop B, Shaker SB, Dirksen A, Bach KS, Hansen H, Prokop M, Pedersen JH: Lung nodule volumetry: segmentation algorithms within the same software package cannot be used interchangeably. Eur Radiol 20(8):1878–1885, 2010. doi: 10.1007/s00330-010-1749-z CrossRefPubMedPubMedCentralGoogle Scholar
  17. 17.
    Kalpathy-Cramer J, Fuller CD: Target Contour Testing/Instructional Computer Software (TaCTICS): a novel training and evaluation platform for radiotherapy target delineation.2010:361–365, 2010Google Scholar
  18. 18.
    Kalpathy-Cramer J, Bedrick SD, Boccia K, Fuller CD: A pilot prospective feasibility study of organ-at-risk definition using Target Contour Testing/Instructional Computer Software (TaCTICS), a training and evaluation platform for radiotherapy target delineation.2011:654–663,2011Google Scholar
  19. 19.
    Kalpathy-Cramer J, Awan M, Bedrick S, Rasch CR, Rosenthal DI, Fuller CD: Development of a software for quantitative evaluation radiotherapy target and organ-at-risk segmentation comparison. J Digit Imaging 27(1):108–119, 2014. doi: 10.1007/s10278-013-9633-4 CrossRefPubMedGoogle Scholar
  20. 20.
    Kalpathy-Cramer J, Napel S, Goldgof D, Zhao B: QIN multi-site collection of Lung CT data with Nodule Segmentations [cited 2015]. Available from: doi: 10.7937/K9/TCIA.2015.1BUVFJR7
  21. 21.
    Zhao B, James LP, Moskowitz CS, Guo P, Ginsberg MS, Lefkowitz RA, Qin Y, Riely GJ, Kris MG, Schwartz LH: Evaluating variability in tumor measurements from same-day repeat CT scans of patients with non-small cell lung cancer. Radiology 252(1):263–272, 2009. doi: 10.1148/radiol.2522081593 CrossRefPubMedPubMedCentralGoogle Scholar
  22. 22.
    Clarke LP, Croft BY, Staab E, Baker H, Sullivan DC: National Cancer Institute initiative: Lung image database resource for imaging research. Acad Radiol 8(5):447–450, 2001. doi: 10.1016/S1076-6332(03)80555-X CrossRefPubMedGoogle Scholar
  23. 23.
    Turner WD, Kelliher TP, Ross JC, Miller JV: An analysis of early studies released by the Lung Imaging Database Consortium (LIDC). Med Image Comput Comput Assist Interv 9(Pt 2):487–494, 2006PubMedGoogle Scholar
  24. 24.
    Armato III, SG, McNitt-Gray MF, Reeves AP, Meyer CR, McLennan G, Aberle DR, Kazerooni EA, MacMahon H, van Beek EJ, Yankelevitz D, Hoffman EA, Henschke CI, Roberts RY, Brown MS, Engelmann RM, Pais RC, Piker CW, Qing D, Kocherginsky M, Croft BY, Clarke LP: The Lung Image Database Consortium (LIDC): an evaluation of radiologist variability in the identification of lung nodules on CT scans. Acad Radiol 14(11):1409–1421, 2007. doi: 10.1016/j.acra.2007.07.008 CrossRefPubMedPubMedCentralGoogle Scholar
  25. 25.
    Gevaert O, Xu J, Hoang CD, Leung AN, Xu Y, Quon A, Rubin DL, Napel S, Plevritis SK: Non-small cell lung cancer: identifying prognostic imaging biomarkers by leveraging public gene expression microarray data—methods and preliminary results. Radiology 264(2):387–396, 2012. doi: 10.1148/radiol.12111607 CrossRefPubMedPubMedCentralGoogle Scholar
  26. 26.
    Zhao B, Tan Y, Tsai WY, Schwartz LH, Lu L: Exploring variability in CT characterization of tumors: a preliminary phantom study. Transl Oncol 7(1):88–93, 2014CrossRefPubMedPubMedCentralGoogle Scholar
  27. 27.
    Gu Y, Kumar V, Hall LO, Goldgof DB, Li C-Y, Korn R, Bendtsen C, Velazquez ER, Dekker A, Aerts H, Lambin P, Li X, Tian J, Gatenby RA, Gillies RJ: Automated delineation of lung tumors from CT images using a single click ensemble segmentation approach. Pattern Recogn 46(3):692–702, 2013. doi: 10.1016/j.patcog.2012.10.005 CrossRefGoogle Scholar
  28. 28.
    Tan Y, Schwartz LH, Zhao B: Segmentation of lung lesions on CT scans using watershed, active contours, and Markov random field. Med Phys 40(4):043502, 2013. doi: 10.1118/1.4793409 CrossRefPubMedPubMedCentralGoogle Scholar
  29. 29.
    Channin DS, Mongkolwat P, Kleper V, Sepukar K, Rubin DL: The caBIG annotation and image Markup project. J Digit Imaging 23(2):217–225, 2010. doi: 10.1007/s10278-009-9193-9 CrossRefPubMedGoogle Scholar
  30. 30.
    Dicom Standards Committee WG. Digital Imaging and Communications in Medicine (DICOM) Supplement 111 [cited 2014]. Available from:
  31. 31.
    Obuchowski NA, Reeves AP, Huang EP, Wang XF, Buckler AJ, Kim HJ, Barnhart HX, Jackson EF, Giger ML, Pennello G, Toledano AY, Kalpathy-Cramer J, Apanasovich TV, Kinahan PE, Myers KJ, Goldgof DB, Barboriak DP, Gillies RJ, Schwartz LH, Sullivan AD: Quantitative imaging biomarkers: A review of statistical methods for computer algorithm comparisons. Statistical methods in medical research. 2014. doi: 10.1177/0962280214537390
  32. 32.
    Raunig DL, McShane LM, Pennello G, Gatsonis C, Carson PL, Voyvodic JT, Wahl RL, Kurland BF, Schwarz AJ, Gonen M, Zahlmann G, Kondratovich M, O’Donnell K, Petrick N, Cole PE, Garra B, Sullivan DC, Group QTPW: Quantitative imaging biomarkers: A review of statistical methods for technical performance assessment. Statistical methods in medical research. 2014. doi: 10.1177/0962280214537344
  33. 33.
    Kessler LG, Barnhart HX, Buckler AJ, Choudhury KR, Kondratovich MV, Toledano A, Guimaraes AR, Filice R, Zhang Z, Sullivan DC, Group QTW: The emerging science of quantitative imaging biomarkers terminology and definitions for scientific studies and regulatory submissions. Stat Methods Med Res, 2014. doi: 10.1177/0962280214537333 PubMedGoogle Scholar
  34. 34.
    Barnhart HX, Haber M, Song J: Overall concordance correlation coefficient for evaluating agreement among multiple observers. Biometrics 58(4):1020–1027, 2002CrossRefPubMedGoogle Scholar
  35. 35.
    Lin LI: A concordance correlation coefficient to evaluate reproducibility. Biometrics 45(1):255–268, 1989CrossRefPubMedGoogle Scholar
  36. 36.
    Shrout PE, Fleiss JL: Intraclass correlations: uses in assessing rater reliability. Psychol Bull 86(2):420–428, 1979CrossRefPubMedGoogle Scholar
  37. 37.
    Barnhart HX, Haber MJ, Lin LI: An overview on assessing agreement with continuous measurements. 17(4):529–569, 2007. doi: 10.1080/10543400701376480
  38. 38.
    Barnhart HX, Barboriak DP: Applications of the repeatability of quantitative imaging biomarkers: a review of statistical analysis of repeat data sets. Transl Oncol 2(4):231–235, 2009CrossRefPubMedPubMedCentralGoogle Scholar
  39. 39.
    Bland JM, Altman DG: Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1(8476):307–310, 1986CrossRefPubMedGoogle Scholar
  40. 40.
    Nevill AM, Atkinson G: Assessing agreement between measurements recorded on a ratio scale in sports medicine and sports science. Br J Sports Med 31(4):314–318, 1997CrossRefPubMedPubMedCentralGoogle Scholar
  41. 41.
    Obuchowski NA, Barnhart HX, Buckler AJ, Pennello G, Wang XF, Kalpathy-Cramer J, Kim HJ, Reeves AP, for the Case Example Working G: Statistical issues in the comparison of quantitative imaging biomarker algorithms using pulmonary nodule volume as an example. Statistical methods in medical research. Stat Methods Med Res 24(1):107–140, 2015. doi: 10.1177/0962280214537392 CrossRefPubMedGoogle Scholar
  42. 42.
    Dice LR: Measures of the amount of ecologic association between species. Ecology 26(3):297–302, 1945CrossRefGoogle Scholar
  43. 43.
    Tukey J: Comparing individual means in the analysis of variance. Biometrics 5(2):99–114, 1949CrossRefPubMedGoogle Scholar
  44. 44.
    Siegel S, Castellan Jr, NJ: Nonparametric Statistics for the Behavioral Sciences, 2nd edition. McGraw-Hill Humanities/Social Sciences/Languages, New York, 1988Google Scholar

Copyright information

© Society for Imaging Informatics in Medicine 2016

Authors and Affiliations

  • Jayashree Kalpathy-Cramer
    • 1
  • Binsheng Zhao
    • 2
  • Dmitry Goldgof
    • 3
  • Yuhua Gu
    • 4
  • Xingwei Wang
    • 5
  • Hao Yang
    • 2
  • Yongqiang Tan
    • 2
  • Robert Gillies
    • 4
  • Sandy Napel
    • 5
    Email author
  1. 1.Massachusetts General Hospital and Harvard Medical SchoolBostonUSA
  2. 2.Department of RadiologyColumbia University Medical CenterNew YorkUSA
  3. 3.Department of Computer Science and EngineeringUniversity of South FloridaTampaUSA
  4. 4.Departments of Cancer Imaging and MetabolismH. Lee Moffitt Cancer Center and Research InstituteTampaUSA
  5. 5.Department of RadiologyStanford University School of MedicineStanfordUSA

Personalised recommendations