European Radiology

, Volume 20, Issue 3, pp 549–557 | Cite as

Computer-aided detection (CAD) of lung nodules in CT scans: radiologist performance and reading time with incremental CAD assistance

  • Justus E. RoosEmail author
  • David Paik
  • David Olsen
  • Emily G. Liu
  • Lawrence C. Chow
  • Ann N. Leung
  • Robert Mindelzun
  • Kingshuk R. Choudhury
  • David P. Naidich
  • Sandy Napel
  • Geoffrey D. Rubin



The diagnostic performance of radiologists using incremental CAD assistance for lung nodule detection on CT and their temporal variation in performance during CAD evaluation was assessed.


CAD was applied to 20 chest multidetector-row computed tomography (MDCT) scans containing 190 non-calcified ≥3-mm nodules. After free search, three radiologists independently evaluated a maximum of up to 50 CAD detections/patient. Multiple free-response ROC curves were generated for free search and successive CAD evaluation, by incrementally adding CAD detections one at a time to the radiologists’ performance.


The sensitivity for free search was 53% (range, 44%–59%) at 1.15 false positives (FP)/patient and increased with CAD to 69% (range, 59–82%) at 1.45 FP/patient. CAD evaluation initially resulted in a sharp rise in sensitivity of 14% with a minimal increase in FP over a time period of 100 s, followed by flattening of the sensitivity increase to only 2%. This transition resulted from a greater prevalence of true positive (TP) versus FP detections at early CAD evaluation and not by a temporal change in readers’ performance. The time spent for TP (9.5 s ± 4.5 s) and false negative (FN) (8.4 s ± 6.7 s) detections was similar; FP decisions took two- to three-times longer (14.4 s ± 8.7 s) than true negative (TN) decisions (4.7 s ± 1.3 s).


When CAD output is ordered by CAD score, an initial period of rapid performance improvement slows significantly over time because of non-uniformity in the distribution of TP CAD output and not to a changing reader performance over time.


Multidetector-row computed tomography MDCT Computer-aided detection CAD Pulmonary nodules Diagnostic performance 


  1. 1.
    Beigelman-Aubry C, Raffy P, Yang W, Castellino RA, Grenier PA (2007) Computer-aided detection of solid lung nodules on follow-up MDCT screening: evaluation of detection, tracking, and reading time. AJR Am J Roentgenol 189:948–955CrossRefPubMedGoogle Scholar
  2. 2.
    Beyer F, Zierott L, Fallenberg EM, Juergens KU, Stoeckel J, Heindel W, Wormanns D (2007) Comparison of sensitivity and reading time for the use of computer-aided detection (CAD) of pulmonary nodules at MDCT as concurrent or second reader. Eur Radiol 17:2941–2947CrossRefPubMedGoogle Scholar
  3. 3.
    Godoy MC, Cooperberg PL, Maizlin ZV, Yuan R, McWilliams A, Lam S, Mayo JR (2008) Detection sensitivity of a commercial lung nodule CAD system in a series of pathologically proven lung cancers. J Thorac Imaging 23:1–6CrossRefPubMedGoogle Scholar
  4. 4.
    Jankowski A, Martinelli T, Timsit JF, Brambilla C, Thony F, Coulomb M, Ferretti G (2007) Pulmonary nodule detection on MDCT images: evaluation of diagnostic performance using thin axial images, maximum intensity projections, and computer-assisted detection. Eur Radiol 17:3148–3156CrossRefPubMedGoogle Scholar
  5. 5.
    Marten K, Engelke C (2007) Computer-aided detection and automated CT volumetry of pulmonary nodules. Eur Radiol 17:888–901CrossRefPubMedGoogle Scholar
  6. 6.
    Rubin GD, Lyo JK, Paik DS, Sherbondy AJ, Chow LC, Leung AN, Mindelzun R, Schraedley-Desmond PK, Zinck SE, Naidich DP, Napel S (2005) Pulmonary nodules on multi-detector row CT scans: performance comparison of radiologists and computer-aided detection. Radiology 234:274–283CrossRefPubMedGoogle Scholar
  7. 7.
    Saba L, Caddeo G, Mallarini G (2007) Computer-aided detection of pulmonary nodules in computed tomography: analysis and review of the literature. J Comput Assist Tomogr 31:611–619CrossRefPubMedGoogle Scholar
  8. 8.
    White CS, Pugatch R, Koonce T, Rust SW, Dharaiya E (2008) Lung nodule CAD software as a second reader: a multicenter study. Acad Radiol 15:326–333CrossRefPubMedGoogle Scholar
  9. 9.
    Yuan R, Vos PM, Cooperberg PL (2006) Computer-aided detection in screening CT for pulmonary nodules. AJR Am J Roentgenol 186:1280–1287CrossRefPubMedGoogle Scholar
  10. 10.
    Das M, Muhlenbruch G, Heinen S, Mahnken AH, Salganicoff M, Stanzel S, Gunther RW, Wildberger JE (2008) Performance evaluation of a computer-aided detection algorithm for solid pulmonary nodules in low-dose and standard-dose MDCT chest examinations and its influence on radiologists. Br J Radiol 81:841–847CrossRefPubMedGoogle Scholar
  11. 11.
    Lee JY, Chung MJ, Yi CA, Lee KS (2008) Ultra-low-dose MDCT of the chest: influence on automated lung nodule detection. Korean J Radiol 9:95–101CrossRefPubMedGoogle Scholar
  12. 12.
    Das M, Muhlenbruch G, Mahnken AH, Flohr TG, Gundel L, Stanzel S, Kraus T, Gunther RW, Wildberger JE (2006) Small pulmonary nodules: effect of two computer-aided detection systems on radiologist performance. Radiology 241:564–571CrossRefPubMedGoogle Scholar
  13. 13.
    Zheng B, Leader JK, Abrams G, Shindel B, Catullo V, Good WF, Gur D (2004) Computer-aided detection schemes: the effect of limiting the number of cued regions in each case. AJR Am J Roentgenol 182:579–583PubMedGoogle Scholar
  14. 14.
    Paik DS, Beaulieu CF, Rubin GD, Acar B, Jeffrey RB Jr, Yee J, Dey J, Napel S (2004) Surface normal overlap: a computer-aided detection algorithm with application to colonic polyps and lung nodules in helical CT. IEEE Trans Med Imaging 23:661–675CrossRefPubMedGoogle Scholar
  15. 15.
    Cheng H, Macaluso M (1997) Comparison of the accuracy of two tests with a confirmatory procedure limited to positive results. Epidemiology 8:104–106CrossRefPubMedGoogle Scholar
  16. 16.
    Eliasziw M, Donner A (1991) Application of the McNemar test to non-independent matched pair data. Stat Med 10:1981–1991CrossRefPubMedGoogle Scholar
  17. 17.
    Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33:159–174CrossRefPubMedGoogle Scholar
  18. 18.
    Brochu B, Beigelman-Aubry C, Goldmard JL, Raffy P, Grenier PA, Lucidarme O (2007) Computer-aided detection of lung nodules on thin collimation MDCT: impact on radiologists’ performance. J Radiol 88:573–578CrossRefPubMedGoogle Scholar
  19. 19.
    Suzuki K, Armato SG 3rd, Li F, Sone S, Doi K (2003) Massive training artificial neural network (MTANN) for reduction of false positives in computerized detection of lung nodules in low-dose computed tomography. Med Phys 30:1602–1617CrossRefPubMedGoogle Scholar
  20. 20.
    Ge Z, Sahiner B, Chan HP, Hadjiiski LM, Cascade PN, Bogot N, Kazerooni EA, Wei J, Zhou C (2005) Computer-aided detection of lung nodules: false positive reduction using a 3D gradient field method and 3D ellipsoid fitting. Med Phys 32:2443–2454CrossRefPubMedGoogle Scholar
  21. 21.
    Roy AS, Armato SG 3rd, Wilson A, Drukker K (2006) Automated detection of lung nodules in CT scans: false-positive reduction with the radial-gradient index. Med Phys 33:1133–1140CrossRefPubMedGoogle Scholar
  22. 22.
    Armato SG 3rd, McNitt-Gray MF, Reeves AP, Meyer CR, McLennan G, Aberle DR, Kazerooni EA, MacMahon H, van Beek EJ, Yankelevitz D, Hoffman EA, Henschke CI, Roberts RY, Brown MS, Engelmann RM, Pais RC, Piker CW, Qing D, Kocherginsky M, Croft BY, Clarke LP (2007) The Lung Image Database Consortium (LIDC): an evaluation of radiologist variability in the identification of lung nodules on CT scans. Acad Radiol 14:1409–1421CrossRefPubMedGoogle Scholar
  23. 23.
    MacMahon H, Austin JH, Gamsu G, Herold CJ, Jett JR, Naidich DP, Patz EF Jr, Swensen SJ (2005) Guidelines for management of small pulmonary nodules detected on CT scans: a statement from the Fleischner Society. Radiology 237:395–400CrossRefPubMedGoogle Scholar
  24. 24.
    Brown MS, Goldin JG, Rogers S, Kim HJ, Suh RD, McNitt-Gray MF, Shah SK, Truong D, Brown K, Sayre JW, Gjertson DW, Batra P, Aberle DR (2005) Computer-aided lung nodule detection in CT: results of large-scale observer test. Acad Radiol 12:681–686CrossRefPubMedGoogle Scholar

Copyright information

© European Society of Radiology 2009

Authors and Affiliations

  • Justus E. Roos
    • 1
    Email author
  • David Paik
    • 1
  • David Olsen
    • 1
  • Emily G. Liu
    • 1
  • Lawrence C. Chow
    • 2
  • Ann N. Leung
    • 1
  • Robert Mindelzun
    • 1
  • Kingshuk R. Choudhury
    • 1
  • David P. Naidich
    • 3
  • Sandy Napel
    • 1
  • Geoffrey D. Rubin
    • 1
  1. 1.Department of RadiologyStanford University Medical CenterStanfordUSA
  2. 2.Department of RadiologyOregon Health and Science UniversityPortlandUSA
  3. 3.Department of RadiologyNew York University Medical CenterNew YorkUSA

Personalised recommendations