Skip to main content
SpringerLink
Log in

Computer-aided detection (CAD) of lung nodules in CT scans: radiologist performance and reading time with incremental CAD assistance

  • Chest
  • Published:
European Radiology Aims and scope Submit manuscript

Abstract

Objective

The diagnostic performance of radiologists using incremental CAD assistance for lung nodule detection on CT and their temporal variation in performance during CAD evaluation was assessed.

Methods

CAD was applied to 20 chest multidetector-row computed tomography (MDCT) scans containing 190 non-calcified ≥3-mm nodules. After free search, three radiologists independently evaluated a maximum of up to 50 CAD detections/patient. Multiple free-response ROC curves were generated for free search and successive CAD evaluation, by incrementally adding CAD detections one at a time to the radiologists’ performance.

Results

The sensitivity for free search was 53% (range, 44%–59%) at 1.15 false positives (FP)/patient and increased with CAD to 69% (range, 59–82%) at 1.45 FP/patient. CAD evaluation initially resulted in a sharp rise in sensitivity of 14% with a minimal increase in FP over a time period of 100 s, followed by flattening of the sensitivity increase to only 2%. This transition resulted from a greater prevalence of true positive (TP) versus FP detections at early CAD evaluation and not by a temporal change in readers’ performance. The time spent for TP (9.5 s ± 4.5 s) and false negative (FN) (8.4 s ± 6.7 s) detections was similar; FP decisions took two- to three-times longer (14.4 s ± 8.7 s) than true negative (TN) decisions (4.7 s ± 1.3 s).

Conclusions

When CAD output is ordered by CAD score, an initial period of rapid performance improvement slows significantly over time because of non-uniformity in the distribution of TP CAD output and not to a changing reader performance over time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Beigelman-Aubry C, Raffy P, Yang W, Castellino RA, Grenier PA (2007) Computer-aided detection of solid lung nodules on follow-up MDCT screening: evaluation of detection, tracking, and reading time. AJR Am J Roentgenol 189:948–955

    Article  PubMed  Google Scholar 

  2. Beyer F, Zierott L, Fallenberg EM, Juergens KU, Stoeckel J, Heindel W, Wormanns D (2007) Comparison of sensitivity and reading time for the use of computer-aided detection (CAD) of pulmonary nodules at MDCT as concurrent or second reader. Eur Radiol 17:2941–2947

    Article  CAS  PubMed  Google Scholar 

  3. Godoy MC, Cooperberg PL, Maizlin ZV, Yuan R, McWilliams A, Lam S, Mayo JR (2008) Detection sensitivity of a commercial lung nodule CAD system in a series of pathologically proven lung cancers. J Thorac Imaging 23:1–6

    Article  PubMed  Google Scholar 

  4. Jankowski A, Martinelli T, Timsit JF, Brambilla C, Thony F, Coulomb M, Ferretti G (2007) Pulmonary nodule detection on MDCT images: evaluation of diagnostic performance using thin axial images, maximum intensity projections, and computer-assisted detection. Eur Radiol 17:3148–3156

    Article  CAS  PubMed  Google Scholar 

  5. Marten K, Engelke C (2007) Computer-aided detection and automated CT volumetry of pulmonary nodules. Eur Radiol 17:888–901

    Article  PubMed  Google Scholar 

  6. Rubin GD, Lyo JK, Paik DS, Sherbondy AJ, Chow LC, Leung AN, Mindelzun R, Schraedley-Desmond PK, Zinck SE, Naidich DP, Napel S (2005) Pulmonary nodules on multi-detector row CT scans: performance comparison of radiologists and computer-aided detection. Radiology 234:274–283

    Article  PubMed  Google Scholar 

  7. Saba L, Caddeo G, Mallarini G (2007) Computer-aided detection of pulmonary nodules in computed tomography: analysis and review of the literature. J Comput Assist Tomogr 31:611–619

    Article  PubMed  Google Scholar 

  8. White CS, Pugatch R, Koonce T, Rust SW, Dharaiya E (2008) Lung nodule CAD software as a second reader: a multicenter study. Acad Radiol 15:326–333

    Article  PubMed  Google Scholar 

  9. Yuan R, Vos PM, Cooperberg PL (2006) Computer-aided detection in screening CT for pulmonary nodules. AJR Am J Roentgenol 186:1280–1287

    Article  PubMed  Google Scholar 

  10. Das M, Muhlenbruch G, Heinen S, Mahnken AH, Salganicoff M, Stanzel S, Gunther RW, Wildberger JE (2008) Performance evaluation of a computer-aided detection algorithm for solid pulmonary nodules in low-dose and standard-dose MDCT chest examinations and its influence on radiologists. Br J Radiol 81:841–847

    Article  CAS  PubMed  Google Scholar 

  11. Lee JY, Chung MJ, Yi CA, Lee KS (2008) Ultra-low-dose MDCT of the chest: influence on automated lung nodule detection. Korean J Radiol 9:95–101

    Article  PubMed  Google Scholar 

  12. Das M, Muhlenbruch G, Mahnken AH, Flohr TG, Gundel L, Stanzel S, Kraus T, Gunther RW, Wildberger JE (2006) Small pulmonary nodules: effect of two computer-aided detection systems on radiologist performance. Radiology 241:564–571

    Article  PubMed  Google Scholar 

  13. Zheng B, Leader JK, Abrams G, Shindel B, Catullo V, Good WF, Gur D (2004) Computer-aided detection schemes: the effect of limiting the number of cued regions in each case. AJR Am J Roentgenol 182:579–583

    PubMed  Google Scholar 

  14. Paik DS, Beaulieu CF, Rubin GD, Acar B, Jeffrey RB Jr, Yee J, Dey J, Napel S (2004) Surface normal overlap: a computer-aided detection algorithm with application to colonic polyps and lung nodules in helical CT. IEEE Trans Med Imaging 23:661–675

    Article  PubMed  Google Scholar 

  15. Cheng H, Macaluso M (1997) Comparison of the accuracy of two tests with a confirmatory procedure limited to positive results. Epidemiology 8:104–106

    Article  CAS  PubMed  Google Scholar 

  16. Eliasziw M, Donner A (1991) Application of the McNemar test to non-independent matched pair data. Stat Med 10:1981–1991

    Article  CAS  PubMed  Google Scholar 

  17. Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33:159–174

    Article  CAS  PubMed  Google Scholar 

  18. Brochu B, Beigelman-Aubry C, Goldmard JL, Raffy P, Grenier PA, Lucidarme O (2007) Computer-aided detection of lung nodules on thin collimation MDCT: impact on radiologists’ performance. J Radiol 88:573–578

    Article  CAS  PubMed  Google Scholar 

  19. Suzuki K, Armato SG 3rd, Li F, Sone S, Doi K (2003) Massive training artificial neural network (MTANN) for reduction of false positives in computerized detection of lung nodules in low-dose computed tomography. Med Phys 30:1602–1617

    Article  PubMed  Google Scholar 

  20. Ge Z, Sahiner B, Chan HP, Hadjiiski LM, Cascade PN, Bogot N, Kazerooni EA, Wei J, Zhou C (2005) Computer-aided detection of lung nodules: false positive reduction using a 3D gradient field method and 3D ellipsoid fitting. Med Phys 32:2443–2454

    Article  PubMed  Google Scholar 

  21. Roy AS, Armato SG 3rd, Wilson A, Drukker K (2006) Automated detection of lung nodules in CT scans: false-positive reduction with the radial-gradient index. Med Phys 33:1133–1140

    Article  PubMed  Google Scholar 

  22. Armato SG 3rd, McNitt-Gray MF, Reeves AP, Meyer CR, McLennan G, Aberle DR, Kazerooni EA, MacMahon H, van Beek EJ, Yankelevitz D, Hoffman EA, Henschke CI, Roberts RY, Brown MS, Engelmann RM, Pais RC, Piker CW, Qing D, Kocherginsky M, Croft BY, Clarke LP (2007) The Lung Image Database Consortium (LIDC): an evaluation of radiologist variability in the identification of lung nodules on CT scans. Acad Radiol 14:1409–1421

    Article  PubMed  Google Scholar 

  23. MacMahon H, Austin JH, Gamsu G, Herold CJ, Jett JR, Naidich DP, Patz EF Jr, Swensen SJ (2005) Guidelines for management of small pulmonary nodules detected on CT scans: a statement from the Fleischner Society. Radiology 237:395–400

    Article  PubMed  Google Scholar 

  24. Brown MS, Goldin JG, Rogers S, Kim HJ, Suh RD, McNitt-Gray MF, Shah SK, Truong D, Brown K, Sayre JW, Gjertson DW, Batra P, Aberle DR (2005) Computer-aided lung nodule detection in CT: results of large-scale observer test. Acad Radiol 12:681–686

    Article  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Radiology, Stanford University Medical Center, 300 Pasteur Drive, Room S-072, Stanford, CA, 94305-5105, USA

    Justus E. Roos, David Paik, David Olsen, Emily G. Liu, Ann N. Leung, Robert Mindelzun, Kingshuk R. Choudhury, Sandy Napel & Geoffrey D. Rubin

  2. Department of Radiology, Oregon Health and Science University, MC L340, 3181 SW Sam Jackson Park Rd, Portland, OR, 97201, USA

    Lawrence C. Chow

  3. Department of Radiology, New York University Medical Center, 550 First Avenue, New York, NY, 10016, USA

    David P. Naidich

Authors
  1. Justus E. Roos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. David Paik
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. David Olsen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Emily G. Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Lawrence C. Chow
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ann N. Leung
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Robert Mindelzun
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Kingshuk R. Choudhury
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. David P. Naidich
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Sandy Napel
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Geoffrey D. Rubin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Justus E. Roos.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Roos, J.E., Paik, D., Olsen, D. et al. Computer-aided detection (CAD) of lung nodules in CT scans: radiologist performance and reading time with incremental CAD assistance. Eur Radiol 20, 549–557 (2010). https://doi.org/10.1007/s00330-009-1596-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00330-009-1596-y

Keywords

Search

Navigation