To evaluate the performance of an AI-powered algorithm for the automatic detection of pulmonary embolism (PE) on chest computed tomography pulmonary angiograms (CTPAs) on a large dataset.
We retrospectively identified all CTPAs conducted at our institution in 2017 (n = 1499). Exams with clinical questions other than PE were excluded from the analysis (n = 34). The remaining exams were classified into positive (n = 232) and negative (n = 1233) for PE based on the final written reports, which defined the reference standard. The fully anonymized 1-mm series in soft tissue reconstruction served as input for the PE detection prototype algorithm that was based on a deep convolutional neural network comprising a Resnet architecture. It was trained and validated on 28,000 CTPAs acquired at other institutions. The result series were reviewed using a web-based feedback platform. Measures of diagnostic performance were calculated on a per patient and a per finding level.
The algorithm correctly identified 215 of 232 exams positive for pulmonary embolism (sensitivity 92.7%; 95% confidence interval [CI] 88.3–95.5%) and 1178 of 1233 exams negative for pulmonary embolism (specificity 95.5%; 95% CI 94.2–96.6%). On a per finding level, 1174 of 1352 findings marked as embolus by the algorithm were true emboli. Most of the false positive findings were due to contrast agent–related flow artifacts, pulmonary veins, and lymph nodes.
The AI prototype algorithm we tested has a high degree of diagnostic accuracy for the detection of PE on CTPAs. Sensitivity and specificity are balanced, which is a prerequisite for its clinical usefulness.
• An AI-based prototype algorithm showed a high degree of diagnostic accuracy for the detection of pulmonary embolism on CTPAs.
• It can therefore help clinicians to automatically prioritize exams with a high suspection of pulmonary embolism and serve as secondary reading tool.
• By complementing traditional ways of worklist prioritization in radiology departments, this can speed up the diagnostic and therapeutic workup of patients with pulmonary embolism and help to avoid false negative calls.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Advanced Modeled Iterative Reconstruction
CT pulmonary angiogram
Deep convolutional neural network
Digital imaging and communications in medicine
File transfer protocol
Negative predictive value
Picture Archiving and Communication System
Positive predictive value
Sinogram Affirmed Iterative Reconstruction
Sample weighted categorical cross-entropy
Brown J, Shesser R (2004) Computed tomography scan use is rising faster than other investigational modalities in the emergency department evaluation of patients. Ann Emerg Med 44:S33. https://doi.org/10.1016/j.annemergmed.2004.07.109
Kocher KE, Meurer WJ, Fazel R, Scott PA, Krumholz HM, Nallamothu BK (2011) National trends in use of computed tomography in the emergency department. Ann Emerg Med 58:452–462.e3. https://doi.org/10.1016/J.ANNEMERGMED.2011.05.020
Estrada-Y-Martin RM, Oldham SA (2011) CTPA as the gold standard for the diagnosis of pulmonary embolism. Int J Comput Assist Radiol Surg 6:557–563. https://doi.org/10.1007/s11548-010-0526-4
Pernod G, Caterino J, Maignan M et al (2017) D-dimer use and pulmonary embolism diagnosis in emergency units: why is there such a difference in pulmonary embolism prevalence between the United States of America and countries outside USA? PLoS One 12:e0169268. https://doi.org/10.1371/journal.pone.0169268
Smith SB, Geske JB, Maguire JM, Zane NA, Carter RE, Morgenthaler TI (2010) Early anticoagulation is associated with reduced mortality for acute pulmonary embolism. Chest 137:1382–1390
Lecun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86:2278–2324. https://doi.org/10.1109/5.726791
Prevedello LM, Erdal BS, Ryu JL et al (2017) Automated critical test findings identification and online notification system using artificial intelligence in imaging. Radiology 285:923–931. https://doi.org/10.1148/radiol.2017162664
Nagel S, Sinha D, Day D et al (2017) e-ASPECTS software is non-inferior to neuroradiologists in applying the ASPECT score to computed tomography scans of acute ischemic stroke patients. Int J Stroke 12:615–622. https://doi.org/10.1177/1747493016681020
Winkel DJ, Heye T, Weikert TJ, Boll DT, Stieltjes B (2018) Evaluation of an AI-based detection software for acute findings in abdominal computed tomography scans. Invest Radiol:1. https://doi.org/10.1097/RLI.0000000000000509
Liu K, Li Q, Ma J et al (2019) Evaluating a fully automated pulmonary nodule detection approach and its impact on radiologist performance. Radiol Artif Intell 1:e180084. https://doi.org/10.1148/ryai.2019180084
Vorontsov E, Cerny M, Régnier P et al (2019) Deep learning for automated segmentation of liver lesions at CT in patients with colorectal cancer liver metastases. Radiol Artif Intell 1:180014. https://doi.org/10.1148/ryai.2019180014
Thian YL, Li Y, Jagmohan P, Sia D, Chan VE, Tan RT (2019) Convolutional neural networks for automated fracture detection and localization on wrist radiographs. Radiol Artif Intell 1:e180001. https://doi.org/10.1148/ryai.2019180001
Li L, Liu Z, Huang H, Lin M, Luo D (2018) Evaluating the performance of a deep learning-based computer-aided diagnosis (DL-CAD) system for detecting and characterizing lung nodules: comparison with the performance of double reading by radiologists. Thorac Cancer 10:1759–7714.12931. https://doi.org/10.1111/1759-7714.12931
Ekert T, Krois J, Meinhold L et al (2019) Deep learning for the radiographic detection of apical lesions. J Endod. https://doi.org/10.1016/j.joen.2019.03.016
Cheng C-T, Ho T-Y, Lee T-Y et al (2019) Application of a deep learning algorithm for detection and visualization of hip fractures on plain pelvic radiographs. Eur Radiol. https://doi.org/10.1007/s00330-019-06167-y
Ye W, Gu W, Guo X et al (2019) Detection of pulmonary ground-glass opacity based on deep learning computer artificial intelligence. Biomed Eng Online 18:6. https://doi.org/10.1186/s12938-019-0627-4
Maizlin ZV, Vos PM, Godoy MB, Cooperberg PL (2007) Computer-aided detection of pulmonary embolism on CT angiography. J Thorac Imaging 22:324–329. https://doi.org/10.1097/RTI.0b013e31815b89ca
Engelke C, Schmidt S, Bakai A, Auer F, Marten K (2008) Computer-assisted detection of pulmonary embolism: performance evaluation in consensus with experienced and inexperienced chest radiologists. Eur Radiol 18:298–307. https://doi.org/10.1007/s00330-007-0770-3
Bouma H, Sonnemans JJ, Vilanova A, Gerritsen FA (2009) Automatic detection of pulmonary embolism in CTA images. IEEE Trans Med Imaging 28:1223–1230. https://doi.org/10.1109/TMI.2009.2013618
Lee CW, Seo JB, Song J-W et al (2011) Evaluation of computer-aided detection and dual energy software in detection of peripheral pulmonary embolism on dual-energy pulmonary CT angiography. Eur Radiol 21:54–62. https://doi.org/10.1007/s00330-010-1903-7
Kligerman SJ, Lahiji K, Galvin JR, Stokum C, White CS (2014) Missed pulmonary emboli on CT angiography: assessment with pulmonary embolism–computer-aided detection. Am J Roentgenol 202:65–73. https://doi.org/10.2214/AJR.13.11049
Masutani Y, MacMahon H, Doi K (2002) Computerized detection of pulmonary embolism in spiral CT angiography based on volumetric image analysis. IEEE Trans Med Imaging 21:1517–1523. https://doi.org/10.1109/TMI.2002.806586
Pichon E, Novak CL, Kiraly AP, Naidich DP (2004) A novel method for pulmonary emboli visualization from high-resolution CT images. Proceedings of the SPIE, Volume 5367, p 161-170 (2004). p 161
Liang J, Bi J (2007) Computer aided detection of pulmonary embolism with tobogganing and mutiple instance classification in CT pulmonary angiography. Inf Process Med Imaging 20:630–641
Zhou C, Chan H-P, Sahiner B et al (2009) Computer-aided detection of pulmonary embolism in computed tomographic pulmonary angiography (CTPA): performance evaluation with independent data sets. Med Phys 36:3385–3396. https://doi.org/10.1118/1.3157102
Özkan H, Osman O, Şahin S, Boz AF (2014) A novel method for pulmonary embolism detection in CTA images. Comput Methods Programs Biomed 113:757–766. https://doi.org/10.1016/j.cmpb.2013.12.014
Digumarthy S, Kagay C, Legasto A, Muse V, Wittram C, Shepard J (2006) Computer-aided detection (CAD) of acute pulmonary emboli: evaluation in patients without significant pulmonary disease. Radiological Society of North America 2006 Scientific Assembly and Annual Meeting, November 26 - December 1, 2006, Chicago IL
Schoepf UJ, Schneider AC, Das M, Wood SA, Cheema JI, Philip Costello P (2007) Pulmonary embolism: computer-aided detection at multidetector row spiral computed tomography. J Thorac Imaging 22:319–323. https://doi.org/10.1097/RTI.0b013e31815842a9
Wittenberg R, Peters JF, Sonnemans JJ, Prokop M, Schaefer-Prokop CM (2010) Computer-assisted detection of pulmonary embolism: evaluation of pulmonary CT angiograms performed in an on-call setting. Eur Radiol 20:801–806. https://doi.org/10.1007/s00330-009-1628-7
Blackmon KN, Florin C, Bogoni L et al (2011) Computer-aided detection of pulmonary embolism at CT pulmonary angiography: can it improve performance of inexperienced readers? Eur Radiol 21:1214–1223. https://doi.org/10.1007/s00330-010-2050-x
Wittenberg R, Berger FH, Peters JF et al (2012) Acute pulmonary embolism: effect of a computer-assisted detection prototype on diagnosis—an observer study. Radiology 262:305–313. https://doi.org/10.1148/radiol.11110372
Sang Cheol Park SC, Chapman BE, Bin Zheng B (2011) A multistage approach to improve performance of computer-aided detection of pulmonary embolisms depicted on CT images: preliminary investigation. IEEE Trans Biomed Eng 58:1519–1527. https://doi.org/10.1109/TBME.2010.2063702
Das M, Mühlenbruch G, Helm A et al (2008) Computer-aided detection of pulmonary embolism: influence on radiologists’ detection performance with respect to vessel segments. Eur Radiol 18:1350–1355. https://doi.org/10.1007/s00330-008-0889-x
Lee G, Lee HY, Park H et al (2017) Radiomics and its emerging role in lung cancer research, imaging biomarkers and clinical management: state of the art. Eur J Radiol 86:297–307. https://doi.org/10.1016/J.EJRAD.2016.09.005
Lahiji K, Kligerman S, Jeudy J, White C (2014) Improved accuracy of pulmonary embolism computer-aided detection using iterative reconstruction compared with filtered back projection. AJR Am J Roentgenol 203:763–771. https://doi.org/10.2214/AJR.13.11838
He K, Zhang X, Ren S, Sun J (2015) Deep residual learning for image recognition. Proceedings of the IEEE Conference on Cumputer Vision and Pattern Recognition: 770-778. https://doi.org/10.1109/CVPR.2016.90
Wilson EB (1927) Probable inference, the law of succession, and statistical inference. J Am Stat Assoc 22:209. https://doi.org/10.2307/2276774
Wittenberg R, Peters JF, van den Berk IAH et al (2013) Computed tomography pulmonary angiography in acute pulmonary embolism. J Thorac Imaging 28:315–321. https://doi.org/10.1097/RTI.0b013e3182870b97
Bhargavan M, Kaye AH, Forman HP, Sunshine JH (2009) Workload of radiologists in United States in 2006-2007 and trends since 1991-1992. Radiology 252:458–467. https://doi.org/10.1148/radiol.2522081895
Tajbakhsh N, Gotway MB, Liang J (2015) Computer-aided pulmonary embolism detection using a novel vessel-aligned multiplanar image representation and convolutional neural networks. In: Navab N, Hornegger J, Wells W, Frangi A (eds) Medical image computing and computer-assisted intervention --MICCAI 2015. MICCAI 2015. Lecture notes in computer science, vol 9350. Springer, Cham
Tajbakhsh N, Shin JY, Gotway MB, Liang J (2019) Computer-aided detection and visualization of pulmonary embolism using a novel, compact, and discriminative image representation. Med Image Anal 58:101541. https://doi.org/10.1016/J.MEDIA.2019.101541
Beydilli İ, Yılmaz F, Sönmez BM et al (2016) Thrombolytic therapy delay is independent predictor of mortality in acute pulmonary embolism at emergency service. Kaohsiung J Med Sci 32:572–578. https://doi.org/10.1016/j.kjms.2016.09.004
Annarumma M, Withey SJ, Bakewell RJ, Pesce E, Goh V, Montana G (2019) Automated triaging of adult chest radiographs with deep artificial neural networks. Radiology:291
Rayan JC, Reddy N, Kan JH, Zhang W, Annapragada A (2019) Binomial classification of pediatric elbow fractures using a deep learning multiview approach emulating radiologist decision making. Radiol Artif Intell 1:e180015. https://doi.org/10.1148/ryai.2019180015
Penaloza A, Kline J, Verschuren F et al (2012) European and American suspected and confirmed pulmonary embolism populations: comparison and analysis. J Thromb Haemost 10:375–381. https://doi.org/10.1111/j.1538-7836.2012.04631.x
The authors state that this work has not received any funding.
The scientific guarantor of this publication is Gregor Sommer, MD, PhD (University Hospital Basel, Switzerland).
Conflict of interest
The authors of this manuscript declare no relationships with any companies whose products or services may be related to the subject matter of the article.
Statistics and biometry
No complex statistical methods were necessary for this paper.
Written informed consent was waived by the Institutional Review Board (project number 2019-01050). We added the document of the IRB decision to this submission.
Institutional Review Board approval was obtained.
• diagnostic study
• performed at one institution
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Weikert, T., Winkel, D.J., Bremerich, J. et al. Automated detection of pulmonary embolism in CT pulmonary angiograms using an AI-powered algorithm. Eur Radiol 30, 6545–6553 (2020). https://doi.org/10.1007/s00330-020-06998-0
- Pulmonary embolism
- Computed tomography angiography
- Artificial intelligence
- Computer-assisted image processing