Advertisement

Crowdsourcing pneumothorax annotations using machine learning annotations on the NIH chest X-ray dataset

  • Ross W. FiliceEmail author
  • Anouk Stein
  • Carol C. Wu
  • Veronica A. Arteaga
  • Stephen Borstelmann
  • Ramya Gaddikeri
  • Maya Galperin-Aizenberg
  • Ritu R. Gill
  • Myrna C. Godoy
  • Stephen B. Hobbs
  • Jean Jeudy
  • Paras C. Lakhani
  • Archana Laroia
  • Sundeep M. Nayak
  • Maansi R. Parekh
  • Prasanth Prasanna
  • Palmi Shah
  • Dharshan Vummidi
  • Kavitha Yaddanapudi
  • George Shih
Original Paper
  • 43 Downloads

Abstract

Pneumothorax is a potentially life-threatening condition that requires prompt recognition and often urgent intervention. In the ICU setting, large numbers of chest radiographs are performed and must be interpreted on a daily basis which may delay diagnosis of this entity. Development of artificial intelligence (AI) techniques to detect pneumothorax could help expedite detection as well as localize and potentially quantify pneumothorax. Open image analysis competitions are useful in advancing state-of-the art AI algorithms but generally require large expert annotated datasets. We have annotated and adjudicated a large dataset of chest radiographs to be made public with the goal of sparking innovation in this space. Because of the cumbersome and time-consuming nature of image labeling, we explored the value of using AI models to generate annotations for review. Utilization of this machine learning annotation (MLA) technique appeared to expedite our annotation process with relatively high sensitivity at the expense of specificity. Further research is required to confirm and better characterize the value of MLAs. Our adjudicated dataset is now available for public consumption in the form of a challenge.

Keywords

Artificial intelligence Machine learning annotations Public datasets Challenge Pneumothorax Chest radiograph 

Notes

Acknowledgements

Anna Zawacki from the Society of Imaging Informatics in Medicine (SIIM) for administrative support during the STR review process.

Compliance with ethical standards

Conflict of Interest

The annotation platform used for this work was provided by MD.ai at no cost. Two authors (Anouk Stein, M.D. and George Shih, M.D., M.S.) serve as stakeholders and/or consultants for MD.ai.

References

  1. 1.
    Yarmus L, Feller-Kopman D: Pneumothorax in the critically ill patient. Chest 141(4):1098–1105, 2012CrossRefGoogle Scholar
  2. 2.
    Gupta D, Hansell A, Nichols T, Duong T, Ayres JG, Strachan D: Epidemiology of pneumothorax in England. Thorax. 55:666–671, 2000CrossRefGoogle Scholar
  3. 3.
    Onuki T, Ueda S, Yamaoka M, Sekiya Y, Yamada H, Kawakami N, Araki Y, Wakai Y, Saito K, Inagaki M, Matsumiya N. Primary and secondary spontaneous pneumothorax: prevalence, clinical features, and in-hospital mortality. Can Respir J. 2017Google Scholar
  4. 4.
    Lakhani P, Prater AB, Hutson RK, Andriole KP, Dreyer KJ, Morey J, Prevedello LM, Clark TJ, Geis JR, Itri JN, Hawkins CM: Machine learning in radiology: applications beyond image interpretation. J Am Coll Radiol 15(2):350–359, 2018 FebCrossRefGoogle Scholar
  5. 5.
    Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg AC, Fei-Fei L. ImageNet large scale visual recognition challenge. arXiv:1409.0575v3 [cs.CV] 30 Jan 2015.Google Scholar
  6. 6.
    Prevedello LM, Halabi SS, Shih G, Wu CC, Kohli MD, Chokshi FH, Erickson BJ, Kalpathy-Cramer J, Andriole KP, Flanders AE. Challenges related to artificial intelligence research in medical imaging and the importance of image analysis competitions. Radiology: Artificial Intelligence. 2019 Jan;1(1).Google Scholar
  7. 7.
    Halabi SS, Prevedello LM, Kalpathy-Cramer J, Mamonov AB, Bilbily A, Cicero M, Pan I, Pereira LA, Sousa RT, Abdala N, Kitamura FC, Thodberg HH, Chen L, Shih G, Andriole K, Kohli MD, Erickson BJ, Flanders AE: The RSNA pediatric bone age machine learning challenge. Radiology. 290(2):498–503, 2019CrossRefGoogle Scholar
  8. 8.
    Rajpurkar P, Irvin J, Bagul A, Ding D, Duan T, Mehta H, Yang B, Zhu K, Laird D, Ball RL, Langlotz C, Shpanskaya K, Lungren MP, Ng AY. MURA: a large dataset for abnormality detection in musculoskeletal radiographs. arXiv: 1712.06957v4 [physics.med-ph] 22 May 2018.Google Scholar
  9. 9.
    Shih G, Wu CC, Halabi SS, Kohli MD, Prevedello LM, Cook TS, Sharma A, Amorosa JK, Arteaga V, Galperin-Aizenberg M. Augmenting the National Institutes of Health chest radiograph dataset with expert annotations of possible pneumonia. Radiology: Artificial Intelligence. 2019 Jan;1(1).Google Scholar
  10. 10.
    SIIM-ACR Pneumothorax Segmentation Challenge. https://www.kaggle.com/c/siim-acr-pneumothorax-segmentation/overview
  11. 11.
    Wang X, Peng Y, Lu L. ChestX-Ray8: hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases arXiv:1705.02315v5 [cs.CV] Dec 2017.Google Scholar
  12. 12.
    Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A. Learning deep features for discriminative localization. arXiv:1512.04150 [cs.CV] 14 Dec 2015.Google Scholar
  13. 13.
    Bach S, Rodriguez D, Liu Y, Luo C, Shao H, Xia C, Souvik S, Ratner A, Hancock B, Al Borzi H, Kuchkal R, Re C, Malkin R. Snorkel Drybell: a case study in deploying weak supervision at industrial scale. arXiv:1812.00417v1 [cs.LG] 2 Dec 2018.Google Scholar
  14. 14.
    Dunnmon J, Ratner A, Khandwala N, Saab K, Markert M, Sagreiya H, Goldman R, Lee-Messer C, Lungren M, Rubin D, Re C. Cross-modal data programming enables rapid medical machine learning. arXiv:1903.11101 [cs.LG] 26 Mar 2019.Google Scholar
  15. 15.
    Girshick R, Donahue J, Darrell T, Malik J. Rich feature hierarchies for accurate object detection and semantic segmentation. arXiv:1311.2524v5 [cs.CV] Oct 2014.Google Scholar
  16. 16.
    He K,Gkioxari G, Dollar P. Mask R-CNN arXiv:1703.06870v3 [cs.CV] Jan 2018.Google Scholar

Copyright information

© Society for Imaging Informatics in Medicine 2019

Authors and Affiliations

  • Ross W. Filice
    • 1
    Email author
  • Anouk Stein
    • 2
  • Carol C. Wu
    • 3
  • Veronica A. Arteaga
    • 4
  • Stephen Borstelmann
    • 5
  • Ramya Gaddikeri
    • 6
  • Maya Galperin-Aizenberg
    • 7
  • Ritu R. Gill
    • 8
  • Myrna C. Godoy
    • 3
  • Stephen B. Hobbs
    • 9
  • Jean Jeudy
    • 10
  • Paras C. Lakhani
    • 11
  • Archana Laroia
    • 12
  • Sundeep M. Nayak
    • 13
  • Maansi R. Parekh
    • 11
  • Prasanth Prasanna
    • 14
  • Palmi Shah
    • 6
  • Dharshan Vummidi
    • 15
  • Kavitha Yaddanapudi
    • 4
  • George Shih
    • 16
  1. 1.Department of RadiologyMedStar Georgetown University HospitalWashingtonUSA
  2. 2.New YorkUSA
  3. 3.Department of RadiologyUniversity of Texas MD Anderson Cancer CenterHoustonUSA
  4. 4.Department of Medical ImagingUniversity of ArizonaTucsonUSA
  5. 5.UCF College of MedicineOrlandoUSA
  6. 6.Department of Radiology and Nuclear MedicineRush University Medical CenterChicagoUSA
  7. 7.Department of Radiology, Perelman School of MedicineHospital of the University of PennsylvaniaPhiladelphiaUSA
  8. 8.Department of Radiology, Beth Israel Deaconess Medical CenterHarvard Medical SchoolBostonUSA
  9. 9.Department of RadiologyUniversity of KentuckyLexingtonUSA
  10. 10.Department of Diagnostic Radiology & Nuclear MedicineUniversity of Maryland School of MedicineBaltimoreUSA
  11. 11.Department of RadiologyThomas Jefferson University HospitalPhiladelphiaUSA
  12. 12.Department of RadiologyUniversity of IowaIowa CityUSA
  13. 13.Division of Thoracic Imaging, Department of Diagnostic RadiologyThe Permanente Medical Group, Inc.San LeandroUSA
  14. 14.Diagnostic Imaging AssociatesSalemUSA
  15. 15.Department of RadiologyUniversity of Michigan Health SystemAnn ArborUSA
  16. 16.Department of RadiologyWeill Cornell MedicineNew YorkUSA

Personalised recommendations