Skip to main content

Advertisement

SpringerLink
Log in

How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals

  • Comment
  • Published:

From Nature Medicine

View current issue Submit your manuscript

A comprehensive overview of medical AI devices approved by the US Food and Drug Administration sheds new light on limitations of the evaluation process that can mask vulnerabilities of devices when they are deployed on patients.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1: Breakdown of 130 FDA-approved medical AI devices by body area.

References

  1. Liu, X. et al. Nat. Med. 26, 1364–1374 (2020).

    Article  CAS  Google Scholar 

  2. Norgeot, B. et al. Nat. Med. 26, 1320–1324 (2020).

    Article  CAS  Google Scholar 

  3. Cruz Rivera, S., Liu, X. & Chan, A. W. et al. Nat. Med. 26, 1351–1363 (2020).

    Article  CAS  Google Scholar 

  4. U.S. Food & Drug Administration Center for Devices and Radiological Health. https://www.fda.gov/downloads/medicaldevices/deviceregulationandguidance/guidancedocuments/ucm514737.pdf (October 2017).

  5. U.S. Food & Drug Administration Center for Devices and Radiological Health. https://www.fda.gov/media/145022/download (January 2021).

  6. U.S. Food & Drug Administration. https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfPMN/denovo.cfm (accessed 14 December 2020).

  7. U.S. Food & Drug Administration. https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfPMA/pma.cfm (accessed 14 December 2020).

  8. U.S. Food & Drug Administration. https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm (accessed 14 December 2020).

  9. Benjamens, S., Dhunnoo, P. & Meskó, B. Npj Digit. Med. 3, 1–8 (2020).

    Article  Google Scholar 

  10. American College of Radiology. https://www.acrdsi.org/DSI-Services/FDA-Cleared-AI-Algorithms (accessed 29 November 2020).

  11. Kaushal, A., Altman, R. & Langlotz, C. J. Am. Med. Assoc. 324, 1212–1213 (2020).

    Article  Google Scholar 

  12. Wang, X. et al. 2017 IEEE Conference on Computer Vision and Pattern Recognition 3462–3471 (Institute of Electrical and Electronics Engineers, 2017).

  13. Irvin, J. et al. The Thirty-Third AAAI Conference on Artificial Intelligence 590–597 (Association for the Advancement of Artificial Intelligence, 2019).

  14. Johnson, A. E. W. et al. Sci. Data 6, 317 (2019).

    Article  Google Scholar 

  15. Huang, G., Liu, Z., Van Der Maaten, L. & Weinberger, K.Q. 2017 IEEE Conference on Computer Vision and Pattern Recognition 4700–4708 (Institute of Electrical and Electronics Engineers, 2017).

  16. Seyyed-Kalantari, L., Liu, G., McDermott, M., Chen, I. Y. & Ghassemi, M. Pac. Symp. Biocomput. 26, 232–243 (2021).

    PubMed  Google Scholar 

  17. U.S. Food & Drug Administration. https://www.fda.gov/medical-devices/device-advice-comprehensive-regulatory-assistance/postmarket-requirements-devices (2018).

  18. Ferryman, K. J. Am. Med. Inform. Assoc. 27, 2016–2019 (2020).

    Article  Google Scholar 

Download references

Acknowledgements

J.Z. is supported by the National Science Foundation (CCF 1763191 and CAREER 1942926), the US National Institutes of Health (P30AG059307 and U01MH098953) and grants from the Silicon Valley Foundation and the Chan-Zuckerberg Initiative. R.D. is supported by the US National Institutes of Health (T32 5T32AR007422-38). Our compiled database of approved medical AI devices, analysis code, and models used for the case study are all available at https://ericwu09.github.io/medical-ai-evaluation.

Author information

Authors and Affiliations

  1. Department of Electrical Engineering, Stanford University, Stanford, CA, USA

    Eric Wu & James Zou

  2. Department of Biomedical Data Science, Stanford University, Stanford, CA, USA

    Kevin Wu, Roxana Daneshjou & James Zou

  3. Department of Dermatology, Stanford University, Stanford, CA, USA

    Roxana Daneshjou

  4. Department of Cardiology, Smidt Heart Institute, Cedars-Sinai Medical Center, Los Angeles, CA, USA

    David Ouyang

  5. Stanford Law School, Stanford University, Stanford, CA, USA

    Daniel E. Ho

  6. Chan-Zuckerberg Biohub, San Francisco, CA, USA

    James Zou

Authors
  1. Eric Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kevin Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Roxana Daneshjou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. David Ouyang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Daniel E. Ho
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. James Zou
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

E.W., K.W. and J.Z. designed the study. E.W., K.W. conducted research with help from R.D. and D.O. All the authors contributed to interpretation of the results and writing of the manuscript.

Corresponding author

Correspondence to James Zou.

Ethics declarations

Competing interests

The authors declare no competing interests.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, E., Wu, K., Daneshjou, R. et al. How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals. Nat Med 27, 582–584 (2021). https://doi.org/10.1038/s41591-021-01312-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41591-021-01312-x

  • Springer Nature America, Inc.

This article is cited by

Search

Navigation