Skip to main content
SpringerLink
Log in

Quality assurance (QA) for monitoring the performance of assisted reproductive technology (ART) staff using artificial intelligence (AI)

Journal of Assisted Reproduction and Genetics volume 40, pages 241–249 (2023)Cite this article

Abstract

Purpose

Deep learning neural networks have been used to predict the developmental fate and implantation potential of embryos with high accuracy. Such networks have been used as an assistive quality assurance (QA) tool to identify perturbations in the embryo culture environment which may impact clinical outcomes. The present study aimed to evaluate the utility of an AI-QA tool to consistently monitor ART staff performance (MD and embryologist) in embryo transfer (ET), embryo vitrification (EV), embryo warming (EW), and trophectoderm biopsy (TBx).

Methods

Pregnancy outcomes from groups of 20 consecutive elective single day 5 blastocyst transfers were evaluated for the following procedures: MD performed ET (N = 160 transfers), embryologist performed ET (N = 160 transfers), embryologist performed EV (N = 160 vitrification procedures), embryologist performed EW (N = 160 warming procedures), and embryologist performed TBx (N = 120 biopsies). AI-generated implantation probabilities for the same embryo cohorts were estimated, as were mean AI-predicted and actual implantation rates for each provider and compared using Wilcoxon singed-rank test.

Results

Actual implantation rates following ET performed by one MD provider: “H” was significantly lower than AI-predicted (20% vs. 61%, p = 0.001). Similar results were observed for one embryologist, “H” (30% vs. 60%, p = 0.011). Embryos thawed by embryologist “H” had lower implantation rates compared to AI prediction (25% vs. 60%, p = 0.004). There were no significant differences between actual and AI-predicted implantation rates for EV, TBx, or for the rest of the clinical staff performing ET or EW.

Conclusions

AI-based QA tools could provide accurate, reproducible, and efficient staff performance monitoring in an ART practice.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

References

  1. Steptoe PC, Edwards RG. Birth after the reimplantation of a human embryo. Lancet. 1978;312:366.

    Article  Google Scholar 

  2. Sunderam S, Kissin DM, Zhang Y, Jewett A, Boulet SL, Warner L, et al. Assisted reproductive technology surveillance - United States, 2018. MMWR Surveill Summ. 2022;71:1–19.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Chambers GM, Dyer S, Zegers-Hochschild F, de Mouzon J, Ishihara O, Banker M, et al. International Committee for Monitoring Assisted Reproductive Technologies world report: assisted reproductive technology, 2014†. Hum Reprod. 2021;36:2921–34.

    Article  PubMed  Google Scholar 

  4. Go KJ. “By the work, one knows the workman”: the practice and profession of the embryologist and its translation to quality in the embryology laboratory. Reprod Biomed Online. 2015;31:449–58.

    Article  PubMed  Google Scholar 

  5. Cirillo F, Patrizio P, Baccini M, Morenghi E, Ronchetti C, Cafaro L, et al. The human factor: does the operator performing the embryo transfer significantly impact the cycle outcome? Hum Reprod. 2020;35:275–82.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  6. Institute of Medicine (US) Committee on Quality of Health Care in America. To Err is human: building a safer health system [Internet]. Kohn LT, Corrigan JM, Donaldson MS, editors. Washington (DC): National Academies Press (US); 2000 [cited 2022 May 8]. Available from: http://www.ncbi.nlm.nih.gov/books/NBK225182/.

  7. Wikland M, Sjöblom C. The application of quality systems in ART programs. Mol Cell Endocrinol. 2000;166:3–7.

    Article  CAS  PubMed  Google Scholar 

  8. ESHRE Guideline Group on Good Practice in IVF Labs, De los Santos MJ, Apter S, Coticchio G, Debrock S, Lundin K, et al. Revised guidelines for good practice in IVF laboratories (2015. Hum Reprod. 2016;31:685–6.

    Google Scholar 

  9. Olofsson JI, Banker MR, Sjoblom LP. Quality management systems for your in vitro fertilization clinic’s laboratory: why bother? J Hum Reprod Sci. 2013;6:3–8.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Coroleu B, Barri PN, Carreras O, Martínez F, Veiga A, Balasch J. The usefulness of ultrasound guidance in frozen-thawed embryo transfer: a prospective randomized clinical trial. Hum Reprod. 2002;17:2885–90.

    Article  CAS  PubMed  Google Scholar 

  11. ESHRE Clinic PI Working Group, Vlaisavljevic V, Apter S, Capalbo A, D’Angelo A, Gianaroli L, et al. The Maribor consensus: report of an expert meeting on the development of performance indicators for clinical practice in ART. Hum Reprod Open. 2021;2021:hoab022.

    Article  PubMed Central  Google Scholar 

  12. ESHRE Special Interest Group of Embryology and Alpha Scientists in Reproductive Medicine. Electronic address: coticchio.biogenesi@grupposandonato.it. The Vienna consensus: report of an expert meeting on the development of ART laboratory performance indicators. Reprod Biomed Online. 2017;35:494–510.

  13. Coello A, Nohales M, Meseguer M, de Los Santos MJ, Remohí J, Cobo A. Prediction of embryo survival and live birth rates after cryotransfers of vitrified blastocysts. Reprod Biomed Online. 2021;42:881–91.

    Article  PubMed  Google Scholar 

  14. Alpha Scientists in Reproductive Medicine and ESHRE Special Interest Group of Embryology. The Istanbul consensus workshop on embryo assessment: proceedings of an expert meeting. Hum Reprod. 2011;26:1270–83.

    Article  Google Scholar 

  15. Storr A, Venetis CA, Cooke S, Kilani S, Ledger W. Inter-observer and intra-observer agreement between embryologists during selection of a single Day 5 embryo for transfer: a multicenter study. Hum Reprod. 2017;32:307–14.

    Article  PubMed  Google Scholar 

  16. Baxter Bendus AE, Mayer JF, Shipley SK, Catherino WH. Interobserver and intraobserver variation in day 3 embryo grading. Fertil Steril. 2006;86:1608–15.

    Article  PubMed  Google Scholar 

  17. Kovačič B, Prados FJ, Plas C, Woodward BJ, Verheyen G, Ramos L, et al. ESHRE Clinical Embryologist certification: the first 10 years. Hum Reprod Open. 2020;2020:hoaa026.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Zheng W, Yang C, Yang S, Sun S, Mu M, Rao M, et al. Obstetric and neonatal outcomes of pregnancies resulting from preimplantation genetic testing: a systematic review and meta-analysis. Hum Reprod Update. 2021;27:989–1012.

    Article  PubMed  Google Scholar 

  19. Bormann CL, Thirumalaraju P, Kanakasabapathy MK, Kandula H, Souter I, Dimitriadis I, et al. Consistency and objectivity of automated embryo assessments using deep neural networks. Fertil Steril. 2020;113:781-787.e1.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Bormann CL, Curchoe CL, Thirumalaraju P, Kanakasabapathy MK, Gupta R, Pooniwala R, et al. Deep learning early warning system for embryo culture conditions and embryologist performance in the ART laboratory. J Assist Reprod Genet. 2021;38:1641–6.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Dimitriadis I, Zaninovic N, Badiola AC, Bormann CL. Artificial intelligence in the embryology laboratory: a review. Reprod Biomed Online. 2022;44:435–48.

    Article  PubMed  Google Scholar 

  22. Abràmoff MD, Lou Y, Erginay A, Clarida W, Amelon R, Folk JC, et al. Improved automated detection of diabetic retinopathy on a publicly available dataset through integration of deep learning. Invest Ophthalmol Vis Sci. 2016;57:5200–6.

    Article  PubMed  Google Scholar 

  23. Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017;542:115–8.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  24. Hosny A, Parmar C, Quackenbush J, Schwartz LH, Aerts HJWL. Artificial intelligence in radiology. Nat Rev Cancer. 2018;18:500–10.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  25. Khosravi P, Kazemi E, Imielinski M, Elemento O, Hajirasouliha I. Deep convolutional neural networks enable discrimination of heterogeneous digital pathology images. EBioMedicine. 2018;27:317–28.

    Article  PubMed  Google Scholar 

  26. Shapiro LG, Stockman GC. Computer vision. Prentice hall New Jersey; 2001.

  27. Bortoletto P, Kanakasabapathy MK, Thirumalaraju P, Gupta R, Pooniwala R, Souter I, et al. Predicting blastocyst formation of day 3 embryos using a convolutional neural network (CNN): a machine learning approach. Fertil Steril. 2019;112:e272–3 (Elsevier).

    Article  Google Scholar 

  28. Dimitriadis I, Christou G, Dickinson K, McLellan S, Brock M, Souter I, et al. Cohort embryo selection (CES): a quick and simple method for selecting cleavage stage embryos that will become high quality blastocysts (HQB). Fertil Steril. 2017;108:e162–3 (Elsevier).

    Article  Google Scholar 

  29. Fitz VW, Kanakasabapathy MK, Thirumalaraju P, Kandula H, Ramirez LB, Boehnlein L, et al. Should there be an “AI” in TEAM? Embryologists selection of high implantation potential embryos improves with the aid of an artificial intelligence algorithm. J Assist Reprod Genet. 2021;38:2663–70.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  30. Bormann CL, Kanakasabapathy MK, Thirumalaraju P, Gupta R, Pooniwala R, Kandula H, et al. Performance of a deep learning based neural network in the selection of human blastocysts for implantation. eLife. 2020;9:e55301.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  31. Kanakasabapathy M, Dimitriadis I, Thirumalaraju P, Bormann CL, Souter I, Hsu J, et al. An inexpensive, automated artificial intelligence (AI) system for human embryo morphology evaluation and transfer selection. Fertil Steril. 2019;111: e11.

    Article  Google Scholar 

  32. Gardner DK, Surrey E, Minjarez D, Leitz A, Stevens J, Schoolcraft WB. Single blastocyst transfer: a prospective randomized trial. Fertil Steril. 2004;81:551–5.

    Article  CAS  PubMed  Google Scholar 

  33. Davar R, Poormoosavi SM, Mohseni F, Janati S. Effect of embryo transfer depth on IVF/ICSI outcomes: a randomized clinical trial. Int J Reprod Biomed. 2020;18:723–32.

    PubMed  PubMed Central  Google Scholar 

  34. Fanchin R, Righini C, Olivennes F, Taylor S, de Ziegler D, Frydman R. Uterine contractions at the time of embryo transfer alter pregnancy rates after in-vitro fertilization. Hum Reprod. 1998;13:1968–74.

    Article  CAS  PubMed  Google Scholar 

  35. Mortimer ST, Mortimer D. Quality and risk management in the IVF laboratory [Internet]. 2nd ed. Cambridge: Cambridge University Press; 2015 [cited 2022 May 20]. Available from: http://ebooks.cambridge.org/ref/id/CBO9781139680936.

  36. Hammond ER, Morbeck DE. Tracking quality: can embryology key performance indicators be used to identify clinically relevant shifts in pregnancy rate? Hum Reprod. 2019;34:37–43.

    Article  CAS  PubMed  Google Scholar 

  37. Dimitriadis I, Bormann CL, Thirumalaraju P, Kanakasabapathy M, Gupta R, Pooniwala R, et al. Artificial intelligence-enabled system for embryo classification and selection based on image analysis. Fertil Steril. 2019;111: e21.

    Article  Google Scholar 

  38. Zhan Q, Sierra ET, Malmsten J, Ye Z, Rosenwaks Z, Zaninovic N. Blastocyst score, a blastocyst quality ranking tool, is a predictor of blastocyst ploidy and implantation potential. F S Rep. 2020;1:133–41.

    PubMed  PubMed Central  Google Scholar 

  39. Zacà C, Coticchio G, Vigiliano V, Lagalla C, Nadalini M, Tarozzi N, et al. Fine-tuning IVF laboratory key performance indicators of the Vienna consensus according to female age. J Assist Reprod Genet. 2022;39:945–52.

    Article  PubMed  Google Scholar 

Download references

Funding

This work was partially supported by the Brigham Precision Medicine Developmental Award and Innovation Evergreen Fund (Brigham Precision Medicine Program, Brigham and Women’s Hospital), Partners Innovation Discovery Grant (Partners Healthcare): IGNITE Award (Connors Center for Women's Health & Gender Biology, Brigham and Women's Hospital) and R01AI138800, R01EB033866, R33AI140489, and R61AI140489 (National Institute of Health).

Author information

Authors and Affiliations

  1. Division of Reproductive Endocrinology and Infertility, Vincent Obstetrics and Gynecology, Massachusetts General Hospital, Harvard Medical School, 55 Fruit Street, Suite 10A, Boston, MA, 02114, USA

    Panagiotis Cherouveim, Victoria S. Jiang, Irene Souter, Irene Dimitriadis & Charles L. Bormann

  2. Division of Engineering in Medicine, Brigham and Women’s Hospital, Harvard Medical School, 65 Landsdowne Street, Cambridge, MA, 02139, USA

    Manoj Kumar Kanakasabapathy, Prudhvi Thirumalaraju & Hadi Shafiee

Authors
  1. Panagiotis Cherouveim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Victoria S. Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Manoj Kumar Kanakasabapathy
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Prudhvi Thirumalaraju
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Irene Souter
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Irene Dimitriadis
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Charles L. Bormann
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Hadi Shafiee
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection, and analysis were performed by Panagiotis Cherouveim, Charles Bormann, Victoria S. Jiang, Manoj Kumar Kanakasabapathy, and Prudhvi Thirumalaraju. The first draft of the manuscript was written by Panagiotis Cherouveim, and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Hadi Shafiee.

Ethics declarations

Ethics approval and consent to participate

Informed consent was obtained from each individual before participation. Study protocols were approved by the Institutional Review Board (IRB#2019P001000) at Massachusetts General Hospital and Brigham and Women’s Hospital.

Consent for publication

Not applicable.

Conflict of interest

Authors Dr. Hadi Shafiee, Dr. Charles Bormann, Prudhvi Thirumalaraju, and Manoj Kumar Kanakasabapathy wish to disclose a patent, currently licensed by a commercial entity, on the use of AI for embryology (US11321831B2). The rest of the authors declare that they have no competing interests.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cherouveim, P., Jiang, V.S., Kanakasabapathy, M.K. et al. Quality assurance (QA) for monitoring the performance of assisted reproductive technology (ART) staff using artificial intelligence (AI). J Assist Reprod Genet 40, 241–249 (2023). https://doi.org/10.1007/s10815-022-02649-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10815-022-02649-z

Keywords

search

Navigation