Deep learning neural networks have been used to predict the developmental fate and implantation potential of embryos with high accuracy. Such networks have been used as an assistive quality assurance (QA) tool to identify perturbations in the embryo culture environment which may impact clinical outcomes. The present study aimed to evaluate the utility of an AI-QA tool to consistently monitor ART staff performance (MD and embryologist) in embryo transfer (ET), embryo vitrification (EV), embryo warming (EW), and trophectoderm biopsy (TBx).
Pregnancy outcomes from groups of 20 consecutive elective single day 5 blastocyst transfers were evaluated for the following procedures: MD performed ET (N = 160 transfers), embryologist performed ET (N = 160 transfers), embryologist performed EV (N = 160 vitrification procedures), embryologist performed EW (N = 160 warming procedures), and embryologist performed TBx (N = 120 biopsies). AI-generated implantation probabilities for the same embryo cohorts were estimated, as were mean AI-predicted and actual implantation rates for each provider and compared using Wilcoxon singed-rank test.
Actual implantation rates following ET performed by one MD provider: “H” was significantly lower than AI-predicted (20% vs. 61%, p = 0.001). Similar results were observed for one embryologist, “H” (30% vs. 60%, p = 0.011). Embryos thawed by embryologist “H” had lower implantation rates compared to AI prediction (25% vs. 60%, p = 0.004). There were no significant differences between actual and AI-predicted implantation rates for EV, TBx, or for the rest of the clinical staff performing ET or EW.
AI-based QA tools could provide accurate, reproducible, and efficient staff performance monitoring in an ART practice.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Price excludes VAT (USA)
Tax calculation will be finalised during checkout.
Steptoe PC, Edwards RG. Birth after the reimplantation of a human embryo. Lancet. 1978;312:366.
Sunderam S, Kissin DM, Zhang Y, Jewett A, Boulet SL, Warner L, et al. Assisted reproductive technology surveillance - United States, 2018. MMWR Surveill Summ. 2022;71:1–19.
Chambers GM, Dyer S, Zegers-Hochschild F, de Mouzon J, Ishihara O, Banker M, et al. International Committee for Monitoring Assisted Reproductive Technologies world report: assisted reproductive technology, 2014†. Hum Reprod. 2021;36:2921–34.
Go KJ. “By the work, one knows the workman”: the practice and profession of the embryologist and its translation to quality in the embryology laboratory. Reprod Biomed Online. 2015;31:449–58.
Cirillo F, Patrizio P, Baccini M, Morenghi E, Ronchetti C, Cafaro L, et al. The human factor: does the operator performing the embryo transfer significantly impact the cycle outcome? Hum Reprod. 2020;35:275–82.
Institute of Medicine (US) Committee on Quality of Health Care in America. To Err is human: building a safer health system [Internet]. Kohn LT, Corrigan JM, Donaldson MS, editors. Washington (DC): National Academies Press (US); 2000 [cited 2022 May 8]. Available from: http://www.ncbi.nlm.nih.gov/books/NBK225182/.
Wikland M, Sjöblom C. The application of quality systems in ART programs. Mol Cell Endocrinol. 2000;166:3–7.
ESHRE Guideline Group on Good Practice in IVF Labs, De los Santos MJ, Apter S, Coticchio G, Debrock S, Lundin K, et al. Revised guidelines for good practice in IVF laboratories (2015. Hum Reprod. 2016;31:685–6.
Olofsson JI, Banker MR, Sjoblom LP. Quality management systems for your in vitro fertilization clinic’s laboratory: why bother? J Hum Reprod Sci. 2013;6:3–8.
Coroleu B, Barri PN, Carreras O, Martínez F, Veiga A, Balasch J. The usefulness of ultrasound guidance in frozen-thawed embryo transfer: a prospective randomized clinical trial. Hum Reprod. 2002;17:2885–90.
ESHRE Clinic PI Working Group, Vlaisavljevic V, Apter S, Capalbo A, D’Angelo A, Gianaroli L, et al. The Maribor consensus: report of an expert meeting on the development of performance indicators for clinical practice in ART. Hum Reprod Open. 2021;2021:hoab022.
ESHRE Special Interest Group of Embryology and Alpha Scientists in Reproductive Medicine. Electronic address: firstname.lastname@example.org. The Vienna consensus: report of an expert meeting on the development of ART laboratory performance indicators. Reprod Biomed Online. 2017;35:494–510.
Coello A, Nohales M, Meseguer M, de Los Santos MJ, Remohí J, Cobo A. Prediction of embryo survival and live birth rates after cryotransfers of vitrified blastocysts. Reprod Biomed Online. 2021;42:881–91.
Alpha Scientists in Reproductive Medicine and ESHRE Special Interest Group of Embryology. The Istanbul consensus workshop on embryo assessment: proceedings of an expert meeting. Hum Reprod. 2011;26:1270–83.
Storr A, Venetis CA, Cooke S, Kilani S, Ledger W. Inter-observer and intra-observer agreement between embryologists during selection of a single Day 5 embryo for transfer: a multicenter study. Hum Reprod. 2017;32:307–14.
Baxter Bendus AE, Mayer JF, Shipley SK, Catherino WH. Interobserver and intraobserver variation in day 3 embryo grading. Fertil Steril. 2006;86:1608–15.
Kovačič B, Prados FJ, Plas C, Woodward BJ, Verheyen G, Ramos L, et al. ESHRE Clinical Embryologist certification: the first 10 years. Hum Reprod Open. 2020;2020:hoaa026.
Zheng W, Yang C, Yang S, Sun S, Mu M, Rao M, et al. Obstetric and neonatal outcomes of pregnancies resulting from preimplantation genetic testing: a systematic review and meta-analysis. Hum Reprod Update. 2021;27:989–1012.
Bormann CL, Thirumalaraju P, Kanakasabapathy MK, Kandula H, Souter I, Dimitriadis I, et al. Consistency and objectivity of automated embryo assessments using deep neural networks. Fertil Steril. 2020;113:781-787.e1.
Bormann CL, Curchoe CL, Thirumalaraju P, Kanakasabapathy MK, Gupta R, Pooniwala R, et al. Deep learning early warning system for embryo culture conditions and embryologist performance in the ART laboratory. J Assist Reprod Genet. 2021;38:1641–6.
Dimitriadis I, Zaninovic N, Badiola AC, Bormann CL. Artificial intelligence in the embryology laboratory: a review. Reprod Biomed Online. 2022;44:435–48.
Abràmoff MD, Lou Y, Erginay A, Clarida W, Amelon R, Folk JC, et al. Improved automated detection of diabetic retinopathy on a publicly available dataset through integration of deep learning. Invest Ophthalmol Vis Sci. 2016;57:5200–6.
Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017;542:115–8.
Hosny A, Parmar C, Quackenbush J, Schwartz LH, Aerts HJWL. Artificial intelligence in radiology. Nat Rev Cancer. 2018;18:500–10.
Khosravi P, Kazemi E, Imielinski M, Elemento O, Hajirasouliha I. Deep convolutional neural networks enable discrimination of heterogeneous digital pathology images. EBioMedicine. 2018;27:317–28.
Shapiro LG, Stockman GC. Computer vision. Prentice hall New Jersey; 2001.
Bortoletto P, Kanakasabapathy MK, Thirumalaraju P, Gupta R, Pooniwala R, Souter I, et al. Predicting blastocyst formation of day 3 embryos using a convolutional neural network (CNN): a machine learning approach. Fertil Steril. 2019;112:e272–3 (Elsevier).
Dimitriadis I, Christou G, Dickinson K, McLellan S, Brock M, Souter I, et al. Cohort embryo selection (CES): a quick and simple method for selecting cleavage stage embryos that will become high quality blastocysts (HQB). Fertil Steril. 2017;108:e162–3 (Elsevier).
Fitz VW, Kanakasabapathy MK, Thirumalaraju P, Kandula H, Ramirez LB, Boehnlein L, et al. Should there be an “AI” in TEAM? Embryologists selection of high implantation potential embryos improves with the aid of an artificial intelligence algorithm. J Assist Reprod Genet. 2021;38:2663–70.
Bormann CL, Kanakasabapathy MK, Thirumalaraju P, Gupta R, Pooniwala R, Kandula H, et al. Performance of a deep learning based neural network in the selection of human blastocysts for implantation. eLife. 2020;9:e55301.
Kanakasabapathy M, Dimitriadis I, Thirumalaraju P, Bormann CL, Souter I, Hsu J, et al. An inexpensive, automated artificial intelligence (AI) system for human embryo morphology evaluation and transfer selection. Fertil Steril. 2019;111: e11.
Gardner DK, Surrey E, Minjarez D, Leitz A, Stevens J, Schoolcraft WB. Single blastocyst transfer: a prospective randomized trial. Fertil Steril. 2004;81:551–5.
Davar R, Poormoosavi SM, Mohseni F, Janati S. Effect of embryo transfer depth on IVF/ICSI outcomes: a randomized clinical trial. Int J Reprod Biomed. 2020;18:723–32.
Fanchin R, Righini C, Olivennes F, Taylor S, de Ziegler D, Frydman R. Uterine contractions at the time of embryo transfer alter pregnancy rates after in-vitro fertilization. Hum Reprod. 1998;13:1968–74.
Mortimer ST, Mortimer D. Quality and risk management in the IVF laboratory [Internet]. 2nd ed. Cambridge: Cambridge University Press; 2015 [cited 2022 May 20]. Available from: http://ebooks.cambridge.org/ref/id/CBO9781139680936.
Hammond ER, Morbeck DE. Tracking quality: can embryology key performance indicators be used to identify clinically relevant shifts in pregnancy rate? Hum Reprod. 2019;34:37–43.
Dimitriadis I, Bormann CL, Thirumalaraju P, Kanakasabapathy M, Gupta R, Pooniwala R, et al. Artificial intelligence-enabled system for embryo classification and selection based on image analysis. Fertil Steril. 2019;111: e21.
Zhan Q, Sierra ET, Malmsten J, Ye Z, Rosenwaks Z, Zaninovic N. Blastocyst score, a blastocyst quality ranking tool, is a predictor of blastocyst ploidy and implantation potential. F S Rep. 2020;1:133–41.
Zacà C, Coticchio G, Vigiliano V, Lagalla C, Nadalini M, Tarozzi N, et al. Fine-tuning IVF laboratory key performance indicators of the Vienna consensus according to female age. J Assist Reprod Genet. 2022;39:945–52.
This work was partially supported by the Brigham Precision Medicine Developmental Award and Innovation Evergreen Fund (Brigham Precision Medicine Program, Brigham and Women’s Hospital), Partners Innovation Discovery Grant (Partners Healthcare): IGNITE Award (Connors Center for Women's Health & Gender Biology, Brigham and Women's Hospital) and R01AI138800, R01EB033866, R33AI140489, and R61AI140489 (National Institute of Health).
Ethics approval and consent to participate
Informed consent was obtained from each individual before participation. Study protocols were approved by the Institutional Review Board (IRB#2019P001000) at Massachusetts General Hospital and Brigham and Women’s Hospital.
Consent for publication
Conflict of interest
Authors Dr. Hadi Shafiee, Dr. Charles Bormann, Prudhvi Thirumalaraju, and Manoj Kumar Kanakasabapathy wish to disclose a patent, currently licensed by a commercial entity, on the use of AI for embryology (US11321831B2). The rest of the authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Cherouveim, P., Jiang, V.S., Kanakasabapathy, M.K. et al. Quality assurance (QA) for monitoring the performance of assisted reproductive technology (ART) staff using artificial intelligence (AI). J Assist Reprod Genet 40, 241–249 (2023). https://doi.org/10.1007/s10815-022-02649-z