Joint Optimization of Convolutional Neural Network and Image Preprocessing Selection for Embryo Grade Prediction in In Vitro Fertilization

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11845)


The convolutional neural network (CNN) is a standard tool for image recognition. To improve the performance of CNNs, it is important to design not only the network architecture but also the preprocessing of the input image. Extracting or enhancing the meaningful features of the input image in the preprocessing stage can help to improve the CNN performance. In this paper, we focus on the use of the well-known image processing filters, such as the edge extraction and denoising, and add the preprocessed images to the input of CNNs. As the optimal filter selection depends on dataset, we develop a joint optimization method of CNN and image processing filter selection. We represent the image processing filter selection by a binary vector and introduce the probability distribution of the vector. To derive the gradient-based optimization algorithm, we compute the gradients of weight and distribution parameters on the expected loss under the distribution. The proposed method is applied to an embryo grading task for in vitro fertilization, where the embryo grade is assigned based on the morphological criterion. The experimental result shows that the proposed method succeeds to reduce the test error by more than 8% compared with the naive CNN models.


Convolutional neural network Image processing filter Embryo grading In vitro fertilization 


  1. 1.
    Akimoto, Y., Shirakawa, S., Yoshinari, N., Uchida, K., Saito, S., Nishida, K.: Adaptive stochastic natural gradient method for one-shot neural architecture search. In: Proceedings of the 36th International Conference on Machine Learning (ICML), vol. 97, pp. 171–180 (2019)Google Scholar
  2. 2.
    Alfaraj, S., Alzaher, F., Alshwaiaer, S., Ahmed, A.: Pregnancy outcome of day 3 versus day 5 embryo transfer: a retrospective analysis. Asian Pac. J. Reprod. 6(2), 89–92 (2017). Scholar
  3. 3.
    Amari, S.: Natural gradient works efficiently in learning. Neural Comput. 10(2), 251–276 (1998)CrossRefGoogle Scholar
  4. 4.
    Calderon, S., et al.: Assessing the impact of the deceived non local means filter as a preprocessing stage in a convolutional neural network based approach for age estimation using digital hand X-ray images. In: 25th IEEE International Conference on Image Processing (ICIP), pp. 1752–1756 (2018).
  5. 5.
    Chen, T.J., Zheng, W.L., Liu, C.H., Huang, I., Lai, H.H., Liu, M.: Using deep learning with large dataset of microscope images to develop an automated embryo grading system. Fertil. Reprod. 01(01), 51–56 (2019). Scholar
  6. 6.
    Craciunas, L., et al.: Conventional and modern markers of endometrial receptivity: a systematic review and meta-analysis. Hum. Reprod. Update 25(2), 202–223 (2019). Scholar
  7. 7.
    He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. In: IEEE International Conference on Computer Vision (ICCV), pp. 1026–1034 (2015).
  8. 8.
    Khan, A., Gould, S., Salzmann, M.: Deep convolutional neural networks for human embryonic cell counting. In: Hua, G., Jégou, H. (eds.) ECCV 2016. LNCS, vol. 9913, pp. 339–348. Springer, Cham (2016). Scholar
  9. 9.
    Khosravi, P., et al.: Deep learning enables robust assessment and selection of human Blastocysts after in vitro fertilization. Nat. Partner J. Digit. Med. 02, 21 (2019). Scholar
  10. 10.
    Lin, M., Chen, Q., Yan, S.: Network in network. In: International Conference on Learning Representations (ICLR) (2014)Google Scholar
  11. 11.
    Moussavi, F., Wang, Y., Lorenzen, P., Oakley, J., Russakoff, D., Gould, S.: A unified graphical models framework for automated human embryo tracking in time lapse microscopy. In: IEEE 11th International Symposium on Biomedical Imaging, pp. 314–320 (2014)Google Scholar
  12. 12.
    Racowsky, C., Jackson, K.V., Cekleniak, N.A., Fox, J.H., Hornstein, M.D., Ginsburg, E.S.: The number of eight-cell embryos is a key determinant for selecting day 3 or day 5 transfer. Fertil. Steril. 73(3), 558–564 (2000). Scholar
  13. 13.
    Shirakawa, S., Iwata, Y., Akimoto, Y.: Dynamic optimization of neural network structures using probabilistic modeling. In: Thirty-Second AAAI Conference on Artificial Intelligence (AAAI), pp. 4074–4082 (2018)Google Scholar
  14. 14.
    Shota, S., Shirakawa, S., Akimoto, Y.: Embedded feature selection using probabilistic model-based optimization. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion (GECCO), pp. 1922–1925 (2018).
  15. 15.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (ICLR) (2015)Google Scholar
  16. 16.
    Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014)MathSciNetzbMATHGoogle Scholar
  17. 17.
    Sutskever, I., Martens, J., Dahl, G., Hinton, G.: On the importance of initialization and momentum in deep learning. In: Proceedings of the 30th International Conference on Machine Learning (ICML), vol. 28, pp. 1139–1147 (2013)Google Scholar
  18. 18.
    Veeck, L.: An Atlas of Human Gametes and Conceptuses: An Illustrated Reference for Assisted Reproductive Technology. The Parthenon Publishing Group, New York (1999)CrossRefGoogle Scholar
  19. 19.
    Wang, Y., Moussavi, F., Lorenzen, P.: Automated embryo stage classification in time-lapse microscopy video of early human embryo development. In: Mori, K., Sakuma, I., Sato, Y., Barillot, C., Navab, N. (eds.) MICCAI 2013. LNCS, vol. 8150, pp. 460–467. Springer, Heidelberg (2013). Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Graduate School of Environment and Information SciencesYokohama National UniversityYokohamaJapan
  2. 2.Department of Information and Communication TechnologyUniversitas NasionalJakartaIndonesia
  3. 3.SkillUp AI Co., Ltd.TokyoJapan
  4. 4.Infertility Laboratory of Permata Hati ProgramRSUP DR SardjitoYogyakartaIndonesia
  5. 5.Department of Electrical and Computer EngineeringCurtin University MalaysiaMiriMalaysia

Personalised recommendations