Abstract
The developmental process of embryos follows a monotonic order. An embryo can progressively cleave from one cell to multiple cells and finally transform to morula and blastocyst. For time-lapse videos of embryos, most existing developmental stage classification methods conduct per-frame predictions using an image frame at each time step. However, classification using only images suffers from overlapping between cells and imbalance between stages. Temporal information can be valuable in addressing this problem by capturing movements between neighboring frames. In this work, we propose a two-stream model for developmental stage classification. Unlike previous methods, our two-stream model accepts both temporal and image information. We develop a linear-chain conditional random field (CRF) on top of neural network features extracted from the temporal and image streams to make use of both modalities. The linear-chain CRF formulation enables tractable training of global sequential models over multiple frames while also making it possible to inject monotonic development order constraints into the learning process explicitly. We demonstrate our algorithm on two time-lapse embryo video datasets: i) mouse and ii) human embryo datasets. Our method achieves 98.1% and 80.6% for mouse and human embryo stage classification, respectively. Our approach will enable more profound clinical and biological studies and suggests a new direction for developmental stage classification by utilizing temporal information.
S. Lukyanenko—Works were done during the internship at Harvard University.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Since the potentials in a CRF do not need to be probabilities, normalization via the softmax function is not strictly necessary. However, we found the normalization to be helpful for stable training. Note that if the unary potential is defined to be the output of a log-softmax function (which is not the case in our approach), the model will reduce to a Maximum Entropy Markov Model.
- 2.
The single-model losses and the CRF negative log likelihood are complementary each other by taking into account local and global predictions, respectively.
References
Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the kinetics dataset. In: proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299–6308 (2017)
Cicconet, M., Gutwein, M., Gunsalus, K.C., Geiger, D.: Label free cell-tracking and division detection based on 2D time-lapse images for lineage analysis of early embryo development. Comput. Biol. Med. 51, 24–34 (2014)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)
Feichtenhofer, C., Pinz, A., Zisserman, A.: Convolutional two-stream network fusion for video action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1933–1941 (2016)
Gers, F.A., Schmidhuber, J., Cummins, F.: Learning to forget: Continual prediction with LSTM. IET (1999)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Horn, B.K., Schunck, B.G.: Determining optical flow. Artif. Intell. 17(1–3), 185–203 (1981)
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., Fei-Fei, L.: Large-scale video classification with convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1725–1732 (2014)
Khan, A., Gould, S., Salzmann, M.: Deep convolutional neural networks for human embryonic cell counting. In: Hua, G., Jégou, H. (eds.) ECCV 2016. LNCS, vol. 9913, pp. 339–348. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46604-0_25
Lau, T., Ng, N., Gingold, J., Desai, N., McAuley, J., Lipton, Z.C.: Embryo staging with weakly-supervised region selection and dynamically-decoded predictions. In: Machine Learning for Healthcare Conference, pp. 663–679. PMLR (2019)
Leahy, B.D., et al.: Automated measurements of key morphological features of human embryos for IVF. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12265, pp. 25–35. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59722-1_3
Ng, N.H., McAuley, J.J., Gingold, J., Desai, N., Lipton, Z.C.: Predicting embryo morphokinetics in videos with late fusion nets & dynamic decoders. In: ICLR (Workshop) (2018)
Rad, R.M., Saeedi, P., Au, J., Havelock, J.: Cell-Net: embryonic cell counting and centroid localization via residual incremental atrous pyramid and progressive upsampling convolution. IEEE Access 7, 81945–81955 (2019)
Rush, A.M.: Torch-struct: Deep structured prediction library. arXiv preprint arXiv:2002.00876 (2020)
Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: Neural Information Processing Systems (2014)
Sutton, O.: Introduction to k nearest neighbour classification and condensed nearest neighbour data reduction. University lectures, University of Leicester, vol. 1 (2012)
Acknowledgements
This work was funded in part by NIH grants 5U54CA225088 and R01HD104969, NSF Grant NCS-FO 1835231, the NSF-Simons Center for Mathematical and Statistical Analysis of Biology at Harvard (award number 1764269), the Harvard Quantitative Biology Initiative, and Sagol fund for studying embryos and stem cells; Perelson Fund.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Lukyanenko, S. et al. (2021). Developmental Stage Classification of Embryos Using Two-Stream Neural Network with Linear-Chain Conditional Random Field. In: de Bruijne, M., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2021. MICCAI 2021. Lecture Notes in Computer Science(), vol 12908. Springer, Cham. https://doi.org/10.1007/978-3-030-87237-3_35
Download citation
DOI: https://doi.org/10.1007/978-3-030-87237-3_35
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87236-6
Online ISBN: 978-3-030-87237-3
eBook Packages: Computer ScienceComputer Science (R0)
