Abstract
State-of-the-art object detection and segmentation methods for microscopy images rely on supervised machine learning, which requires laborious manual annotation of training data. Here we present a self-supervised method based on time arrow prediction pre-training that learns dense image representations from raw, unlabeled live-cell microscopy videos. Our method builds upon the task of predicting the correct order of time-flipped image regions via a single-image feature extractor followed by a time arrow prediction head that operates on the fused features. We show that the resulting dense representations capture inherently time-asymmetric biological processes such as cell divisions on a pixel-level. We furthermore demonstrate the utility of these representations on several live-cell microscopy datasets for detection and segmentation of dividing cells, as well as for cell state classification. Our method outperforms supervised methods, particularly when only limited ground truth annotations are available as is commonly the case in practice. We provide code at https://github.com/weigertlab/tarrow.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: ICML, pp. 1597–1607 (2020)
Dorkenwald, M., Xiao, F., Brattoli, B., Tighe, J., Modolo, D.: SCVRL: shuffled contrastive video representation learning. In: CVPR, pp. 4132–4141 (2022)
Ericsson, L., Gouk, H., Loy, C.C., Hospedales, T.M.: Self-supervised representation learning: introduction, advances, and challenges. IEEE Sig. Process. Mag. 39(3), 42–62 (2022)
Etournay, R., Popović, M., Merkel, M., Nandi, A., Blasse, C., Aigouy, B., et al.: Interplay of cell dynamics and epithelial tension during morphogenesis of the Drosophila pupal wing. eLife 4, e07090 (2015)
Gidaris, S., Singh, P., Komodakis, N.: Unsupervised representation learning by predicting image rotations. In: ICLR, OpenReview.net (2018)
Greenwald, N.F., Miller, G., Moen, E., Kong, A., Kagel, A., et al.: Whole-cell segmentation of tissue images with human-level performance using large-scale data annotation and deep learning. Nat. Biotechnol. 40(4), 555–565 (2021)
Han, H., Dmitrieva, M., Sauer, A., Tam, K.H., Rittscher, J.: Self-supervised voxel-level representation rediscovers subcellular structures in volume electron microscopy. In: CVPRW, pp. 1874–1883 (2022)
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: CVPR, pp. 16000–16009 (2022)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)
Hsu, J., Gu, J., Wu, G., Chiu, W., Yeung, S.: Capturing implicit hierarchical structure in 3d biomedical images with self-supervised hyperbolic representations. In: NeurIPS, vol. 34, pp. 5112–5123 (2021)
Hu, K., Shao, J., Liu, Y., Raj, B., Savvides, M., Shen, Z.: Contrast and order representations for video self-supervised learning. In: ICCV, pp. 7939–7949 (2021)
Hua, T., Wang, W., Xue, Z., Ren, S., Wang, Y., Zhao, H.: On feature decorrelation in self-supervised learning. In: CVPR, pp. 9598–9608 (2021)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (2015)
Lee, H.Y., Huang, J.B., Singh, M., Yang, M.H.: Unsupervised representation learning by sorting sequences. In: ICCV, pp. 667–676 (2017)
Misra, I., Zitnick, C.L., Hebert, M.: Shuffle and learn: unsupervised learning using temporal order verification. In: ECCV, pp. 527–544 (2016)
Padovani, F., Mairhörmann, B., Falter-Braun, P., Lengefeld, J., Schmoller, K.M.: Segmentation, tracking and cell cycle analysis of live-cell imaging data with Cell-ACDC. BMC Biol. 20, 174 (2022)
Padovani, F., Mairhörmann, B., Lengefeld, J., Falter-Braun, P., Schmoller, K.: Cell-ACDC: segmentation, tracking, annotation and quantification of microscopy imaging data (dataset). https://zenodo.org/record/6795124 (2022)
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: feature learning by inpainting. In: CVPR, pp. 2536–2544 (2016)
Pickup, L.C., et al.: Seeing the arrow of time. In: CVPR, pp. 2043–2050 (2014)
Piscitello-Gómez, R., Gruber, F.S., Krishna, A., Duclut, C., Modes, C.D., et al.: Core PCP mutations affect short time mechanical properties but not tissue morphogenesis in the Drosophila pupal wing. bioRxiv (2022)
Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Schiappa, M.C., Rawat, Y.S., Shah, M.: Self-supervised learning for videos: a survey. ACM Comput. Surv. 55(13s), 1–37 (2022)
Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-CAM: visual explanations from deep networks via gradient-based localization. In: ICCV, pp. 618–626 (2017)
Stelzer, E.H.K., et al.: Light sheet fluorescence microscopy. Nat. Rev. Methods Primers 1(1), 1–25 (2021)
Stringer, C., Wang, T., Michaelos, M., Pachitariu, M.: Cellpose: a generalist algorithm for cellular segmentation. Nat. Methods 18(1), 100–106 (2021)
Tomer, R., Khairy, K., Keller, P.J.: Shedding light on the system: studying embryonic development with light sheet microscopy. Curr. Opin. Genet. Dev. 21(5), 558–565 (2011)
Ulicna, K., Vallardi, G., Charras, G., Lowe, A.: Mdck cell tracking reference dataset. https://rdr.ucl.ac.uk/articles/dataset/Cell_tracking_reference_dataset/16595978
Ulicna, K., Vallardi, G., Charras, G., Lowe, A.R.: Automated deep lineage tree analysis using a bayesian single cell tracking approach. Front. Comput. Sci. 3, 734559 (2021)
Ulman, V., Maška, M., Magnusson, K.E.G., Ronneberger, O., Haubold, C., et al.: An objective comparison of cell-tracking algorithms. Nat. Methods 14(12), 1141–1152 (2017). https://doi.org/10.1038/nmeth.4473
Wei, D., Lim, J., Zisserman, A., Freeman, W.T.: Learning and using the arrow of time. In: CVPR, pp. 8052–8060 (2018)
Weigert, M., Schmidt, U., Haase, R., Sugawara, K., Myers, G.: Star-convex polyhedra for 3d object detection and segmentation in microscopy. In: WACV, pp. 3666–3673 (2020)
Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R.R., Smola, A.J.: Deep sets. In: NeurIPS (2017)
Zbontar, J., Jing, L., Misra, I., LeCun, Y., Deny, S.: Barlow twins: self-supervised learning via redundancy reduction. In: ICML, pp. 12310–12320 (2021)
Acknowledgements
We thank Albert Dominguez (EPFL) and Uwe Schmidt for helpful comments, Natalie Dye (PoL Dresden) and Franz Gruber for providing the Flywing dataset, Benedikt Mairhörmann and Kurt Schmoller (Helmholtz Munich) for providing additional Yeast training data, and Alan Lowe (UCL) for providing the Mdck dataset. M.W. and B.G. are supported by the EPFL School of Life Sciences ELISIR program and CARIGEST SA.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Gallusser, B., Stieber, M., Weigert, M. (2023). Self-supervised Dense Representation Learning for Live-Cell Microscopy with Time Arrow Prediction. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14227. Springer, Cham. https://doi.org/10.1007/978-3-031-43993-3_52
Download citation
DOI: https://doi.org/10.1007/978-3-031-43993-3_52
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43992-6
Online ISBN: 978-3-031-43993-3
eBook Packages: Computer ScienceComputer Science (R0)
